Evaluating LLMs Code Reasoning Under Real-World Context Changshu Liu
[email protected] University of Illinois Urbana-Champaign Urbana, Illinois, USA
arXiv:2604.12881v1 [cs.SE] 14 Apr 2026
Abstract Code reasoning tasks are increasingly crucial to evaluating large language models (LLMs). Yet most existing benchmarks rely on simplistic, LLM-generated snippets or human-written solutions to code challenges and often restrict inputs and outputs to primitive types, failing to reflect the structure and dependencies of real-world projects. These simplifications limit their ability to measure practical generalizability. We present R2 Eval1 , a benchmark of 135 code reasoning problems drawn from ten widely used Python projects. Unlike prior work, R2 Eval serializes compound and custom types, preserving real-world data complexity and enabling a more realistic assessment of LLMs.
CCS Concepts • Software and its engineering → Software testing and debugging; Software notations and tools; Software functional properties. ACM Reference Format: Changshu Liu. 2026. Evaluating LLMs Code Reasoning Under Real-World Context. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-Companion ’26), April 12–18, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3774748.3787747
1
Research Problem & Motivation
Recently, code reasoning, which measures LLMs’ capability of understanding the behavior of programs during execution, has become a popular evaluation strategy to assess LLMs. This capability is particularly crucial for tasks that require the simulation of the control/data flow of the program, e.g., code translation[10, 11, 20], program repair [22, 24], and code generation [5, 12, 17]. Existing benchmarks assess LLMs’ code reasoning ability in input/output prediction on either LLM-generated snippets or humanwritten solutions to programming challenges, but neither captures the structural and contextual complexity of production code. CRUXEval [7] contains short, self-contained Python programs. Datasets such as Avatar [1], ClassEval [6], and HumanEval [3] are more challenging—some problems involve code spanning multiple methods—yet they remain far from reflecting the rich inter- and intraprocedural dependencies typical of real repositories. CodeSense [21] makes a step further and extracts methods from real-world projects, but it discards those with non-primitive variables in the input/output. 1 R2 Eval stands for Realistic Code Reasoning Evaluation
This work is licensed under a Creative Commons Attribution 4.0 International License. ICSE-Companion ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2296-7/2026/04 https://doi.org/10.1145/3774748.3787747
Consequently, although state-of-the-art LLMs perform strongly on the existing benchmarks, especially the widely used CRUXEval, their ability to generalize to real-world projects remains uncertain and warrants further investigation.
2
Approach and Contributions
To overcome the limitation of existing approaches, we propose R2 Eval, a code reasoning benchmark drawn from real-world projects. Our pipeline combines program analysis to automatically serialize/deserialize compound, complex, and custom types, well beyond the primitive-only assumptions common in prior work [13, 21]. R2 Eval first collects code reasoning problems from ten popular Python projects, including: scikit-learn, django, requests, seaborn, sphinx, pytest, astropy, xarray, matplotlib, and sympy. R2 Eval extracts target methods from them and records the runtime inputs and outputs. A big challenge here is that values in real-world projects often involve complex, custom objects. Unlike primitive types, e.g., int, str, which can be directly printed, custom objects often lack the actual implementation of __str__() or str(). Therefore, printing such objects typically yields only their memory address or hash value, rather than actual values. To overcome this challenge, R2 Eval leverages static and dynamic program analysis to iteratively decompose complex data types of inputs/outputs into primitives (or core compound types), serializing them into JSON formats. The serialized inputs or outputs will be provided to the LLM, along with code and intra/interdependencies, to predict outputs or inputs. Another big challenge is automated detection of false negatives, which may happen if we textually compare predictions and ground truth. To overcome this in a scalable fashion, R2 Eval deserializes LLM’s predictions and creates tests with objects to evaluate correctness at runtime. Figure 1-a shows a reasoning problem _pixel_to_world_values_all that takes a custom-typed variable _wcs extended from the base class BaseWCSWrapper (Figure 1-b) as input. R2 Eval serializes _wcs into the JSON representation shown in Figure 1-c. R2 Eval currently offers 135 real-world reasoning problems, each represented as a triplet {𝑃, 𝐼, 𝑂 }, where 𝑃 is the code for the reasoning task (including relevant method dependencies and class context), and 𝐼 /𝑂 denote serialized method input(s)/output. Experiment results on R2 Eval show that the performance of assessed LLMs significantly drops by 64.32% and 52.22%, on input prediction and output prediction, respectively, compared to their performance on CRUXEval. These results question bold claims about code reasoning abilities of LLMs, promote more meaningful evaluation of code reasoning, aiming to advance LLMs for robust code reasoning under real-world settings.
3
Evaluation
We compare R2 Eval with CRUXEval, a widely used benchmark for code reasoning. Because R2 Eval is smaller, we sample 135 problems
ICSE-Companion ’26, April 12–18, 2026, Rio de Janeiro, Brazil
Changshu Liu
class BaseWCSWrapper(): class SlicedLowLevelWCS(BaseWCSWrapper): # wcs : The WCS object to wrap …… def __init__(self, wcs): def _pixel_to_world_values_all( self._wcs = wcs self, *pixel_arrays ): ...... …… for ipix in range(self._wcs. class WCS(): pixel_n_dim): def __init__(self, naxis=None,keysel=None, if isinstance( colsel=None): self._slices_pixel[ipix], self._init_kwargs = { numbers.Integral): "keysel": copy.copy(keysel), pixel_arrays_new.append "colsel": copy.copy(colsel), (self._slices_pixel[ipix]) } …… self.naxis = naxis ......
(a) Reasoning problem _pixel_to_world_values_all
(b) Definition of class WCS
self._wcs: { "_init_kwargs": { "keysel": None, "colsel": None }, "naxis": 3, "_naxis": None, "_pixel_bounds": None }
(c) Serialization of self._wcs
Figure 1: Example of custom type variable serialization in R2 Eval Input Prediction Only
Output Prediction Only
Both Input and Output Prediction
CRUXEval R2Eval
Figure 2: Unique and common problems each LLM succeeds in predicting their inputs and outputs Table 1: Code Reasoning Performance of LLMs on R2 Eval and CRUXEval. The ↓ indicates the performance drop from CRUXEval to R2 Eval Subject LLMs o4-mini Gemini-2.5-Pro DeepSeek-R1 GPT-4.1 Gemini-1.5-Pro DeepSeekCoder-33B-Inst Total
R2 Eval Input Prediction(%) Output Prediction(%) 20.00 (↓ 72.59) 28.15 (↓ 63.70) 15.56 (↓ 76.29) 34.07 (↓ 54.82) 21.48 (↓ 73.33) 31.85 (↓ 55.56) 14.81 (↓ 72.60) 28.15 (↓ 59.26) 17.04 (↓ 60.74) 25.93 (↓ 50.37) 12.59 (↓ 30.37) 20.74 (↓ 29.63) 16.91 (↓ 64.32) 28.15 (↓ 52.22)
from CRUXEval to match R2 Eval ’s size. We evaluate six LLMs—o4mini [19], Gemini-2.5-Pro [4], DeepSeek-R1 [8], GPT-4.1 [18], Gemini-1.5-Pro [23], and DeepSeek-Coder-Inst-33B [9]—and report input-prediction and output-prediction performance in Table 1. We have the following key findings: • Reasoning LLMs (o4-mini, Gemini-2.5, and DeepSeek-R1) consistently outperform non-reasoning LLMs across both R2 Eval and CRUXEval. On average, the margin is 13.95% for input prediction and 12.22% for output prediction. • Moving from CRUXEval to R2 Eval, LLMs’ performance drops substantially: 64.32% for input prediction and 52.22% for output prediction. We attribute this to code properties that are intrinsic to real-world programs but largely absent from CRUXEval, including inter- and intra-procedural dependencies, third-party API usage, and non-primitive (compound or custom) data types. • Figure 2 shows the unique and common problems each LLM succeeds in predicting their inputs and outputs on both R2 Eval and CRUXEval. We observe that moving from CRUXEval (inner doughnuts) to R2 Eval (outer doughnuts), the percentages of overlap between input and output prediction success decrease
CRUXEval Input Prediction(%) Output Prediction(%) 92.59 91.85 91.85 88.89 94.81 87.41 87.41 87.41 77.78 76.30 42.96 50.37 81.23 80.37
drastically. We speculate that it is because in real-world projects, the complex code constructs and rich dependencies make the backward reasoning in the input prediction obviously more difficult than the forward reasoning in the output prediction.
4
Background and Related Work
CRUXEval [7] and CodeMind [16] are early works in evaluating LLMs for input-ouptut prediction. REval [2] and CES [14] take one step forward and complement output prediction with prediction on the intermediate state of the code execution. ExerScope [15] use program analysis tools to identify code properties impacting the performance of LLMs on code reasoning. However, all these techniques collect reasoning problems from human-written/LLM-generated simple code that is far from real-world complexity. Another two closely related works are EXE [13] and CodeSense [21], which evaluate LLM code reasoning on open-source Python repositories; however, neither of them considers non-primitive data types in the input. In contrast, our work preserves the complexity of the input in the real-world project via serializing complex custom types into a readable JSON-like format.
Evaluating LLMs Code Reasoning Under Real-World Context
References [1] Wasi Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2023. Avatar: A parallel corpus for java-python program translation. In Findings of the Association for Computational Linguistics: ACL 2023. 2268–2281. [2] Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025. Reasoning Runtime Behavior of a Program with LLM: How Far Are We? . In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, 140–152. doi:10.1109/ICSE55347.2025. 00012 [3] Mark Chen. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [4] Google DeepMind. 2025. Gemini 2.5 Pro (March 25 version). https://cloud.google. com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro. Multimodal large language model. [5] Yangruibo Ding, Jinjun Peng, Marcus Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. 2024. Semcoder: Training code language models with comprehensive semantics reasoning. Advances in Neural Information Processing Systems 37 (2024), 60275–60308. [6] Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manuallycrafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861 (2023). [7] Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065 (2024). [8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). [9] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). [10] Ali Reza Ibrahimzada. 2024. Program decomposition and translation with static analysis. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 453–455. [11] Ali Reza Ibrahimzada, Kaiyao Ke, Mrigank Pawagi, Muhammad Salman Abid, Rangeet Pan, Saurabh Sinha, and Reyhaneh Jabbarvand. 2025. AlphaTrans: A Neuro-Symbolic Compositional Approach for Repository-Level Code Translation and Validation. Proceedings of the ACM on Software Engineering 2, FSE (2025), 2454–2476. [12] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515 (2024). [13] Rob Kopel. 2025. EXecution-Eval:Can language models execute real-world code? (2025). [14] Changshu Liu, Yang Chen, and Reyhaneh Jabbarvand. 2025. Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models. arXiv preprint arXiv:2510.15079 (2025). [15] Changshu Liu and Reyhan Jabbarvand. 2025. A tool for in-depth analysis of code execution reasoning of large language models. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1178–1182. [16] Changshu Liu, Shizhuo Dylan Zhang, Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand. 2024. Codemind: A framework to challenge large language models for code reasoning. arXiv preprint arXiv:2402.09664 (2024). [17] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2023), 21558–21572. [18] OpenAI. 2025. GPT-4.1 (April 14 version). https://openai.com/index/gpt-4-1/. Large language model. [19] OpenAI. 2025. o4-mini (April 16 version). https://platform.openai.com/docs/ models/o4-mini. Large language model. [20] Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in translation: A study of bugs introduced by large language models while translating code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. [21] Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek, Jinjun Peng, Gail Kaiser, Baishakhi Ray, and Wei Le. 2025. CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning. arXiv preprint arXiv:2506.00750 (2025). [22] Yuheng Tang, Hongwei Li, Kaijie Zhu, Michael Yang, Yangruibo Ding, and Wenbo Guo. 2025. Co-PatcheR: Collaborative Software Patching with Component (s)specific Small Reasoning Models. arXiv preprint arXiv:2505.18955 (2025). [23] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint
ICSE-Companion ’26, April 12–18, 2026, Rio de Janeiro, Brazil
arXiv:2312.11805 (2023). [24] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 (2024).