CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction Jun Gao1 Yun Peng3 Qian Qiao 7 Changhai Zhou5 Yuhua Zhou1 Shiyang Zhang6 Shichao Weng5 Zhenchang Xing4 Xiaoxue Ren12 * 1 School of Software Technology, Zhejiang University 2 Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security 3 Chinese University of Hong Kong 4 CSIRO’s Data61 5 Fudan University 6 Yale University 7 Independent Researcher {jgao1106, xxren}@zju.edu.cn
arXiv:2604.25399v1 [cs.SE] 28 Apr 2026
Abstract
Input
Despite strong performance on code generation tasks, it remains unclear whether large language models (LLMs) genuinely reason about code execution. Existing code reasoning benchmarks primarily evaluate final output correctness under a single canonical implementation, leaving two critical aspects underexplored: (1) whether LLMs can maintain consistency to functionally equivalent implementations, and (2) whether LLMs can accurately reason about intermediate execution states. We introduce CoRE, a Code Reasoning benchmark that evaluates code reasoning through implementation invariance and process transparency. Extensive evaluations on eight frontier LLMs reveal two fundamental limitations. First, models exhibit a substantial robustness gap, with performance varying significantly across equivalent implementations. Second, we observe superficial execution, where models arrive at correct final outputs without correctly reasoning about intermediate execution states. Together, these findings demonstrate that output-only evaluations are insufficient for assessing code reasoning and position CoRE as a necessary benchmark for evaluating robust and faithful code reasoning.1
1
def func(n): <GPT-Style Code> assert func(x) == ?? def func(n: int): <Gemini-Style Code> assert func(x) == ??
At step 22, when i=1, j=2, and k=3, what is the sum `l[i] + l[j] % l[k]`? In the 1st-iter of the loop, what is the value of `x`?
Implementation Invariance
# Gemini-Style assert func(x) == y'
# DeepSeek-Style assert func(x) == y
Reasoning
Process Transparency
At step 22, the sum of `l[i] + l[j] % l[k]` is 78. After executing `x -= 1`, x is 10.
(b) Holistic evaluation of our CoRE.
Figure 1: The code reasoning comparison of standard code reasoning evaluation and our holistic evaluation with diverse implementations and intermediate probing.
2025; Gao et al., 2025b). This discrepancy raises a fundamental question: Do LLMs truly understand the execution logic or are they merely relying on superficial heuristics?
* *Corresponding Author
available
# GPT-Style assert func(x) == y
def func(n: int): <DeepSeek-Style Code> assert func(x) == ??
The capabilities of Large Language Models (LLMs) in code-relevant tasks have evolved rapidly from simple code completion to solving complex programming problems (Hurst et al., 2024; Liu et al., 2024a; Yang et al., 2024; Bouzenia et al., 2024; Sun et al., 2025; Guo et al., 2025; Zhong et al., 2025; Zhou et al., 2026a). However, recent studies indicate that the ability to generate syntactically correct code does not necessarily imply a genuine understanding of its execution (Zhao et al., Data and code are https://github.com/ZJUSig/CoRE.
assert func(x) == y
Reasoning (a) Standard evaluation.
Introduction
1
Output
def func(n): <Canonical Code> assert func(x) == ??
at
In this paper, we argue that current benchmarks are insufficient due to two critical flaws. First, they lack implementation invariance, defined as the ability to evaluate reasoning robustness across functionally equivalent but structurally or lexically distinct code implementations. However, benchmarks like CruxEval (Gu et al., 2024), LiveCodeBenchO (Jain et al.), and REval (Chen et al., 2025) primarily rely on a single canonical solutions, as illustrated in Fig.1(a). Second, existing benchmarks lack process transparency, as they typically predict the final outputs without verifying intermediate execution states. While REval pioneered the exploration of intermediate reasoning states, it remains limited by a rigid templating paradigm and a fundamental disregard for implementation invariance.
Avg. Cyclomatic Complexity
6
CoRE (Ours)
5
LiveCodeBench-O REval
4 3
CruxEval
2 0
2
4
6
Avg. Number of Test Cases
Samples Tests Impl.
J ↓ CC ↑ Inter. ↑
CruxEval LiveCodeBench-O REval
800 478 955
1.0 5.2 6.4
1.0 1.0 1.0
1.0 1.0 1.0
2.4 4.1 4.0
% % 3.0
CoRE (ours)
1,978
7.8
4.3
0.6
5.0
4.1
8
Figure 2: Comparison of CoRE against existing code reasoning benchmarks. (Left) Distribution of Average Cyclomatic Complexity and Average Number of Test Cases. (Right) Comparison of code reasoning benchmarks. Samples: Total number of samples for inference. Tests: Average number of test cases per instance. Impl.: Average number of diverse code implementations per coding problem. J : Average Jaccard similarity between diverse implementations for each coding problem, where lower indicates higher diversity. CC: Average Cyclomatic Complexity of code implementations. Inter.: Average number of intermediate state probes per coding problem.
In light of these limitations, we introduce CoRE, a Code Reasoning benchmark that jointly evaluates implementation invariance and process transparency in code reasoning. CoRE is constructed from 60 coding problems from HumanEval (Chen et al., 2021) and LiveCodeBench (Jain et al., 2024), selected for their diverse test cases and algorithmic complexity. For each problem, CoRE evaluates whether LLMs produce consistent predictions across diverse but functionally equivalent implementations. As illustrated in Fig.1(b), we leverage implementations generated by the various LLMs listed in Tab.3 to quantify implementation invariance. Beyond this, CoRE further evaluates process transparency by examining the model’s capacity to reason about intermediate execution states. To this end, we utilize verified codebases and employ five LLMs listed in Tab.3 to generate probes targeting intermediate states within nested loops and complex conditional branches. These intermediate state probes cover four empirical reasoning dimensions, Arithmetic, Logic, State, and Boundary, and are designed to test whether LLMs follow faithful step-by-step reasoning rather than relying on superficial heuristics. In total, as shown in Fig.2 and Fig.4, CoRE comprises 255 unique code implementations, with an average of 4.1 implementations per problem for implementation invariance evaluation. Each coding problem contains 7.8 test cases on average, and each implementation exhibits an average cyclomatic complexity of 5.0, as shown in Fig.2. For process transparency, each problem includes an average of 4.1 intermediate state probes, each covering 2.2 reasoning dimensions on average. Our evaluation of eight frontier LLMs, including GPT-5, o3, Claude-4.5, and DeepSeek-V3.2,
reveals critical limitations in code reasoning. First, we identify a significant Robustness Gap, where performance is inconsistent across syntactically diverse but functionally identical codes. Specifically, we observe that LLMs typically exhibit the highest proficiency with code generated by their own model family, and second-best on code from the OpenAI series, indicating an overfitting where LLMs are biased toward familiar styles. Second, we observe Superficial Execution, where LLMs produce correct function outputs but hallucinate intermediate states, revealing reliance on superficial heuristics rather than genuine execution understanding. Overall, our contributions are threefold: (1) We propose CoRE, a challenging benchmark designed to rigorously assess code reasoning through implementation invariance and process transparency. (2) We reveal that LLMs frequently fail due to stylistic overfitting and superficial execution. (3) We introduce a holistic evaluation protocol that exposes the fragility of current LLMs, demonstrating that genuine execution understanding lags significantly behind output prediction.
2
Related Studies
Code reasoning tasks have emerged as a rigorous method to assess execution understanding in LLMs (Liu et al., 2024b; Gao et al., 2025b; Zhao et al., 2025). Although benchmarks like CruxEval (Gu et al., 2024) and LiveCodeBench-O (Jain et al., 2024) establish a foundation for this domain, they prioritize output prediction and effectively treat the reasoning mechanism as a black box. The reliance on single canonical implementations and low-complexity code makes it difficult
Calculate the sum of squares of even numbers in an array.
def sum_even_arr(arr): return sum([ x**2 for x in arr if x % 2 == 0]) def sum_even_arr(arr, idx): if idx == len(arr):return 0 elif arr[idx] % 2 == 0: return arr[idx] ** 2 + \ sum_even_arr(arr, idx+1) else: return sum_even_arr(arr, idx+1)
Arithmetic
Execution Verification
Lexical Similarity Deduplication
Expert Verification
Implementation Cleaning Process
Problem Description
def sum_even_arr(arr): res = 0 for i in arr if i and i % 2 == 0: res += i**2 return res
What is the exact value of `idx` at Step n?
def sum_even_arr(arr): res = 0 for i in arr: if i and i % 2 == 0: res += i**2 return res
{"i": 2, "res": 4}
{"i": 4, "res": 20}
Logic
{"i": 6, "res": 56}
Does this condition def sum_even_arr(arr): evaluate to True or False? res = 0 for i in arr: if i and i % 2 == 0: res += i**2 State return res What is the cumulative value of res after exiting the loop?
Boundary What is the value of res after exiting the loop?
I. Constructing Diverse Code Implementations
II. Formulating Intermediate State Probes
Figure 3: The construction pipeline of CoRE. It consists of two stages: (I) Constructing Diverse Code Implementations, which produces functionally equivalent code via massive LLMs, and validates its functionality and diversity, and (II) Formulating Intermediate State Probes, where LLMs are employed to synthesize probes grounded in captured execution traces across four dimensions: Arithmetic, Logic, State, and Boundary.
to distinguish genuine execution simulation from heuristic input-output mapping. This distinction is vital given that LLM reasoning remains fragile and often unfaithful. Even with Chain-of-Thought prompting (Wei et al., 2022), models frequently generate plausible yet hallucinated traces derived from shallow heuristics rather than actual logic reasoning (Liu et al., 2024b; Beger and Dutta, 2025; Gao et al., 2025c; Lanham et al., 2023; Turpin et al., 2023; Wang et al., 2024; Gao et al., 2025a; Ji et al., 2025; Zhou et al., 2024, 2026b, 2025). Such unreliability necessitates validating intermediate states instead of relying solely on terminal outputs (Uesato et al., 2022; Lightman et al., 2023). While REval (Chen et al., 2025) attempts to incorporate intermediate state prediction, its dependence on rigid templates and simple canonical code restricts its ability to evaluate reasoning robustness across varied coding styles. Current methods fail to verify if LLMs maintain consistency when processing functionally equivalent but syntactically diverse solutions. CoRE addresses these limitations by enforcing implementation invariance and process transparency to provide a strict evaluation of code reasoning.
3
CoRE Benchmark
In this section, we detail the construction process of CoRE, which aims to evaluate both the implementation invariance and process transparency of LLMs in code reasoning. The construction pipeline is shown in Fig.3, which consists of two stages. The first stage constructs diverse code implementations, and the second stage formulates intermediate state probes.
3.1
Constructing Diverse Code Implementations
To establish a high-quality foundation for code reasoning, we first collect an initial coding problem set from HumanEval (Chen et al., 2021) and LiveCodeBench (Jain et al., 2024). These benchmarks are selected for their rigorous coverage of test cases and algorithmic complexity, which can be used for evaluating code reasoning. We initially aggregate a pool of problems P comprising 164 coding problems from HumanEval and 880 from LiveCodeBench. As illustrated in first stage of our construction workflow in Fig.3, we then leverage seven diverse LLMs M to generate potential solutions, formulating a comprehensive pool S containing 7,308 distinct code implementations for 1,044 unique problems, defined as: S = {s | s ∼ m(p), ∀p ∈ P, ∀m ∈ M}.
(1)
To ensure that the benchmark effectively evaluates code reasoning, we apply a hierarchical implementation cleaning process. We first perform execution validation to ensure functional equivalence, retaining only the solutions that pass their corresponding test sets TP . We further refine this set to guarantee reasoning depth and lexical variety. Specifically, an instance s is retained if and only if its cyclomatic complexity C(s) exceeds a threshold τcc and its 1-gram Jaccard similarity J with other implementations sharing the same problem ID remains below τsim . The 1-gram Jaccard similarity (Wikipedia, 2026) are formulated as: J (s, t) =
|G1 (s) ∩ G1 (t)| , |G1 (s) ∪ G1 (t)|
(2)
where G1 (s) denotes the set of unique 1-grams (tokens) in code implementation s. Overall, the im-
The Four Dimensions of Intermediate Execution Probing
Aspect
1. ARITHMETIC (Numerical Precision)
2. LOGIC (Nested Conditions)
Targeting mixed precedence & complex calculations.
Resolving nested booleans & control flow.
idx = (base) + I * step % limit
3. STATE (Deep Accumulation) Monitoring variable history at deep depth. [Iteration 50/100] running_total += current_value
Probe Focus: What is the exact cumulative value at this specific step?
Basic Statistic
60 255 243 1978
Impl. Statistic
OpenAI DeepSeek Gemini Claude Qwen
74 68 51 41 21
Probe Statistic
Arithmetic Logic State Boundary Avg. Dimensions
77 178 185 97 2.2
if (not valid and retry > max) or force
Probe Focus: What is the precise integer value of idx?
Probe Focus: Does the condition evaluate to True or False? 4. BOUNDARY (Loop Transitions) Inspecting critical loop termination points. nums = [2, 3, 4, 5] count = 0 for num in nums:
if num < 3: count += 1 else: break
Probe Focus: At the execution of `break`, what is the final value of variable `count`?
Number
Code Problems Code Candidates Intermediate State Probes Samples
Figure 4: The taxonomy and distribution of Intermediate Probing dimensions. The left panel illustrates the four dimensions of execution probing. The right table summarizes the dataset composition. The middle section details the distribution of code candidates generated by various LLMs, and the bottom specifies the number of intermediate probing questions assigned to each reasoning dimension.
plementation cleaning process is defined as: C(s) ≥ τcc ′ S = s ∧ max J (s, t) ≤ τsim , t∈Dio \{s}
(3)
id(s)=id(t)
Recognizing that evaluating LLMs on such diverse candidates with massive test suites presents significant computational challenges, we optimize evaluation efficiency by computing TP ∗ , the minimal subset of test cases achieving maximal coverage for problem P. This reduction strategy reduces the average number of test cases per instance from 24.1 to 7.8 without compromising coverage integrity. Therefore, the larger number of test cases reported in Fig.2 is not a superficial increase in quantity, but reflects test cases that are necessary to ensure adequate coverage. Finally, human expert verification ensures implementation diversity, producing 255 curated implementations across 60 problems, as shown in Fig.2. 3.2
Formulating Intermediate State Probes
To formulate intermediate state probes, we first employ Python execution tracing to capture the runtime values of all variables, as illustrated in the middle of Fig.3, serving as ground truth for process transparency evaluation. Then, we employ an ensemble of LLMs to automatically produce intermediate state probes targeting four critical dimensions of program behavior: Arithmetic, Logic, State, and Boundary. Specifically, the Arithmetic dimension tracks numerical precision in composite operations and complex indexing, e.g., res = (a + b) * c % d. The Logic dimension evaluates the resolution
of compound boolean conditions and nested control flow decisions. To ensure reasoning depth, the State dimension monitors variable histories across long execution paths, whereas the Boundary dimension focuses on critical transition points, such as loop terminations. Notably, to increase challenge and reasoning depth, we ask LLMs to prioritize probes that span multiple dimensions, for example, tracking state accumulation within nested logical branches. Finally, human experts validate the intermediate probes for logical correctness, diversity, and complexity to ensure the rigor of CoRE. This process yields an intermediate state probe set for each problem P, denoted as QP , which contains an average of 4.1 probes, with each probe spanning 2.2 dimensions on average. 3.3
Benchmark Statistics Analysis
Basic statistics. CoRE expands existing benchmarks by an order of magnitude with 1,978 samples. It raises the evaluation complexity, with an average cyclomatic complexity of 5.0 and an average of 7.8 test cases per coding problem, markedly surpassing existing benchmarks, as shown in Fig.2. Implementation Invariance. As detailed in the basic statistic in Fig.4, CoRE consists of 60 coding problems, which contain a total of 255 code candidates, namely, an average of 4.3 implementations per coding problem. The diversity of the dataset is demonstrated in the Impl. Statistics in Fig.4, comprising 74 from OpenAI, 51 from Gemini, 68 from DeepSeek, 41 from Claude, and 21 from Qwen. The inner diversity of implementations within each coding problem is evidenced by a low 1-gram Jaccard similarity score of J = 0.6
as listed in Fig.2. Process Transparency. To evaluate LLMs in Process Transparency, we also formulate intermediate probes for each coding problem across four different dimensions. As shown in probe statistic in Fig.4, 77 probes involving Arithmetic, 178 involving Logic, 185 involving State, and 97 involving Boundary. These dimensions are not mutually exclusive, and they are often combined. For example, the State dimension is frequently interacted with others as illustrated in Fig.7. On average, each individual probe covers 2.2 dimensions, ensuring that the evaluation effectively challenges LLMs’ process reasoning.
4
Evaluation Protocol
Unlike traditional evaluations, which typically focus on the output accuracy of a single canonical implementation, our protocol provides a holistic assessment of code reasoning by leveraging the diverse implementations SP and intermediate probes QP constructed in Sec.3.1 and Sec.3.2. 4.1
Preliminaries
We first recall some background in code reasoning evaluation in this section. Standard Code Reasoning. Traditional benchmarks (Chen et al., 2021; Jain et al., 2024) primarily focus on output prediction using a canonical Implementation c. This approach treats the reasoning process as a black box, where the model is evaluated on its ability to map an input x to the execution result ŷ for the given c: ŷ = Pred(c, x),
(4)
where ŷ is expect to match the ground truth y. Intermediate State Reasoning. Benchmarks such as REval (Chen et al., 2025) frame intermediatestate reasoning as a question-answering task, covering path reachability and variable prediction. Formally, given a code implementation c, a intermediate probes p, and a specific test input x, the model predicts the intermediate state as: â = Pred(p, c, x),
(5)
where â is the predicted answer to the intermediate probes p. 4.2
Implementation Invariance
A robust model should maintain consistency across diverse yet functionally equivalent implementa-
tions. For a given problem P, we define its diverse code implementations in CoRE as a set SP = {c1 , c2 , ..., cn }. Implementation invariance is quantified via the Strict Output Consistency I, a binary indicator that is satisfied only if the model M correctly predicts the final output for every candidate solution across all associated test cases TP : ( 1, ∀ci ∈ SP , M(ci , TP x ) = TP y I(P) = 0, otherwise. (6) Additionally, we also define the soft accuracy Is across multiple code implementations within each question P: P x y ci ∈SP M(ci , TP ) == TP Is (P) = (7) |SP | 4.3
Process Transparency
Moreover, reliable LLMs are expected to effectively demonstrate process transparency in code reasoning rather than simple input-output mapping. We define QP = {(q1 , a1 ), (q2 , a2 ), . . . , (qm , am )} as the probing question set of question P. Process transparency of code reasoning is calculated by Process Fidelity Weight Ws , the ratio of probing questions correctly answered by the model M: P (qi ,ai )∈QP ⊮(M(qi , P) = ai ) Ws (P) = , (8) |QP | where ⊮(·) is the indicator function, and |QP | represents the total number of probes for the problem P. Additionally, we define a strict fidelity indicator W ∈ {0, 1}, where W = 1 if and only if all intermediate probing questions in QP are answered correctly, i.e., Ws = 1, and W = 0 otherwise. 4.4
Reasoning Consistency Score
To rigorously quantify the robustness of code reasoning and penalize superficial heuristics, we propose the Reasoning Consistency Score (RCS). The final RCS for problem P is computed as the product of the strict output consistency and the process fidelity weight: RCS(P) = I × Ws
(9)
This formulation ensures that any failure in implementation invariance nullifies the final score, while scaling that score by the depth of its internal execution probing. By coupling these two dimensions,
LLMs. We evaluate the code reasoning capabilities of eight leading LLMs, including GPT5 (OpenAI, 2025a), o3 (OpenAI, 2025b), Claude4.5 (Anthropic, 2025), DeepSeek-V3.2 (Liu et al., 2025), DeepSeek-R1 (Guo et al., 2025), and Qwen3 (Yang et al., 2025). Prompting Methods. To provide a comprehensive assessment, we employ diverse prompting strategies including standard Input-Output (IO), Chainof-Thought (CoT) (Wei et al., 2022), and Chainof-Code (CoC) (Li et al.). We also evaluate the RHDA framework (Zhao et al., 2025) as a strong reasoning baseline. In this setting, RHDA-1 and RHDA-2 correspond to the framework configured with one and two iterations, respectively. Details of the prompts and postprocessing are provided in Appendix B. Metrics. We report the metrics defined in Sec. 4, including Strict Output Consistency (I), soft accuracy across implementations (Is ), Process Fidelity Weight (Ws ), and strict process consistency (W ). Our primary metric is the Reasoning Consistency Score (RCS), which assesses whether correct outputs are supported by consistent intermediateexecution reasoning across functionally equivalent implementations. We additionally report Cons., defined as 1−MSE(I, Ws ), indicating how closely final output performance matches intermediate probe performance. All experiments are performed in three times, and the results are presented as the average. 5.2
Results
Tab.1 presents a comprehensive comparison of code reasoning performance across various LLM series and prompting methods. We observe a significant performance gap, characterized by a disconnect between the reasoning step and the final prediction. For instance, in the IO setting, GPT-5 achieves a high strict output accuracy I of 84.62 but a low process fidelity weight Ws of 18.96. This discrepancy indicates that models often hallucinate intermediate states despite generating correct final answers, a phenomenon we term superficial execution. Incorporating CoT substantially mitigates this
Qwen
OpenAI
Experimental Setup
Claude
98.18
88.50
91.95
91.15
71.96
DeepSeek
5.1
Gemini
97.43
91.54
88.65
90.77
76.23
Gemini
Experiments
DeepSeek
97.10
85.10
93.38
87.76
68.60
Claude
5
OpenAI
97.16
87.00
88.97
94.12
70.12
Qwen
the RCS effectively filters out shortcut reasoning, distinguishing genuine code reasoning from a reliance on superficial heuristics.
96.80
87.44
89.27
85.84
76.26
Rank 1 Score
Rank 2 Score
Figure 5: Is scores performance heatmap across five LLM families. Red and gold boxes indicate the rank1 and rank-2 Is scores. The pronounced diagonal pattern indicates a strong style bias.
issue, increasing Ws to 64.93 for GPT-5, which demonstrates that explicitly reasoning improves process fidelity. Regarding the reflection-based RHDA framework, implementation invariance is consistently enhanced, whereas gains in process transparency remain limited. RHDA-2 even underperforms RHDA-1 in some cases. This behavior reflects the inherent fragility of LLM reasoning. When feedback is derived solely from outputs, surface-level discrepancies in results can influence previous reasoning steps, leading models to erroneously revise previously correct intermediate states. Additionally, a performance gap between I and Is implicitly suggests that models struggle to maintain consistency across different implementations, revealing a potential robustness gap. A detailed analysis of this phenomenon is provided in Sec. 6.1.
6
Analysis
To gain deeper insights into reasoning behavior in CoRE, we analyze several factors that influence their code reasoning performance. 6.1
Robustness Gap and Style Overfitting
To examine the detailed implementation invariance of LLMs in CoRE, we analyze their performance consistency across diverse LLM families. Fig.5
Impl. Invar. I Is OpenAI Series Method
Proc. Trans. W Ws
Overall RCS Cons.
Impl. Invar. I Is
Proc. Trans. W Ws
Overall RCS Cons.
IO 84.62 CoT 87.91 CoC 86.81 RHDA-1 86.81 RHDA-2 86.81 DeepSeek Series
95.44 96.77 96.04 95.84 95.83
GPT-5 0.00 18.96 26.37 64.93 20.88 59.14 19.78 59.49 30.77 68.92
15.93 55.75 50.24 51.70 58.48
0.15 0.27 0.30 0.29 0.33
83.52 82.42 87.91 82.42 85.71
95.74 94.74 97.05 94.45 95.75
0.00 28.57 31.87 31.87 29.67
9.73 67.60 67.55 70.46 68.28
8.90 54.25 58.24 58.32 59.16
0.16 0.31 0.35 0.36 0.33
IO 56.04 CoT 64.84 CoC 60.44 RHDA-1 62.64 RHDA-2 71.43 Claude Series
88.04 90.38 88.13 88.50 91.18
DeepSeek-V3.2 29.67 67.95 40.49 25.27 64.76 41.94 26.37 66.47 40.40 27.47 66.67 41.48 30.77 67.07 48.70
0.56 0.41 0.48 0.45 0.42
32.97 34.07 50.55 70.33 72.53
84.43 86.19 90.01 90.85 91.51
DeepSeek-R1 24.18 62.93 20.88 63.26 21.98 62.60 25.27 65.29 24.18 64.49
20.60 18.52 32.36 44.34 47.16
0.58 0.52 0.54 0.37 0.38
Claude-4.5 25.27 62.40 27.47 64.23 23.08 64.78 28.57 65.71 29.67 68.74
39.45 43.90 39.78 44.34 54.45
0.37 0.37 0.44 0.43 0.29
12.09 56.04 48.35 56.04 64.84
67.17 87.38 86.27 86.04 88.75
Claude-3.7 20.88 61.90 18.68 63.00 27.47 64.76 24.18 65.15 21.98 64.32
6.81 37.53 31.48 39.73 42.77
0.69 0.49 0.53 0.53 0.44
Gemini-2.5 1.10 27.73 30.77 64.74 18.68 59.58 21.98 62.88 29.67 64.54
12.11 33.13 23.41 38.90 44.25
0.57 0.51 0.58 0.47 0.38
5.49 12.09 6.59 16.48 47.25
53.02 78.07 71.75 76.39 84.22
Qwen-3 0.00 9.38 0.00 18.90 0.00 17.53 0.00 23.99 0.00 22.20
0.27 2.69 1.26 4.18 10.66
0.95 0.88 0.93 0.84 0.53
IO 65.93 89.75 CoT 68.13 88.50 CoC 61.54 88.05 RHDA-1 65.93 89.39 RHDA-2 81.32 93.71 Gemini & Qwen Series IO CoT CoC RHDA-1 RHDA-2
41.76 51.65 38.46 61.54 71.43
86.83 88.04 87.15 87.82 91.78
o3
Table 1: Code Reasoning performance of baselines on the CoRE benchmark. The best performance is bolded and the second best is underline. Across all frontier LLMs, models exhibit preferences over different code implementations, and frequently produce correct outputs despite incorrect intermediate execution states.
shows a heatmap of evaluation scores with a clear diagonal dominance, indicating that models typically exhibit the highest proficiency with code generated by their own model family. For example, OpenAI models score 98.18 on OpenAI-generated implementations, noticeably higher than on code from other families. This pattern suggests that current LLMs are strongly influenced by familiar implementation styles, indicating a tendency toward stylistic overfitting. The lexical diversity of implementations, evidenced by a low 1-gram Jaccard similarity of J = 0.6, further highlights that surface-level differences can trigger this inconsistent reasoning. 6.2
Investigating Superficial Execution
A central contribution of the CoRE benchmark is the identification of superficial execution, defined as the phenomenon where models arrive at correct
final outputs without accurately reasoning about intermediate states. For instance, under the native IO prompting, GPT-5 achieves a high strict output accuracy I = 84.62 but a remarkably low process fidelity weight Ws of only 18.96. This discrepancy yields a poor RCS score of 15.93, as it penalizes such heuristic reasoning, proving that RCS effectively distinguishes genuine code reasoning from superficial execution. 6.3
Analysis of Dimensional Challenges
To analyze how four dimensions challenge LLMs in process transparency evaluation, we report model performance across Arithmetic, Logic, State, and Boundary in Tab.2. The Arithmetic dimension exposes a severe lack of numerical precision. Under IO prompting, GPT-5 and o3 achieve WS scores of only 11.11 and 9.26, respectively. This suggests that native models often bypass exact arith-
IO
Model
CoT
CoC
A
L
S
B
A
L
S
B
A
L
S
B
GPT-5 o3 Claude-4.5 Claude-3.7 DeepSeek-V3.2 DeepSeek-R1 Gemini-2.5 Qwen-3
11.11 9.26 68.52 75.93 79.63 75.93 24.07 11.11
25.00 10.48 72.58 67.74 71.77 69.35 32.26 10.48
20.78 11.69 72.73 72.08 76.62 71.43 31.82 11.04
21.33 17.33 78.67 72.00 77.33 68.00 30.67 12.00
68.52 77.78 72.22 77.78 74.07 72.22 77.78 18.52
74.19 75.00 71.77 70.16 70.97 72.58 69.35 24.19
77.27 80.52 74.68 73.38 74.03 72.73 72.08 18.18
76.00 81.33 78.67 72.00 76.00 77.33 77.33 16.00
74.07 77.78 70.37 75.93 74.07 75.93 70.37 27.78
63.71 73.39 72.58 73.39 71.77 67.74 67.74 22.58
68.83 77.92 74.68 74.68 77.27 70.13 71.43 20.78
70.67 78.67 78.67 73.33 80.00 73.33 72.00 18.67
Average
44.45
44.96
46.02
47.17
67.36
66.03
67.86
69.33
68.29
64.11
66.97
68.17
Exposure
Table 2: Comparison of model performance across four problem complexity levels (A: Arithmetic, L: Logical, S: State, B: Boundary). The process fidelity weight Ws is reported. The best performance is bolded and the second best is underline. Overall, CoT and CoC substantially improve intermediate probe performance across all complexity dimensions compared to direct IO.
1.0 0.8 0.6 0.4 0.2 0.0
6.4
10th - 90th Percentile Range Mean Exposure Trend
0
1
2 3 4 5 6 7 Number of Test Cases
8
Figure 6: Exposure trend across increasing test cases. The solid blue line denotes the mean exposure ratio, while the shaded region represents the 10 th to 90th percentile distribution. The model’s performance shows initial variability before converging to saturation as the number of test cases increases.
metic. Structural fragility is evident in the limited performance of Logic and Boundary dimensions, which yield average Ws of 44.96 and 47.17, respectively. These results indicate that models often struggle to handle nested conditions and critical transition points. Furthermore, the State dimension, which monitors variable histories across deep execution paths, shows an average WS of only 46.02, revealing an inability to maintain a persistent internal memory of variable updates. This poor performance stems from the fact that the State dimension frequently interacts with the other three dimensions as illustrated in Fig.7, exposing their fundamental inability to maintain a persistent memory of variable updates.
Impact of Prompting Method
Although CoT, CoC, and RHDA enhance implementation invariance and process transparency evaluation, they still fail to address robustness gaps and superficial execution thoroughly. Specifically, considering RHDA is an iterative reflection framework, we investigate why RHDA-2 sometimes underperforms RHDA-1 as observed in Tab.1. We find that when a correct final output is mistakenly judged as incorrect in RHDA, models readily revise previously correct intermediate reasoning simply in response to this erroneous feedback, highlighting the existence of superficial execution. 6.5
Soundness of Test Cases
To improve computational efficiency, we employ a coverage-based reduction strategy to identify the minimal subset of test cases that maintains maximal execution coverage. This optimization results in an average of 7.8 test cases per instance, which significantly raises the evaluation ceiling compared to prior benchmarks. The exposure trend illustrated in Fig.6 demonstrates that model performance initially varies but eventually converges to saturation as the number of test cases increases.
7
Conclusion
We introduce CoRE, a benchmark designed to evaluate code reasoning through implementation invariance and process transparency. Our evaluation of eight leading LLMs identifies a pronounced robustness gap and the phenomenon of superficial execution in existing frontier LLMs. To quantify this, we propose RCS, a composite protocol that
penalizes heuristic shortcuts and rewards genuine code reasoning. Our experimental findings highlight the fragility of current LLMs, underscoring the critical role of CoRE in facilitating a deeper exploration for code reasoning.
8
Limitations
Despite its rigor, the CoRE benchmark has several limitations that provide directions for future work. First, our current implementation and execution behavior tracing are primarily focused on the Python programming language. While CoRE derives instances from well-established benchmarks like HumanEval and LiveCodeBench, these source datasets may still be susceptible to potential training data leakage. Second, although we incorporate a human verification stage to ensure logical validity, the reliance on expert verification limits the rapid and automated scaling of the dataset. Finally, the distribution of probing questions across the four dimensions, Arithmetic, Logic, State, and Boundary, is not perfectly uniform due to the inherent complexity of intermediate execution.
9
Acknowledgements
This project is supported by the National Natural Science Foundation of China (No. 62302437) , and Yongjiang Talent Program (No. 2023A-402-G).
References Anthropic. 2025. Introducing claude sonnet 4.5. Technical report. Claas Beger and Saikat Dutta. 2025. Coconut: Structural code understanding does not fall out of a tree. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 128– 136. IEEE. Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llmbased agent for program repair. arXiv preprint arXiv:2403.17134. Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025. Reasoning runtime behavior of a program with llm: How far are we? In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 1869–1881. IEEE. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
Jun Gao, Yongqi Li, Ziqiang Cao, and Wenjie Li. 2025a. Interleaved-modal chain-of-thought. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19520–19529. Jun Gao, Yun Peng, and Xiaoxue Ren. 2025b. \texttt {ReMind}: Understanding deductive code reasoning in llms. arXiv preprint arXiv:2511.00488. Jun Gao, Qian Qiao, Tianxiang Wu, Zili Wang, Ziqiang Cao, and Wenjie Li. 2025c. Aim: Let any multimodal large language models embrace efficient in-context learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 3077–3085. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. oppt-4o system card. arXiv preprint arXiv:2410.21276. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando SolarLezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974. Yicheng Ji, Jun Zhang, Heming Xia, Jinpeng Chen, Lidan Shou, Gang Chen, and Huan Li. 2025. Specvlm: Enhancing speculative decoding of video llms via verifier-guided token pruning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7216–7230. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, and 1 others. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702. Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li FeiFei, Fei Xia, and 1 others. Chain of code: Reasoning
with a language model-augmented code emulator. In Forty-first International Conference on Machine Learning. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. In The Twelfth International Conference on Learning Representations. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. Changshu Liu, Shizhuo Dylan Zhang, Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand. 2024b. Codemind: A framework to challenge large language models for code reasoning. arXiv e-prints, pages arXiv–2402. OpenAI. 2025a. Gpt-5 system card: Capabilities and safety evaluations. Technical report. OpenAI. 2025b. Openai o3 and o4-mini system card. Technical report. Zhensu Sun, Xiaoning Du, Fu Song, Shangwen Wang, Mingze Ni, Li Li, and David Lo. 2025. Don’t complete it! preventing unhelpful code completion for productive and sustainable neural code completion systems. ACM Transactions on Software Engineering and Methodology, 34(1):1–22.
Wikipedia. 2026. Jaccard index. Technical report. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528– 50652. Yuze Zhao, Tianyun Ji, Wenjun Feng, Zhenya Huang, Qi Liu, Zhiding Liu, Yixiao Ma, Kai Zhang, and Enhong Chen. 2025. Unveiling the magic of code reasoning through hypothesis decomposition and amendment. In The Thirteenth International Conference on Learning Representations. Zhizhou Zhong, Yicheng Ji, Zhe Kong, Yiying Liu, Jiarui Wang, Jiasun Feng, Lupeng Liu, Xiangyi Wang, Yanjia Li, Yuqing She, and 1 others. 2025. Anytalker: Scaling multi-person talking video generation with interactivity refinement. arXiv preprint arXiv:2511.23475. Changhai Zhou, Shiyang Zhang, Yuhua Zhou, Qian Qiao, Jun Gao, Shichao Weng, Weizhong Zhang, and Cheng Jin. 2026a. Balancing fidelity and plasticity: Aligning mixed-precision fine-tuning with linguistic hierarchies. Preprint, arXiv:2505.03802. Changhai Zhou, Yuhua Zhou, Shijie Han, Qian Qiao, and Hongguang Li. 2024. Qpruner: Probabilistic decision quantization for structured pruning in large language models. Preprint, arXiv:2412.11629.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-ofthought prompting. Advances in Neural Information Processing Systems, 36:74952–74965.
Yuhua Zhou, Ruifeng Li, Changhai Zhou, Fei Yang, and Aimin PAN. 2025. BSLoRA: Enhancing the parameter efficiency of loRA with intra-layer and inter-layer sharing. In Proceedings of International Conference on Machine Learning.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcomebased feedback. arXiv preprint arXiv:2211.14275.
Yuhua Zhou, Changhai Zhou, Shiyang Zhang, Fei Yang, Yi Zhang, and Aimin Pan. 2026b. Lara: Layer-wise rank allocation for efficient fine-tuning of pruned large language models. Information Processing & Management, 63(3):104538.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and 1 others. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440–9450. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
31.0%
Code Generation Prompt
60 Percentage (%)
50 40 30 20
17.1% 11.0% 10.0% 5.7% 5.2% 4.8% 3.8% 2.9% 2.9% 1.9% 1.4% 1.4% 1.0%
10 0 25.7%
Arithmetic
35.7%
Boundary
59.0%
Logic
72.9%
### Question: {question_content}
State 100
You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests.
0
Figure 7: The interaction of intermediate probes.
Table 3: Overview of Large Language Models used in different stages of the study. II indicates implementation invariance, and IP represents intermediate probing.
Model GPT-5 o3 o1 4o-mini DeepSeek-V3 DeepSeek-V3.2 DeepSeek-R1 Claude-3.7 Claude-4.5 Qwen-3-235B Gemini-2.5
II
IP
Eval
✓
✓ ✓
✓ ✓ ✓
### Answer: (use the provided format with backticks) A.2
✓ ✓ ✓ ✓ ✓
### Format: You will use the following starter code to write the solution to the problem and enclose your code within delimiters. “‘python {starter_code} “‘
✓ ✓ ✓
✓ ✓ ✓ ✓ ✓ ✓
Details of Intermediate Probing
To alleviate the impact of LLM bias, we employ GPT-5, DeepSeek-V3.2, Claude-4.5, DeepSeekR1, and Qwen-3 in a cycle to formulate probing questions in this stage. We encourage questions to be diverse to challenge LLMs, and we thus slightly increase the temperature to 0.8. The following few-shot examples are provided to the model to demonstrate the desired reasoning depth and output format. Reference Examples for Code Reasoning
A A.1
Benchmark Construction Details Details of Identical Implementation Generation
We leverage an ensemble of state-of-the-art LLMs to generate diverse but functionally equivalent code implementations for problems sourced from HumanEval and LiveCodeBench. As shown in Table 3, seven models, such as o1, 4o-mini, DeepSeekV3, and Qwen-3, were primarily utilized for this stage. During this stage, we set the temperature to 0.7. In the implementation cleaning process, the cyclomatic complexity and Jaccard similarity threshold τcc and τsim are set to 3.0 and 0.7, respectively.
## Example 1 Input Source Code: def u p d a t e _ a r r ( arr , f a c t o r ) : f o r i i n range ( l e n ( a r r ) ) : i f i > 0: arr [ i ] = ( arr [ i ] * f a c t o r + a r r [ i − 1 ] ) % 10 return a r r Input Execution Trace: [Step 1] Line 2: for i in range(len(arr)) (Iter 1: i=0) | locals: ’arr’: [1, 5, 9], ’factor’: 2 ... [Step 5] Line 3: if i > 0 (Evaluated: True) [Step 6] Line 5: arr[i] = (arr[i] * factor + arr[i-1]) % 10 | locals: ’arr’: [1, 1, 9]
Output (JSON): {
}
"target_step_index": 6, "complexity_type": "Arithmetic + State", "question": "In the 2nd iteration (i=1), what is the exact value assigned to 'arr[1]' on line 5?", "ground_truth": "1",
## Example 2 Input Source Code: def matrix_sum ( matrix ) : total = 0 f o r row i n m a t r i x : f o r v a l i n row : i f v a l == −1: break t o t a l += v a l return t o t a l Input Execution Trace: ... [Step 10] Line 3: for row in matrix (Iter 2) | locals: {’row’: [10, -1, 5], ’total’: 6} [Step 11] Line 4: for val in row (Iter 1) | locals: {’val’: 10} [Step 12] Line 5: if val == -1 (Evaluated: False) [Step 13] Line 6: total += val | locals: {’total’: 16} [Step 14] Line 4: for val in row (Iter 2) | locals: {’val’: -1} [Step 15] Line 5: if val == -1 (Evaluated: True) [Step 16] Line 5: break | locals: {’total’: 16} Output (JSON): {
}
"target_step_index": 16, "complexity_type": "Boundary + State", "question": "Given the input row [10, -1, 5], what is the value of variable ' total' immediately after the break statement executes on line 5?", "ground_truth": "16",
Then we combine the above reference examples with the following instruction, providing LLMs with sample questions targeting four dimensions: Arithmetic, Logic, State, and Boundary
Intermediate Probing Prompt # System Instruction You are an expert Code Logic Auditor. Your goal is to create a “Challenging Code Reasoning Benchmark” by analyzing execution traces. ## Mission: Identify the most error-prone step in the provided execution trace and formulate a question about it. We want to test the model’s ability to handle complexity. ## Selection Criteria: 1. Complex Arithmetic: Lines with multiple operators (e.g., res = (a + b) * c % d) or list indexing with math. 2. Loop Boundaries: The last iteration of a loop, or the state immediately after a loop finishes. 3. Nested Logic: Steps inside a nested loop or a nested if block where context is deep. 4. Compound Conditions: Boolean evaluations involving and, or, not. 5. State Accumulation: A variable modified multiple times (e.g., total after the 5th iteration). ## Rules: • Answer Integrity: The ‘ground_truth‘ must be extracted EXACTLY from the provided trace. Do not compute it yourself. • Precision: The question must specify the exact context (e.g., "At the end of the 3rd iteration...", "In the evaluation of the condition on line 5..."). # Few-Shot Examples 〈Reference Examples from Appendix A.2 are inserted here〉 # Current Task Input Source Code: “‘python {source_code}
“‘ Input Execution Trace: {execution_trace} A.3
Details of Human Verification
To ensure the logical validity and diversity of the benchmark, five experts executed a multi-stage verification protocol. During the implementation variance phase, the experts manually refined samples with a Jaccard similarity within the range of [0.7, 0.9] to recover implementations that exhibited high 1-gram similarity but featured distinct structural logic or algorithmic strategies. Regarding intermediate reasoning traces, the experts evaluated logical correctness and the rationality of state transitions while simultaneously removing redundant questions to maximize diversity. This rigorous calibration process required approximately 110 human-hours and resulted in a finalized pool of 255 expert-verified implementations and 243 intermediate questions across 60 unique questions. The effort achieved high inter-annotator agreement with Cohen’s Kappa κ = 0.82 for the implementation variance phase and κ = 0.86 for the intermediate probing phase.
a s s e r t f ( " x 9 j " ) == ? ? # Answer : a s s e r t f ( " x 9 j " ) == " x 9 j a " { code } a s s e r t { func_name } ( { x } ) == ? ? # Answer :
Chain-of-Thought Prompt You are given a Python function and an assertion containing an input to the function. Complete the assertion with a literal (no unsimplified expressions, no function calls) containing the output when executing the provided code on the given input, even if the function is incorrect or incomplete. Execute the program step by step before arriving at an answer, and provide the full assertion with the correct output, following the examples. def f ( s ) : s = s + s return "b" + s + " a " a s s e r t f ( " h i " ) == ? ? Let’s execute the code step by step:
B
Details of Experimental Baselines
B.1
Prompting Methods
We evaluated the models using three primary strategies: Input-Output (IO), Chain-of-Thought (CoT), and Chain-of-Code (CoC). For CoT, we specifically implemented a two-shot approach to standardize the reasoning "scratchpad" across all frontier models. Standard Output Prompt You are given a Python function and an assertion containing an input to the function. Complete the assertion with a literal (no unsimplified expressions, no function calls) containing the output when executing the provided code on the given input, even if the function is incorrect or incomplete. Provide the full assertion with the correct output, following the examples. def f ( s ) : return s + " a "
• 1. The function f is defined, which takes a single argument s. • 2. The function is called with the argument "hi", so within the function, s is initially "hi". • 3. Inside the function, s is concatenated with itself, so s becomes "hihi". • 4. The function then returns a new string that starts with "b", followed by the value of s (which is now "hihi"), and ends with "a". • 5. The return value of the function is therefore "bhihia". Answer : a s s e r t f ( " h i " ) == " b h i h i a " { code } a s s e r t { func_name } ( { x } ) == ? ? Let’s execute the code step by step:
Chain-of-Code Prompt You are given a Python function and an assertion containing an input to the function. Complete the assertion with a literal (no unsimplified expressions, no function calls) containing the output when executing the provided code on the given input, even if the function is incorrect or incomplete. Execute the program step by step before arriving at an answer, and provide the full assertion with the correct output, following the examples. def f ( s ) : s = s + s r e s u l t = "b" + s + "a" return r e s u l t a s s e r t f ( " h i " ) == ? ? [TRACE] state: {} line: f("hi") explanation: Python execution. delta state: ’s’: ’hi’ line: s = s + s explanation: Python execution. delta state: ’s’: ’hihi’ line: result = "b" + s + "a" explanation: Python execution. delta state: ’result’: ’bhihia’ line: return result explanation: Python execution. delta state: [/TRACE] Answer: a s s e r t f ( " h i " ) == " b h i h i a " { code } a s s e r t { func_name } ( { x } ) == ? ? [TRACE]