arXiv CSopen access

Do Machines Struggle Where Humans Do? LLM and Human Comprehension of Obfuscated Code

Do Machines Struggle Where Humans Do? LLM and Human Comprehension of Obfuscated Code Jack Le1,* , Anh H.N. Nguyen1,* , and Tien N. Nguyen1

arXiv:2606.31725v1 [cs.SE] 30 Jun 2026

Department of Computer Science University of Texas at Dallas Richardson, Texas, USA 1 {jvl210002, Tien.N.Nguyen}@utdallas.edu

different LLM classes mirror human difficulty patterns, whether reasoning-tuned models are more aligned than coder or instruction-tuned models, and whether models reproduce the specific failure modes obfuscation induces in humans: effort that scales with task difficulty, and confident misinterpretation under misleading identifiers. Together, these questions move beyond aggregate accuracy to examine the nature of model understanding under controlled distortion. To answer them, we conduct a mixed empirical study combining model evaluation and comparison with prior human results [1]. We evaluate several LLMs on obfuscated versions of two datasets in Python and JavaScript across five obfuscation tiers, including identifier renaming, adversarial renaming, control-flow flattening, and their combination. We analyze both output prediction performance and reasoning traces, and we interpret failures through Schulte’s Block Model of program comprehension [2], which separates atom-level, block-level, I. I NTRODUCTION relational, and macro-level understanding. This enables direct Code obfuscation preserves program semantics while de- comparison between model behavior and previously reported liberately making code harder to understand. It is widely human difficulty patterns under the same obfuscation types [1]. used to protect software from reverse engineering, tampering, This paper makes the following contributions: and unauthorized reuse, but it also provides a useful lens for • We formulate the problem of measuring human–machine studying program comprehension itself. Because obfuscation alignment in obfuscated-code understanding, following removes or distorts familiar lexical and structural cues without Nguyen et al. [1] as the human baseline for our evaluation. changing program behavior, it creates a controlled setting • We conduct a systematic study of LLMs across multiple for testing whether an agent truly understands program logic obfuscation tiers in Python and JavaScript, comparing their or instead relies on patterns such as meaningful identifiers, behavior against the reported human difficulty patterns. common idioms, or recognizable control-flow structures. • We analyze model failures using the Block Model and reaNguyen et al. [1] showed that obfuscation disrupts the soning traces, showing where alignment emerges and where cues programmers use to form high-level semantic hypotheses, machine understanding diverges from human understanding. increases cognitive effort, and forces more deliberate reasoning. • We report the findings and implications for future studies in This raises a fundamental question for code LLMs: do machines both software engineering and software security. understand obfuscated code in ways that align with human II. BACKGROUND understanding, or do they fail differently? If they align, obfuscated code becomes a principled benchmark for human-like A. Code Obfuscation reasoning, and robustness would signal genuine comprehension. Prior work categorizes obfuscation techniques by transforIf they diverge, high benchmark performance may mask a brittle mation properties, most notably distinguishing layout-based competence that succeeds on standard tasks yet breaks under from control-flow-based obfuscations [3]. Identifier renaming transformations humans can still reason through. is a layout-based obfuscation, while control-flow flattening is a Motivated by this, we study the alignment between human control-flow-based obfuscation [4]. We adopt these obfuscation and machine understanding of obfuscated code, asking whether levels to align with Nguyen et al. [1]’s human study on code *Jack Van Le and Anh H.N. Nguyen contributed equally to this work. understandability under obfuscation.

Abstract—While code obfuscation impairs human code comprehension, it remains unclear if large language models share these failure modes. Building directly on a recent human study of program comprehension under code obfuscation [1], we evaluate whether large language models share the failure modes that obfuscation induces in human programmers. Evaluating several LLMs with five obfuscation tiers using the Block Model, we localize comprehension failures at the atom, block, relational, and macro levels. We find that reasoning-tuned models demonstrate significant alignment with human difficulty patterns across experience levels, whereas instruction and coder-tuned models show near-zero correlation. Chain-of-Thought trace length tracks task difficulty across tasks. Results indicate that performance under control-flow flattening degrades in proportion to statespace complexity, while adversarial identifier renaming disrupts comprehension through the interaction of semantic displacement and identifier-level interference. These findings suggest that reasoning-tuned LLMs approximate human sensitivity to code complexity more effectively than instruction-tuned variants.

elements), Blocks (semantically meaningful regions), Relations (dependencies and data flow between blocks), and Macro Structure (the overall algorithm and purpose). We use output prediction as a proxy for understanding as in Nguyen et al. [1]. The Block Model allows us to localize where comprehension breaks down and connect each failure mode to a specific structural property of obfuscated code.

Fig. 1: Block Model Schema L1: Identifier Renaming. Identifier renaming is a layoutbased obfuscation that replaces meaningful identifiers (e.g., function/variable names) with short, incoherent, or minimally informative names. This disrupts the semantic cues normally provided by identifiers, making it harder to understand. L1b: Adversarial Renaming. We use L1b, a variant of L1 to capture a distinct effect of identifier obfuscation. While L1 removes semantic cues, L1b replaces identifiers with semantically meaningful but misleading names in the new context. This semantic mismatch can induce incorrect mental models of the code, leading to high-confidence but erroneous interpretations. L2: Control-Flow Alteration. This obfuscates the execution by decoupling control logic from its original syntactic structure. It decomposes code into smaller blocks and introduces artificial control constructs (e.g., dispatchers or indirect jumps), making the original execution order difficult to reconstruct [5]. L3: Combination of L1 and L2. L3 combines identifier renaming and control-flow alteration. As the most complex setting, L3 serves as an upper bound on obfuscation difficulty for the assessment of both human and model code understanding. B. Human Understanding of Obfuscated Code

Together these questions probe two levels of alignment. RQ1–RQ2 ask whether models find the same tasks hard as humans (outcome-level), spanning all Block-Model levels — including the relational level, where control-flow obfuscation forces models to resolve dispatcher-mediated inter-block dependencies. RQ3–RQ4 ask whether models reproduce the failure modes obfuscation induces in humans (mechanism-level): effort scaling at the macro level, proxied by CoT trace length (RQ3), and confident misinterpretation under misleading identifiers at the atom level (RQ4). RQ1: To what extent do LLMs succeed and fail on the same obfuscated code tasks as human participants? Nguyen et al. [1] shows that obfuscation shifts human reasoning from System 1 to System 2. We investigate whether LLMs exhibit the same task-level difficulty patterns as humans. RQ2: Do different model architectures and sizes align with human expertise levels? While RQ1 examines whether models track human difficulty patterns, RQ2 investigates if any model’s reasoning profile resembles that of a particular experience bracket. Humans with varied experience employ qualitatively different strategies under obfuscation. We investigate whether model size or architecture produces analogous stratification. RQ3: Can CoT Reasoning length traces measure an LLM’s reasoning and task difficulty?

At the macro level of the Block Model, we use CoT trace Nguyen et al. [1] studied human program comprehension length as a proxy for mental effort. The length of a model’s CoT under code obfuscation. They recruited 50 undergraduate comtokens has been shown to correlate with the time humans take to puter science students to perform output-prediction tasks under solve the same problems [6]. Building on this, we investigate controlled conditions. Each participant was asked to inspect a whether CoT trace length serves as a reliable indicator of Python or JavaScript function together with a concrete input and reasoning depth and inherent task difficulty, or whether longer predict the exact output, while the study recorded correctness, traces reflect model confusion rather than productive deduction. response time, and self-reported programming experience. The Because the human study records response time as a measure dataset contains 20 function-level output-prediction tasks from of effort [1], this lets us compare directly how human and HumanEval-X, with 10 JavaScript snippets and 10 Python model effort scale with obfuscation. snippets with cyclomatic complexity between 4 and 8 and fewer than 15 lines, making them non-trivial but still feasible RQ4: Does adversarial renaming cause LLMs to generate for manual reasoning. Each snippet is instantiated across five high-confidence, incorrect answers due to semantic misobfuscation tiers L0-L3. We use this dataset to enable direct framing? Human understanding treats identifiers as semantic comparison between LLM behavior and human patterns. Our beacons, so misleading names can induce confident but incorwork extends this study from human participants to large rect mental models–the failure mode adversarial renaming language models, reusing its tasks, obfuscation tiers, and (L1b) was designed to elicit [1]. We ask whether models reported difficulty patterns. inherit the same atom-level vulnerability. At the atom level, standard and adversarial renaming present different forms of III. R ESEARCH Q UESTIONS semantic transformation, which may affect models beyond We organize our research questions using Carsten Schulte’s accuracy. We study how adversarial renaming and semantic disBlock Model [2] (Fig. 1), which organizes program comprehen- placement relate to accuracy, confidence, and high-confidence sion into four hierarchical levels: Atoms (individual language incorrect predictions.

TABLE I: Details of selected models. Size

Model

Type

Release

Small

DeepSeek-R1-Distill-Qwen-1.5B [12] Phi-4-mini-flash-reasoning (4B) [13] SmolLM3-3B [14] Qwen3-0.6B [15] Qwen2.5-0.5B-Instruct [16] Phi-3.5-mini-instruct [17]

Reason Jan ’25 Reason Jul ’25 Reason Jul ’25 Reason Apr ’25 Instr. Sep ’24 Instr. Aug ’24

Med.

DeepSeek-R1-Distill-Qwen-7B [12] Qwen-7B [18] Llama-3.1-8B-Instruct [19] CodeLlama-7B-Instruct-hf [20] Qwen2.5-Coder-7B-Instruct [21] DeepSeek-Coder-6.7B-Instruct [22]

Reason Instr. Instr. Coder Coder Coder

Jan ’25 Sep ’23 Jul ’25 Jul ’23 Sep ’24 Nov ’23

Fig. 2: Prompt Design: Different Variations IV. E XPERIMENTAL M ETHODOLOGY B. Prompt Design A. Datasets and Model Selection 1) Datasets: We used two datasets A and B: Dataset A is drawn from Nguyen et al.’s study [1] to enable the study on the alignment of human and model understanding as they contain the human labels for understandability (Section II-B). To further study models’ understanding, we collected our own dataset B, comprising 250 snippets in total, with 10 unique questions sampled from each benchmark family (HumanEvalX Python/JavaScript [7], CruxEval-X Python/JavaScript [8], LeetCode [9]), instantiated across all five obfuscation tiers (L0–L3). To mitigate training data contamination, we consider a temporal cutoff based on the LeetCode dataset construction date (05/2025). For problems with multiple input-output pairs, we randomly select a single pair with a fixed seed and reuse it consistently across all tiers. We retain only self-contained, deterministic tasks, excluding cases involving external state, or randomized behavior. Obfuscation tiers L1–L3 are generated using ObfuXtreme [10] (Python) and javascript-obfuscator [11] (JavaScript), configured to isolate identifier renaming (L1) and control-flow flattening (L2), with L3 combining both. All obfuscated programs are validated by execution to ensure functional equivalence with the original code. Dataset A is used for RQ1–RQ2 (human–model alignment); Dataset B for RQ3–RQ4 (model-based output prediction). 2) Models: Our experimental framework spans diverse architectures and scales, including the Llama, Qwen, DeepSeek, Phi, and SmolLM families. To capture emerging reasoning capabilities, we include models such as SmolLM3-3B and Qwen3-0.6B, which provide explicit “think” modes enabling analysis of System 1 vs. System 2 behavior; both are evaluated in reasoning mode. We also include a small number of postcutoff models (e.g., SmolLM3-3B, Phi-4-mini-flash-reasoning, and Llama-3.1-8B-Instruct) to improve coverage across model types and architectures. As release dates are an imperfect proxy for training data cutoff, results are interpreted with appropriate caution rather than strict temporal separation.

Our prompt suite is designed around five orthogonal axes— reasoning depth, cognitive interference, verification, token budget, and external scaffolding—so that any performance shift can be attributed to a specific manipulation rather than incidental wording. We began from two anchor prompts, a bare output-prediction request (BASELINE) and an explicit line-by-line simulation request (S2), and derived the remaining conditions by applying one controlled modification at a time (e.g., a memory tax, a token cap, a one-shot exemplar) while holding the task instruction and answer-extraction format fixed. This yields the prompt conditions summarized in Figure 2, which probe several dimensions of model reasoning: we vary the depth of reasoning elicited (Category 1), from rapid baseline responses to line-by-line simulation (S2); introduce cognitive interference (Category 2), such as holding a static key in memory (WM_TAX) or prepending distractor code (CONTEXT_LOAD); require multi-pass verification (Category 3), where the model re-checks its own output (TWO_PASS_THINK); constrain the token budget (Category 4), from unconstrained to a 50-token hard limit, separating reasoning verbosity from answer quality; and supply exogenous guidance (Category 5), such as hints or one-shot examples, to probe how external scaffolding interacts with intrinsic reasoning. To keep conditions comparable across models, instruction bodies are identical for all models, with only the official per-model chat template applied and no permodel prompt tuning; reasoning-mode models are run in their native “think” mode. Complete prompt text is available on replication package [23]. All experiments were done on four NVIDIA RTX A6000 GPUs. Dataset A comprises 21 prompt conditions (25 total passes, as one condition repeats 5 times for self-consistency) × 20 snippets × 5 obfuscation tiers × 7 models, totaling 17,500 evaluation runs. Dataset B comprises 5 prompt conditions × 50 snippets × 5 obfuscation tiers × 9 models, totaling 11,250 runs. C. Evaluation Metrics and Block-Model Measures We assess comprehension with one primary outcome – output accuracy – and a set of measures that localize failures to specific

levels of the Block Model (Fig. 1). V. LLM ALIGNMENT WITH HUMANS UNDER CODE OBFUSCATION (RQ1) Output accuracy (All). All models perform output prediction and a response is correct only if its predicted output A. Accuracy under Obfuscation is an exact match to the ground-truth output obtained by executing the reference program on that input. All comparisons TABLE II: Obfuscation Tier vs. Accuracy (%) for Models and use mixed-effects logistic regression (binomial GLMMs with Humans on Dataset A crossed random effects for snippet and model); proportions Agent / Model L0 L1 L1b L2 L3 Max are reported with Wilson 95% confidence intervals, p-values Diff are Benjamini–Hochberg FDR-corrected within each research CodeLlama-7B 12.4 13.2 9.6 10.8 11.4 +0.8 question, and we report effect sizes (odds ratios, Cliff’s δ). DS-Coder-6.7B 9.4 13.4 13.4 12.4 12.2 +4.0 Macro level — CoT trace length (RQ3). The number of DS-R1-Qwen-7B 63.8 64.2 51.8 57.0 56.2 +0.4 Llama-3.1-8B 15.6 23.4 12.4 16.8 19.8 +7.8 reasoning tokens generated before the final answer, used as a Phi-4-Mini 6.0 4.2 4.4 3.4 3.2 -2.8 proxy for mental effort. Qwen3-0.6B 5.2 5.2 5.0 5.8 4.6 +0.6 Relational/block level — dispatcher complexity (RQ1). SmolLM3-3B 45.0 44.0 34.2 37.6 33.0 -12.0 Under control-flow flattening, original control flow is replaced Human (all) 40.46 38.68 38.02 34.15 31.09 -9.37 by dispatcher constructs. We quantify their complexity with the Beginner 34.21 26.67 26.32 31.43 23.08 -11.13 Python while-if state count, JavaScript dispatch-call references, Intermediate 45.10 43.14 37.50 32.76 33.33 -12.34 Expert 40.48 44.00 55.56 40.00 37.50 +15.08 and object-dispatch helpers; these are used in Section V-A. Atom level — adversarial-renaming measures (RQ4). Table II shows performance for models and humans. Across These measures isolate the lexical-semantic effects of adversarmodels, accuracy generally declines as obfuscation increases ial renaming. We measure semantic distance at the identifier (mixed-effects logistic regression, tier effect χ2 (4) = 45.4, level using SFR embeddings [24], computing a weighted score p < 0.001), although the pattern is rarely monotonic. Several combining cosine distance with a semantic-shift term (capturing models—including DS-Coder-6.7B, DS-R1-Qwen-7B, and how much the renamed identifier becomes more semantically Llama-3.1-8B— achieve their highest accuracy not on the suggestive than the original). Snippet-level distance is the mean original code (L0) but on mildly obfuscated versions (L1). This over all aligned identifier pairs, with identifiers embedded in suggests that slight perturbations to identifiers may suppress isolation after normalization to isolate lexical-semantic effects superficial pattern matching and encourage deeper reasoning. from broader context. Adversarial mis-framing is the additional However, this effect is limited, as accuracy typically declines semantic distance introduced by L1b relative to L1. at higher obfuscation tiers (L2–L3). This decline has a structural cause. At L2/L3, control∆misframe = DL0→L1b − DL0→L1 . flow flattening imposes a new execution architecture rather than modifying existing logic: dispatcher structures are absent We introduce two measures of confident, localized failure. High- through L1b and appear only at L2/L3 (Fig. 3). In Block Model Confidence Incorrect (HCI) flags wrong answers produced terms, this zero-to-nonzero transition shifts reasoning from local at group-normalized confidence zi ≥ 1.0, where zi is the code units to global execution flow, where helper functions and confidence defined earlier. Identifier-level disruption is captured dispatch states act as block-level units and dispatcher calls as by the Identifier Spike Fraction (ISF), the share of generation relation-level indirection, obscuring macro-level flow through uncertainty concentrated on identifier tokens, binned by Q3 into implicit state tracking. Consistent with this hierarchy, dispatcher a spike regime {low, med, high}. A spike is a token position complexity within L2/L3 correlates negatively with accuracy where the model is unusually surprised by its own generated in both languages—most strongly for the Python while-if state token relative to the same forward pass; token t is flagged if count (r = −0.196, q = 3.08 × 10−23 ), and also for JavaScript dispatch-call references (r=−0.130, q=2.76×10−7 ) and objectpplt = exp(− log p(xt | x<t )) > median(ppl)+3·MAD(ppl) dispatch helpers (r=−0.095, q=2.69 × 10−4 ), all significant after multiple-testing correction. The effect is graded—stateor falls in the top 2% of per-token perplexities in the trace, based dispatch degrades accuracy more than relation-level where MAD = median(|pplt − median(ppl)|). indirection, indicating that failures stem from reconstructing Prior work showed that model probability-based uncertainty execution state, not reasoning over local structure. This is not a signals can serve as effective quality/confidence indicators [25], single-model artifact: the Python state-count effect holds across [26], [27]. Thus, we define model confidence as the mean token all seven models and the JavaScript effect across two instructionlog-probability of a generated response (length-normalized), tuned models. Critically, the effect ties to correctness loss, yielding a sample-level logprob_score. To enable cross-model not longer reasoning. Its correlation with CoT length is and cross-condition comparison, this score is normalized via insignificant (r ≈ 0.03–0.05), so the decline reflects genuine z-scoring within each model × condition group. The resulting comprehension difficulty, not merely longer reasoning traces. normalized confidence is used to define high-confidence incorPerformance varies considerably by architecture. The rect predictions and to compute task-level confidence shifts. strongest reasoning-tuned models (DS-R1-Qwen-7B,

JavaScript L2

TABLE III: Tier Breakdown: Models vs Humans (%) on Dataset A

function f(d, n) {

Models

Block level — object helper functions (×3)

const op = { lt: (a,b) => a < b, eq: (a,b) => a === b, sub: (a,b) => a - b, };

1 2 3

Relation level — object dispatcher usages (×3)

for (let i=0; op.lt(i,n); i++) { let k = Object.keys(d); if (op.eq(k.length,0)) break; delete d[k[op.sub(k.length,1)]]; }

1

2 3

}

Python L2 def isBalanced(num): s = 0 while s < 3:

L. T.

Humans

CL DS-C R1 L3.1 Phi4 Qw3 Smol All Beg. Int. Exp. 7B 6.7B Q7B 8B M 0.6B 3B

JS L0 12.8 L1 13.2 L1b 14.8 L2 9.6 L3 11.6

9.6 14.0 15.2 15.2 14.4

57.6 10.4 58.0 30.4 38.4 7.6 44.8 13.6 45.6 19.2

4.8 2.4 4.0 1.6 1.6

2.8 4.0 4.0 4.0 4.8

39.6 45.2 26.4 26.8 30.4

43.3 42.9 53.6 27.8 21.3 9.1 20.8 33.3 26.2 28.0 20.0 36.4 25.8 29.4 21.9 30.8 15.9 9.1 17.2 25.0

Py L0 12.0 9.2 70.0 20.8 7.2 L1 13.2 12.8 70.4 16.4 6.0 L1b 4.4 11.6 65.2 17.2 4.8 L2 12.0 9.6 69.2 20.0 5.2 L3 11.2 10.0 66.8 20.4 4.8

7.6 6.4 6.0 7.6 4.4

50.4 42.8 42.0 48.4 35.6

37.5 23.5 34.8 50.0 52.5 36.8 63.0 53.9 50.0 23.1 51.6 68.8 42.6 33.3 46.2 47.1 48.2 41.2 57.9 45.0

Block level — dispatch states (×3) 1

2

3

if s == 0: f = [0, 0]; s += 1 if s == 1: for i,c in enumerate(num): f[i] += int(c) s += 1 if s == 2: return f[0] == f[1]

Fig. 3: Dispatch-related metric, with structure highlighting

SmolLM3-3B) outperform coders and instruction-tuned models across all tiers, largely independent of model size. Weaker models (Phi-4-Mini, Qwen3-0.6B) show a clear floor effect: already near chance at baseline, they change little across tiers, since added obfuscation produces only small absolute shifts. Stronger models, starting from higher baselines, show larger absolute drops. Table III shows a different pattern for human participants. Overall human accuracy decreases as obfuscation increases, particularly for beginners and intermediate programmers. However, expert programmers show a different trend: their accuracy improves under moderate obfuscation (L1b), reaching the highest performance among all human groups. This suggests that experts rely less on lexical cues such as identifier names and instead reason about code structure and semantics, a human pattern that, as shown in Section VI, models do not reproduce.

Fig. 4: Accuracy across conditions for reasoning, instruct, and coder models.

C. Human–model Alignment

The strongest performers are DeepSeek-R1-Qwen-7b and SmolLM3-3b, both of which demonstrated robust reasoning capabilities. Conditional on producing a final answer, DSR1-Qwen-7B is correct 58.8% of the time, while SmolLM3B. Language Effects 3B is correct 49.6% of the time. Fig. 4 shows that the Table III reveals a language effect that is stronger in humans strongest reasoning models (DS-R1-Qwen-7B, and marginally than models. Among reasoning models the Python advantage SmolLM3-3B) exceed human baselines but collapse under is clear only for DS-R1-Qwen-7B (high and stable on Python low token limits (OR=12.5, 95% CI [6.4,24.2], p<0.001). This while JavaScript drops at L1b); SmolLM3-3B is weaker and suggests that their performance is a function of System 2. inconsistent, so the effect is not a reliable main effect but is tier- All models show a performance lift with S2 + One-Shot dependent (significant only at L2, p = 0.036). Humans show prompts. This suggests that combining explicit reasoning paths a stronger, more consistent pattern: JavaScript accuracy falls with in-context examples effectively reduces the logical search sharply with obfuscation while Python stays robust, especially space. Finally, the negligible impact of "WM Tax" suggests that, for intermediate and expert programmers, with experts peaking unlike humans, model failures are driven by logical execution on Python at L1b — evidence of structural rather than identifier- bottlenecks rather than information retention limits. based reasoning. Even strong LLMs remain less robust under Table IV assesses if model difficulty aligns with human this condition, highlighting a gap between model reasoning performance at the task level. Reasoning models correlate and expert human understanding. significantly with human accuracy; coder and instruct

TABLE IV: Correlation between human accuracy and model accuracy. Significant correlations (p < 0.05) are highlighted. Model Qwen3-0.6B Phi-4 Mini (4B) SmolLM3-3B DS-R1 Qwen-7B Llama 3.1-8B CodeLlama 7B DS-Coder 6.7B

Size Small Small Small Medium Medium Medium Medium

Type Spearman ρ p-value Reasoning 0.30 0.003 Reasoning 0.47 < 0.001 Reasoning 0.36 < 0.001 Reasoning 0.37 < 0.001 Instruct 0.08 0.440 Coder 0.02 0.838 Coder 0.10 0.349

models do not. Among reasoning models, Phi-4 Mini has the highest correlation (ρ = 0.47), followed by DS-R1-Qwen7B, SmolLM3-3B, and Qwen3-0.6B. CodeLlama-7B and DSCoder-6.7B show near-zero correlations. Notably, the alignment and raw accuracy are distinct: DS-R1-Qwen-7B is the strongest overall performer, but Phi4 Mini best mirrors human task-level behavior. Similarly, positive correlation doesn’t imply high performance, even weak reasoning models correlate with humans, suggesting they share the same sources of difficulty even when failing.

Fig. 5: Accuracy vs. obfuscation tier (L0–L3) by model size and prompt setting.

Fig. 6: Accuracy vs. obfuscation tier (L0–L3) by model type and prompt setting.

RQ1 Takeaway: Reasoning-tuned models show significant alignment with human task-level difficulty (Spearman ρ = 0.30–0.47), tracking which obfuscated tasks humans find hard. Coder and instruct-tuned models show near-zero correlation.

Prompt condition modulates absolute performance but does not alter the relative size ordering. Reasoning models consistently outperform coder and instruct models across all prompt conditions and obfuscation tiers, with the gap most clearly under System 2 prompting (Fig. 6). Instruct and coder models perform comparably, though instruct models exhibit a notable sensitivity peak at L1b under System 2, suggesting heightened reliance on semantic cues, before recovering at higher tiers. Reasoning models are the most stable across obfuscation tiers, though not strictly monotonic.

VI. H OW IS MODEL ALIGNMENT WITH HUMAN EXPERIENCE ? (RQ2)

TABLE VI: Accuracy by obfuscation tier and model on Dataset A divided by language Model Format: JS / PY

TABLE V: Accuracy vs. obfuscation level by model type Model

Size

Type

L0

L1

L1b

L2

L3

CodeLlama 7B DS-Coder 6.7B

Medium Medium

Coder Coder

12% 9%

13% 13%

10% 13%

11% 12%

11% 12%

Llama 3.1-8B

Medium

Instruct

16%

23%

12%

17%

20%

Qwen3-0.6B Phi-4 Mini (4B) SmolLM3-3B DS-R1 Qwen-7B

Small Small Small Medium

Reasoning Reasoning Reasoning Reasoning

5% 6% 45% 64%

5% 4% 44% 64%

5% 4% 34% 52%

6% 3% 38% 57%

5% 3% 33% 56%

As seen in Table V, reasoning model performance scales with size. DS-R1 Qwen-7B leads all models at 64% baseline accuracy, with the notable exception of Phi-4 Mini, which underperforms despite its 4B parameters. Coder and instruct models occupy the middle of the range, above the floor-bound small reasoning models but well below DS-R1-Qwen-7B and SmolLM3-3B. Across all models, accuracy declines across obfuscation tiers (no model shows a strictly monotonic trend). We divide prompts into two broad categories: System 1 (quick thinking, e.g., prompting not to think) and System 2 (deliberate thinking, e.g., prompting to take time or doublecheck), comprising 14 and 7 prompts respectively. As shown in Fig. 5, medium models outperform small models at every obfuscation tier across all prompt conditions, though System 2 prompting induces a slight accuracy dip at L1b.

L0 acc

L1 acc

L1b acc

L2 acc

L3 acc

Qwen3-0.6B (Small) Phi-4 Mini (4B) (Small) SmolLM3-3B (Small)

3% / 8% 5% / 7% 40% / 50%

4% / 6% 2% / 6% 45% / 43%

4% / 6% 4% / 5% 26% / 42%

4% / 8% 2% / 5% 27% / 48%

5% / 4% 2% / 5% 30% / 36%

DS-R1 Qwen-7B (Medium) Llama 3.1-8B (Medium) CodeLlama 7B (Medium) DS-Coder 6.7B (Medium)

58% / 70% 10% / 21% 13% / 12% 10% / 9%

58% / 70% 30% / 16% 13% / 13% 14% / 13%

38% / 65% 8% / 17% 15% / 4% 15% / 12%

45% / 69% 14% / 20% 10% / 12% 15% / 10%

46% / 67% 19% / 20% 12% / 11% 14% / 10%

Table VI shows that the size-dependent performance pattern holds within each language. DS-R1 Qwen-7B is the top performer in both JavaScript and Python, reaching 58% and 70% at L0 respectively. SmolLM3-3B stands out among small models, achieving comparable baseline accuracy to some medium models (JS: 40%, PY: 50%), though it degrades more steeply under obfuscation. Coder models consistently underperform reasoning and instruct models at the same size across both languages, and Python baseline accuracy is generally higher than JavaScript, though relative model rankings remain stable between languages. Obfuscation sensitivity is greater in JavaScript, particularly at L1b, suggesting language-specific features amplify difficulty without altering capability rankings. The S2_ONESHOT prompt (Table VII) benefits models with near-zero baseline accuracy most strongly (Qwen3-0.6B and Phi-4 Mini improve from 0% to non-trivial accuracy across most tiers), while SmolLM3-3B benefits least. DS-R1 Qwen-7B shows a mixed pattern: gains at L1 and L3, slight reductions at L0 and L1b.

TABLE VII: Accuracy by obfuscation tier and model on Dataset A. Each cell reports Baseline/S2_ONESHOT prompts (for details on prompts, see Section IV-B) accuracy. Model

L0

L1

L1b

L2

L3

Qwen3-0.6B (Small) Phi-4 Mini (4B) (Small) SmolLM3-3B (Small)

0/ 25 % 0/ 65 % 25/ 55 %

0/ 30 % 0/ 35 % 35/ 55 %

0/ 15 % 0/ 45 % 25/25%

0/ 45 % 0/ 15 % 45/45%

0/ 50 % 0/ 45 % 40/40%

DS-R1 Qwen-7B (Medium) Llama 3.1-8B (Medium) CodeLlama 7B (Medium) DS-Coder 6.7B (Medium)

70/65% 5/ 40 % 30/ 35 % 0/ 20 %

65/ 70 % 10/ 45 % 25/ 30 % 0/ 25 %

55/45% 15/ 35 % 20/ 25 % 0/ 30 %

70/55% 10/ 25 % 15/15% 0/ 20 %

60/ 80 % 30/ 35 % 25/ 35 % 5/ 20 %

Fig. 8: Human–model alignment by experience level. Each cell reports Spearman ρ and BH-FDR q; bordered cells are significant (q < .05). Reasoning models show significant positive alignment across all three tiers, with no tier reliably distinguishable from the others.

Fig. 7: Accuracy (%) by programming language (JavaScript vs. Python) across obfuscation tiers (L0–L3). Left: model performance averaged across all models. Right: human performance across all experience levels, reproduced from [1].

Fig. 7 (right) shows that human performance varies substan- Fig. 9: Mean Accuracy per Question vs. Mean CoT Tokens tially across languages, with consistently higher accuracy in per Answer with 95% confidence thresholds Python compared to JavaScript across all obfuscation tiers (L1L3) [1]. In contrast, Fig. 7 (left) illustrates that while models also exhibit language-dependent variation, these differences are tuned models approximate the human difficulty patterns that less pronounced and follow a different pattern of degradation. obfuscation induces, while coder and instruct models do not [1]. If models align with human expertise, we would expect similar relative trends across language and tiers. However, the RQ2 Takeaway: The strongest reasoning models outdivergence between human and model performance patterns perform coder and instruct models and align with human suggests that models do not possess the same languagedifficulty patterns across all experience tiers, with no dependent comprehension behavior observed in humans. These tier robustly distinguishable from the others. No model differences suggest model performance may be influenced more reproduces the expert accuracy inversion under L1b. by factors such as capacity and architectural specialization than by language-specific experience. VII. C HAIN OF T HOUGHT (C OT) T RACE L ENGTHS (RQ3) To directly assess alignment with human expertise levels, Unlike RQ1–RQ2, which rely on Dataset A for huwe compare model performance patterns with human groups stratified by experience. Fig. 8 shows that reasoning models ex- man–model alignment, the analyses in this section are computed hibit significant positive correlations across all three experience over Dataset B, our larger model-only output-prediction set tiers, with no tier reliably distinguishable from the others (mean spanning all five obfuscation tiers. Fig. 9 shows a significant negative correlation between mean ρ ≈ 0.24–0.30; leave-one-model-out ordering is unstable and a mixed model finds no tier × alignment interaction). Coder and accuracy and mean CoT trace length (Spearman ρ = −0.520, instruct models, by contrast, show statistically insignificant p = 1.11 × 10−4 ; Pearson r = −0.515, p = 1.32 × 10−4 ), correlations across all experience levels. The presence of persistent across models, prompts, and obfuscation tiers. alignment across every tier exclusively in reasoning models As shown in Table VIII, incorrect predictions have descripreinforces the RQ1 finding: reasoning models succeed and tively higher mean token counts than correct ones, exceeding fail on the same tasks as human participants and share their 1,000 tokens across L1–L3, though this difference is not difficulty profiles across obfuscation tiers, whereas coder and significant once snippet and model are accounted for (p = 0.33). instruct models show no such alignment, implying their failures Mean token usage for incorrect answers is also nearly identical are driven by fundamentally different processes. Reasoning- between structural (L1: 1,011.4) and semantic (L1b: 1,014.4)

TABLE VIII: Overall mean CoT trace length stratified by correct/incorrect and obfuscation tier Outcome Correct Incorrect

Obfuscation Tier L0

L1

L1b

L2

L3

910.7 978.1

924.9 1011.4

962.1 1014.4

976.3 1024.3

938.1 1028.3

obfuscation, indicating that added obfuscation does not translate into proportionally longer reasoning on failed attempts. In contrast, successful predictions show a monotonic increase in token length from L0 to L2 (negative-binomial regression, p = 0.004), indicating that models expend progressively more effort on increasingly obfuscated code. Correct-response length does not differ between L1 and L1b (negative-binomial GLMM, estimated marginal means contrast, ratio = 1.01, 95% CI [0.91, 1.12], p = 0.85). The negative accuracy–length correlation across questions is driven by task difficulty: harder questions elicit longer traces and are also answered incorrectly more often. A notable exception occurs at L3, where correct responses average fewer tokens than at L2, likely survivorship bias: compounding complexity leaves only the simplest programs (e.g., trivial boolean returns) solvable, which require shorter traces.

Fig. 10: Log length of CoT

Fig. 11: Human response time [1]

Fig. 12: Accuracy by obfuscation tiers (L0–L3), stratified by CoT length tercile (easy = shortest third, hard = longest third).

Fig. 13: Distribution of semantic distance (SFR embeddings) for L0→L1 and L0→L1b. Points show snippet-level means; boxplots summarize distributions.

RQ3 Takeaway: CoT length tracks task difficulty rather than serving as a confusion signal: the tasks eliciting the longest traces are consistently those the model answers incorrectly, so length marks difficulty across questions (ρ = −0.52). CoT length also rises with obfuscation tier (p = 0.004).

VIII. A DVERSARIAL RENAMING (RQ4) Figs. 10 and 11 contrast how models and humans respond to increasing code obfuscation. For models, higher obfuscation Having established that reasoning models track which tasks generally drives a monotonic, language-agnostic increase humans find hard (RQ1–RQ2), we now ask whether they in mean Chain-of-Thought (CoT) length, culminating in a also reproduce a specific human failure mode: confident significant spike at L3. The only exception is L1b, which misinterpretation under misleading identifiers. exhibits minor language-specific variances. In contrast to Prior work shows identifier names can mislead neural this steady scaling of model effort, human response times models under semantics-preserving transformations [28], [29], jump significantly from L0 to L1 but remain flat through L3. but does not quantify the semantic displacement introduced This divergence suggests that while humans utilize distinct or its effect on confidence–accuracy relationships and highcognitive processing modes (System 1 and System 2), models confidence errors. Building on this, we situate adversarial scale their computational effort more monotonically. renaming at the atom level of the Block Model, isolating Fig. 12 stratifies tasks by Chain-of-Thought length tercile: whether comprehension failures arise from lexical-semantic the shortest third of traces are labeled easy, the longest third misinterpretation rather than structural complexity. Fig. 13 confirms construct validity: L1b produces substanhard. The three terciles stay cleanly separated and roughly flat across all tiers, with easy tasks near 43–54% and hard tially higher semantic distance than L1 (standard renaming), tasks around 20–28%. This persistent separation from L0 to with well-separated distributions and minimal overlap. L3 reinforces the RQ3 finding: the tasks eliciting the longest Fig. 14 examines model behavior within L1b, stratified by traces are consistently those the model gets wrong, indicating semantic mis-framing magnitude (DL0→L1b − DL0→L1 ). In that trace length tracks with difficulty. the low-confidence region, high- and low-displacement subsets

(a) Tercile stratification

(b) Decile stratification

TABLE X: Accuracy and HCI rate across L1b tail-regime subsets. Low/Low baseline = both metrics in bottom 20%. Regime Low/Low baseline Top 20% distance Top 20% spike Top distance ∩ spike

n 202 450 450 80

Accuracy 31.68% 33.11% 36.00% 21.25%

HCI Rate 4.95% 6.89% 8.44% 10.00%

Fig. 14: Confidence–accuracy relationship in L1b across semantic-displacement strata. Left: terciles; Right: deciles. TABLE IX: Localization results by semantic distance and identifier-spike regime. ∆ measures the change from low spike to medium spike within each semantic-distance band. Accuracy HCI Low Med. Low Med. ∆ Sem. Dist. ∆ Spike Spike Spike Spike Low 29.44% 38.61% +9.18% 1.52% 5.94% +4.42% Medium 32.51% 30.96% -1.55% 3.99% 8.79% +4.80% High 36.33% 31.25% -5.08% 5.66% 5.77% +0.11%

behave similarly. In the mid-confidence range, however, they diverge: the low-displacement subset shows a gradual, nearmonotonic increase in accuracy (∼0.38–0.42), consistent with a well-calibrated confidence–accuracy relationship, whereas the high-displacement subset exhibits non-monotonic behavior, peaking at the third decile (∼0.55) before dropping sharply (∼0.33) at higher confidence levels. At the highest confidence decile, both subsets collapse, likely due to degenerate outputs in which the model enters a repetitive generation loop and assigns near-certain probability to an incorrect token sequence. Reorganizing the same data by matched confidence deciles (Fig. 14b) shows that the gap between subsets varies inconsistently across deciles, with no stable ordering. This indicates that semantic displacement does not systematically shift accuracy at a fixed confidence level, but instead destabilizes the confidence–accuracy relationship: predictions at similar confidence can yield substantially different outcomes depending on displacement magnitude. From a Block Model perspective, this instability originates at the atom level — misleading identifier semantics disrupt the mapping between surface cues and underlying computation without altering structural reasoning. This variability motivates a finer-grained analysis of where confident errors concentrate, which we examine through HCI and identifier-level disruption below. Table IX reports combined effects on accuracy and HCI. Spike intensity affects accuracy in a displacement-dependent way — decreasing it under medium and high displacement, increasing it under low — with no consistent relationship. HCI, by contrast, rises with spike intensity for low and medium displacement and barely changes for high, pointing to identifierlevel disruption as the more consistent driver of confident failures. Having examined both factors at moderate ranges, we next isolate tail subsets to test whether joint extremes exceed

Fig. 15: Per-task changes in wrong-answer confidence (∆z = zL1b − zL1 ), sorted by magnitude. Each point is a task; the dashed line indicates zero change.

what either factor alone predicts. Table X compares performance within tail subsets, defined by high semantic displacement, high identifier-level disruption, and their intersection, against a low-distance/low-spike baseline. Semantic displacement alone has limited effect: the top20% distance subset shows accuracy comparable to baseline (33.11% vs. 31.68%) with a moderate HCI increase (6.89% vs. 4.95%). Identifier-level spikes alone produce a different pattern: accuracy rises to 36.00% while HCI also increases to 8.44%, pointing toward more decisive but less reliable predictions. These results localize semantic mis-framing to the atom level. Identifier-level disruption (spikes) increases high-confidence errors, indicating that failures originate from instability around renamed identifiers rather than from structural reasoning. When both factors co-occur, however, the interaction is nonadditive and severe. Accuracy collapses to 21.25%, a drop of 15 percentage points relative to the spike-alone subset, while HCI reaches 10.00%, the highest value observed across all conditions. The sharpness of the accuracy collapse, rather than the modest HCI increment, is the clearest signal that joint extremes constitute a qualitatively distinct failure regime rather than an additive combination of two moderate effects. Whether these concentrated failures reflect consistent task-level patterns or are distributed unevenly across the evaluation set remains open; we examine this next. We examine task-level variation by aggregating wronganswer confidence changes across problems. Fig. 15 shows the distribution of ∆z (L1b–L1) over persistently incorrect cases, ranging from −0.8 to +0.65, with no central concentration. This indicates that adversarial renaming does not produce a consistent directional effect on wrong-answer confidence. Table XI quantifies this heterogeneity: at the aggregate level the distribution is nearly balanced (26/50 tasks with ∆z < 0 vs. 24/50 with ∆z > 0), but model-level breakdowns reveal

TABLE XI: Counts of tasks with decreases (∆z < 0), increases (∆z > 0), and no change in wrong-answer confidence (∆z = zL1b − zL1 ), grouped by model. Group Overall (task mean) smollm3-3b deepseek-r1-distill-qwen-1.5b deepseek-r1-distill-qwen-7b phi-3.5-mini-instruct qwen-0.5b qwen-7b llama-3.1-8b-instruct qwen-7b-coder deepseek-coder-6.7b

Type All Reasoning Reasoning Reasoning Instruct Instruct Instruct Instruct Coder Coder

Size All Small Small Med. Small Small Med. Med. Med. Med.

∆z < 0 26/50 25/36 19/49 8/22 20/39 27/44 18/30 16/33 21/28 30/49

∆z > 0 24/50 11/36 30/49 14/22 19/39 17/44 12/30 17/33 7/28 19/49

∆z = 0 0/50 0/36 0/49 0/22 0/39 0/44 0/30 0/33 0/28 0/49

that this near-even split reflects opposing tendencies rather than a weak effect. Code-specialized models trend toward confidence suppression, while reasoning-distilled models show the opposite, indicating that the direction of the effect is jointly determined by task characteristics and model behavior. RQ4 Takeaway: Adversarial renaming triggers highconfidence wrong answers only when semantic displacement and identifier-level disruption occur together. The direction of confidence shifts is task- and modeldependent, marking this as a localized, atom-level failure rather than global structural degradation.

traces may be post-hoc rather than faithful [41], [42]. Tokenlevel uncertainty is linked to confusion in humans/models [43]. Researchers capture human attention from eye movements and use it to improve neural code summarization [44]. EyeTrans [45] integrates human attention into Transformer-based code summarization. More recent work further explores training code models to mimic human visual attention [46]. The attention weights of neural models of code are compared with human visual attention during code summarization, asking whether models attend to the same tokens as developers [47]. Expert–Novice Differences in Program Comprehension. Experts rely on plan-based and hierarchical reasoning, whereas novices tend to process code locally and line by line [48], [49]. Pennington [50] shows that expert understanding integrates control-flow and goal-oriented representations, while Corritore and Wiedenbeck [51] identify identifiers as semantic “beacons.” Identifier Sensitivity and Adversarial Robustness in Code Models. Identifiers carry rich semantics in programs [52], [53], yet adversarial renaming misleads models [54], [28], [55]. Code models often rely on surface cues over deep semantics [56], and identifier quality affects performance [57]. XI. I MPLICATIONS AND C ONCLUSION A. Implications for software engineering

1. Our findings bear on human-AI collaboration in SE. Tasks like code review, debugging, and comprehension require developers to interpret and act on model outputs, so model IX. T HREATS TO VALIDITY selection should weigh not only accuracy but whether the Contamination Control. We use model release timing relative model’s reasoning aligns with how developers understand code. to our LeetCode dataset’s creation (May 2025) as a proxy to Reasoning-tuned models are both more accurate and more mitigate training data contamination. However, release dates are aligned with developers’ difficulty patterns, making human an imperfect indicator of training data cutoff, and some models alignment an additional criterion for developer-facing tools: post-date this threshold; thus, overlap with training corpora a model whose difficulty profile resembles a developer’s yields cannot be ruled out. Datasets may limit generalizability. reasoning traces and failure modes easier to inspect, calibrate, Task and Evaluation Design. We restrict the study to and supervise, especially when explaining unfamiliar code, deterministic, self-contained snippets, which may bias results assisting debugging, or code comprehension and review. toward simpler programs and omit real-world complexities like 2. Our results also expose a concrete risk: adversarial external dependencies. Variants such as adversarial renaming or suspicious naming can cause high-confidence incorrect may not fully represent obfuscations. answers when semantic displacement and identifier-level Evaluation Proxy. Output prediction imperfectly proxies disruption co-occur. Since developers often face poorly named, code understanding. We address this through various metrics. generated, minified, legacy, or unfamiliar code, tools should not treat confident explanations as evidence of understanding X. R ELATED W ORK when identifier semantics are suspicious; such cases should Obfuscation and Human Code Comprehension. [3] provides trigger extra checks like running, testing, or review. a taxonomy motivating our tiered design. Surveys establish 3. Our results also caution against treating verbose reasoning obfuscation as a controlled probe of model robustness [30], [31]. as reliable: very long CoT traces show diminishing returns, so Identifier renaming significantly reduces task performance [32], length past a moderate budget is a weak reliability signal and program comprehension depends on working memory, under stronger obfuscation. Unusually long traces, repeated attention, and semantic cues [33], [34]. Meaningful identifiers self-correction, or uncertainty around identifiers should instead reduce cognitive load; poor naming increases it [35], [36]. Most be read as signs of a struggling model, warranting extra checks. directly, Nguyen et al. [1] measured human comprehension under these obfuscation tiers; we build on their paradigm and B. Implications for software security data to study whether LLMs exhibit the same difficulty patterns. 1. Obfuscation is widely used to protect software from LLM Reasoning and Cognitive Alignment. CoT prompting reverse engineering, code pirating, and tampering. If LLMs improves performance on difficult tasks [37], [38], [39], and struggle with obfuscated code as humans do, obfuscation reasoning effort scales with task difficulty [6], [40]. Reasoning stays relevant even under LLM-assisted analysis: our findings

suggest current LLMs do not eliminate its protective value, as transformations like adversarial renaming or control-flow flattening still create meaningful comprehension barriers. 2. Our findings also identify a concrete risk for malware and malicious code: adversaries may intentionally choose benign-looking identifiers to mislead LLM-based analyzers, leading a model to confidently infer safe behavior while missing malicious functionality. As in the SE setting (§XI-A), confident explanations are not sufficient evidence of safety when identifier semantics are suspicious. The shared failure patterns between humans and LLMs also identify security-relevant blind spots. Adversarial renaming can confuse both humans and models, causing models to produce high-confidence incorrect answers. 3. Not all models fail equally: reasoning-tuned models are more robust and human-aligned, suggesting they may serve as analyst-facing assistants for malware analysis and vulnerability triage. Their value is not to replace analysts but to fail more predictably, making them easier to supervise—and in security workflows, where a wrong explanation can mean missed malicious behavior or incorrect assessment, humaninterpretable failure is as crucial as accuracy. Data Availability Statement: Our package is available at [23].

R EFERENCES [1] A. H. N. Nguyen, J. Le, I. L. Coronado, and T. N. Nguyen, “The effect of code obfuscation on human program comprehension,” 2026. [Online]. Available: https://arxiv.org/abs/2603.07668 [2] C. Schulte, “Block model: an educational model of program comprehension as a tool for a scholarly approach to teaching,” in Proceedings of the Fourth International Workshop on Computing Education Research, ser. ICER ’08. New York, NY, USA: Association for Computing Machinery, 2008, p. 149–160. [Online]. Available: https://doi.org/10.1145/1404520.1404535 [3] C. Collberg, C. Thomborson, and D. Low, “A taxonomy of obfuscating transformations,” University of Auckland, Tech. Rep. 148, 07 1997. [Online]. Available: https://researchspace.auckland.ac.nz/handle/2292/ 3491 [4] M. Ceccato, M. Di Penta, J. Nagra, P. Falcarin, F. Ricca, M. Torchiano, and P. Tonella, “Towards experimental evaluation of code obfuscation techniques,” in Proceedings of the 4th ACM Workshop on Quality of Protection, ser. QoP ’08. New York, NY, USA: Association for Computing Machinery, 2008, pp. 39–46. [Online]. Available: https://doi.org/10.1145/1456362.1456371 [5] T. Laszlo and A. Kiss, “Obfuscating c++ programs via control flow flattening,” vol. 30, 06 2007. [6] A. G. de Varda, F. P. D’Elia, H. Kean, A. Lampinen, and E. Fedorenko, “The cost of thinking is similar between large reasoning models and humans,” Proceedings of the National Academy of Sciences, vol. 122, no. 47, p. e2520077122, 2025. [7] Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang, Y. Li, T. Su, Z. Yang, and J. Tang, “Codegeex: A pre-trained model for code generation with multilingual benchmarking on humanevalx,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 5673–5684. [8] R. Xu, J. Cao, Y. Lu, H. Lin, X. Han, B. He, S.-C. Cheung, and L. Sun, “Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution,” 2024. [Online]. Available: https://arxiv.org/abs/2408.13001 [9] Y. Xia, W. Shen, Y. Wang, J. K. Liu, H. Sun, S. Wu, J. Hu, and X. Xu, “Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms,” 2025. [Online]. Available: https://arxiv.org/abs/2504.14655 [10] spyboy productions, “Obfuxtreme,” gitHub repository, last accessed September 24, 2025. [Online]. Available: https: //github.com/spyboy-productions/ObfuXtreme [11] javascript-obfuscator contributors, “javascript-obfuscator,” 2025, javaScript obfuscation tool, package version 4.1.1, last accessed October 17, 2025. [Online]. Available: https://github.com/javascript-obfuscator/ javascript-obfuscator [12] DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [13] L. Ren, C. Chen, H. Xu, Y. J. Kim, A. Atkinson et al., “Decoderhybrid-decoder architecture for efficient reasoning with long generation,” 2025. [14] E. Bakouch, L. Ben Allal, A. Lozhkov, N. Tazi, L. Tunstall, C. M. Patiño, E. Beeching, A. Roucher, A. J. Reedi, Q. Gallouédec, K. Rasul, N. Habib, C. Fourrier, H. Kydlicek, G. Penedo, H. Larcher, M. Morlon, V. Srivastav, J. Lochner, X.-S. Nguyen, C. Raffel, L. von Werra, and T. Wolf, “SmolLM3: smol, multilingual, long-context reasoner,” https: //huggingface.co/blog/smollm3, 2025. [15] Q. Team, “Qwen3 technical report,” 2025. [16] A. Yang et al., “Qwen2 technical report,” 2024. [17] M. Abdin et al., “Phi-3 technical report: A highly capable language model locally on your phone,” 2024. [18] J. Bai et al., “Qwen technical report,” 2023. [19] A. Grattafiori, A. Dubey, A. Jauhri et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [20] B. R. et al., “Code Llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2024, [Online]. Available: https://arxiv.org/ abs/2308.12950. [21] B. H. et al., “Qwen2.5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024, [Online]. Available: https://arxiv.org/abs/2409. 12186. [22] D. G. et al., “DeepSeek-Coder: When the large language model meets programming – the rise of code intelligence,” arXiv preprint

arXiv:2401.14196, 2024, [Online]. Available: https://arxiv.org/abs/2401. 14196. [23] Anonymous, “Do machines struggle where humans do? llm and human comprehension of obfuscated code,” 2026. [Online]. Available: https://doi.org/10.5281/zenodo.19337381 [24] Y. Liu, R. Meng, S. Jot, S. Savarese, C. Xiong, Y. Zhou, and S. Yavuz, “Codexembed: A generalist embedding model family for multiligual and multi-task code retrieval,” 2024. [Online]. Available: https://arxiv.org/abs/2411.12644 [25] M. Fomicheva, S. Sun, L. Yankovskaya, F. Blain, F. Guzmán, M. Fishel, N. Aletras, V. Chaudhary, and L. Specia, “Unsupervised quality estimation for neural machine translation,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 539–555, 2020. [Online]. Available: https://aclanthology.org/2020.tacl-1.35/ [26] N. M. Guerreiro, E. Voita, and A. Martins, “Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, A. Vlachos and I. Augenstein, Eds. Dubrovnik, Croatia: Association for Computational Linguistics, May 2023, pp. 1059–1075. [Online]. Available: https://aclanthology.org/2023.eacl-main.75/ [27] Z. Kang, X. Zhao, and D. Song, “Scalable best-of-n selection for large language models via self-certainty,” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. [Online]. Available: https://openreview.net/forum?id=29FRqmVQK8 [28] N. Yefet, U. Alon, and E. Yahav, “Adversarial examples for models of code,” Proc. ACM Program. Lang., vol. 4, no. OOPSLA, Nov. 2020. [Online]. Available: https://doi.org/10.1145/3428230 [29] Y. Wainakh, M. Rauf, and M. Pradel, “Idbench: Evaluating semantic representations of identifier names in source code,” in Proceedings of the 43rd International Conference on Software Engineering, ser. ICSE ’21. IEEE Press, 2021, p. 562–573. [Online]. Available: https://doi.org/10.1109/ICSE43902.2021.00059 [30] S. Schrittwieser, S. Katzenbeisser, J. Kinder, G. Merzdovnik, and E. Weippl, “Protecting software through obfuscation: Can it keep pace with progress in code analysis?” ACM Comput. Surv., vol. 49, no. 1, Apr. 2016. [Online]. Available: https://doi.org/10.1145/2886012 [31] S. Banescu, C. Collberg, and A. Pretschner, “Predicting the resilience of obfuscated code against symbolic execution attacks via machine learning,” in 26th USENIX Security Symposium (USENIX Security 17). Vancouver, BC: USENIX Association, Aug. 2017, pp. 661–678. [Online]. Available: https://www.usenix.org/conference/usenixsecurity17/ technical-sessions/presentation/banescu [32] M. Ceccato, M. Penta, P. Falcarin, F. Ricca, M. Torchiano, and P. Tonella, “A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques,” Empirical Softw. Engg., vol. 19, no. 4, p. 1040–1074, Aug. 2014. [Online]. Available: https://doi.org/10.1007/s10664-013-9248-x [33] J. Siegmund, C. Kästner, S. Apel, C. Parnin, A. Bethmann, T. Leich, G. Saake, and A. Brechmann, “Understanding understanding source code with functional magnetic resonance imaging,” in Proceedings of the 36th International Conference on Software Engineering, ser. ICSE 2014. New York, NY, USA: Association for Computing Machinery, 2014, p. 378–389. [Online]. Available: https://doi.org/10.1145/2568225.2568252 [34] J. Siegmund, N. Peitek, C. Parnin, S. Apel, J. Hofmeister, C. Kästner, A. Begel, A. Bethmann, and A. Brechmann, “Measuring neural efficiency of program comprehension,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2017. New York, NY, USA: Association for Computing Machinery, 2017, p. 140–150. [Online]. Available: https://doi.org/10.1145/3106237.3106268 [35] D. Lawrie, C. Morrell, H. Feild, and D. Binkley, “What’s in a name? a study of identifiers,” in 14th IEEE International Conference on Program Comprehension (ICPC’06), 2006, pp. 3–12. [36] D. Binkley, M. Davis, D. Lawrie, J. Maletic, C. Morrell, and B. Sharif, “The impact of identifier style on effort and comprehension,” Empirical Software Engineering, vol. 18, 04 2013. [37] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY, USA: Curran Associates Inc., 2022. [38] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” in International Conference

on Learning Representations (ICLR), 2023. [Online]. Available: https://arxiv.org/abs/2203.11171 [39] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY, USA: Curran Associates Inc., 2022. [40] D. Yan, Z. Gao, and Z. Liu, “A closer look at different difficulty levels code generation abilities of chatgpt,” in Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’23. IEEE Press, 2024, p. 1887–1898. [Online]. Available: https://doi.org/10.1109/ASE56229.2023.00096 [41] M. Turpin, J. Michael, E. Perez, and S. R. Bowman, “Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting,” 2023. [Online]. Available: https: //arxiv.org/abs/2305.04388 [42] T. Lanham, A. Chen, A. Radhakrishnan, B. Steiner, C. Denison, D. Hernandez, D. Li, E. Durmus, E. Hubinger, J. Kernion, K. Lukošiūtė, K. Nguyen, N. Cheng, N. Joseph, N. Schiefer, O. Rausch, R. Larson, S. McCandlish, S. Kundu, S. Kadavath, S. Yang, T. Henighan, T. Maxwell, T. Telleen-Lawton, T. Hume, Z. Hatfield-Dodds, J. Kaplan, J. Brauner, S. R. Bowman, and E. Perez, “Measuring faithfulness in chain-of-thought reasoning,” 2023. [Online]. Available: https://arxiv.org/abs/2307.13702 [43] Y. Abdelsalam, N. Peitek, A.-M. Maurer, M. Toneva, and S. Apel, “How do humans and llms process confusing code?” 2025. [Online]. Available: https://arxiv.org/abs/2508.18547 [44] A. Bansal, B. Sharif, and C. McMillan, “Towards modeling human attention from eye movements for neural source code summarization,” Proceedings of the ACM on Human-Computer Interaction, vol. 7, no. ETRA, pp. 1–19, May 2023. [Online]. Available: http://dx.doi.org/10.1145/3591136 [45] Y. Zhang, J. Li, Z. Karas, A. Bansal, T. J.-J. Li, C. McMillan, K. Leach, and Y. Huang, “Eyetrans: Merging human and machine attention for neural code summarization,” Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643732 [46] Y. Zhang, C. Huang, Z. Karas, T. D. Nguyen, K. Leach, and Y. Huang, “Enhancing code llm training with programmer attention,” in Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, ser. FSE Companion ’25. ACM, Jun. 2025, pp. 616–620. [Online]. Available: http://dx.doi.org/10.1145/3696630.3728510 [47] M. Paltenghi and M. Pradel, “Thinking like a developer? comparing the attention of humans with neural models of code,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, Nov. 2021, pp. 867–879. [Online]. Available: http://dx.doi.org/10.1109/ASE51524.2021.9678712 [48] E. Soloway and K. Ehrlich, “Empirical studies of programming knowledge,” IEEE Trans. Softw. Eng., vol. 10, no. 5, p. 595–609, Sep. 1984. [Online]. Available: https://doi.org/10.1109/TSE.1984.5010283 [49] V. Fix, S. Wiedenbeck, and J. Scholtz, “Mental representations of programs by novices and experts,” in Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, 1993, pp. 74–79. [50] N. Pennington, “Stimulus structures and mental representations in expert comprehension of computer programs,” Cognitive Psychology, vol. 19, no. 3, pp. 295–341, 1987. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/0010028587900077 [51] C. L. CORRITORE and S. WIEDENBECK, “An exploratory study of program comprehension strategies of procedural and objectoriented programmers,” International Journal of Human-Computer Studies, vol. 54, no. 1, pp. 1–23, 2001. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1071581900904233 [52] M. Allamanis, E. T. Barr, C. Bird, and C. Sutton, “Suggesting accurate method and class names,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2015. New York, NY, USA: Association for Computing Machinery, 2015, p. 38–49. [Online]. Available: https://doi.org/10.1145/2786805.2786849 [53] M. Allamanis, M. Brockschmidt, and M. Khademi, “Learning to represent programs with graphs,” 2018. [Online]. Available: https://arxiv.org/abs/1711.00740 [54] H. Zhang, Z. Li, G. Li, L. Ma, Y. Liu, and Z. Jin, “Generating adversarial examples for holding robustness of source code processing models,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 01, pp. 1169–1176, Apr. 2020. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/5469

[55] J. Henkel, G. Ramakrishnan, Z. Wang, A. Albarghouthi, S. Jha, and T. Reps, “Semantic robustness of models of source code,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2022, pp. 526–537. [56] M. R. I. Rabin, N. D. Bui, K. Wang, Y. Yu, L. Jiang, and M. A. Alipour, “On the generalizability of neural program models with respect to semantic-preserving program transformations,” Information and Software Technology, vol. 135, p. 106552, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950584921000379 [57] S. L. Nikiema, J. Samhi, A. K. Kaboré, J. Klein, and T. F. Bissyandé, “The code barrier: What llms actually understand?” 2025. [Online]. Available: https://arxiv.org/abs/2504.10557