arXiv:2606.29155v1 [cs.SE] 28 Jun 2026
OASIF: An Efficient Obfuscation-Aware Self-Improving Framework for LLM-Based Assembly Code Instruction Following and Comprehension
Xinyi Wang1,∗ , Rongze Chen1 , Ke Wang2 , Qiyuan Chen3 , Yanming Liu4 , Xiang Li1 , Chunfu Jia1 1 College of Cryptology and Cyber Science, Nankai University 2 Renmin University of China 3 Ant Group 4 Zhejiang University {[email protected]}
Abstract Large Language Models (LLMs) have recently shown promise in automated binary analysis, yet they remain brittle under commercial-grade obfuscation. We present OASIF, an Obfuscation-Aware Self-evolving Instruction-Following framework for obfuscated assembly comprehension. OASIF couples a token-efficient assembly encoder with a lightweight projector to expose long obfuscated code to a pretrained code LLM under a bounded context budget and follows a three-phase training: (i) feature-space alignment, (ii) supervised instruction fine-tuning, and (iii) online self-evolving reinforcement learning with hybrid rewards, enabling continual adaptation with minimal manual verification. On VMISA-Bench, a challenging out-of-distribution suite featuring three commercial VM-based obfuscators, OASIF consistently improves open-source backbones; Qwen2.5-Coder-Instruct-14B attains Success Rate gains of +15.9, +5.8, and +16.9 percentage points (pp) on Code Virtualizer, Themida (v3.0.7), and VMProtect (v3.5), respectively, and improves the OASIF-Bench average by +9.8. OASIF further delivers stable gains across seven standard BCSD benchmarks while preserving general and domain-relevant capabilities on HumanEval, VulBench, and HumanEval-Decompile.
1
Introduction
The analysis and comprehension of assembly code underpin a wide range of security-critical tasks, including reverse engineering [Kargén and Shahmehri, 2017, Megira et al., 2018], malware analysis [Megira et al., 2018], vulnerability detection [Mantovani et al., 2022, Taviss et al., 2024], and software hardening [Thirumoorthy et al., 2022]. Unlike high-level programming languages, assembly code exposes low-level hardware operations with minimal syntactic structure and weak semantic abstraction, resulting in long instruction sequences with low information density [Wang et al., 2022] that are difficult to interpret and reason about. Bridging such hardware-centric representations with natural language semantics is therefore essential for downstream tasks such as semantic recovery, behavioral explanation [Tan et al., 2024], and deobfuscation [Al-Kaswan et al., 2023], yet it remains a long-standing challenge. Recent years have witnessed growing interest in applying LLMs [Achiam et al., 2023, Touvron et al., 2023, Bai et al., 2023] to binary analysis [Mic, 2025]. Early efforts predominantly rely on masked language modeling or contrastive representation learning to extract semantic embeddings from assembly code, exemplified by Asm2Vec [Ding et al., 2019], CodeArt [Su et al., 2024], and CLAP [Wang et al., 2024]. These methods capture instruction-level semantics and support similaritybased retrieval, but they are not designed for instruction following or multi-step reasoning. More Preprint.
recent approaches adapt decoder-based LLMs to assembly code through structural modeling or supervised fine-tuning [Jiang et al., 2023, Tan et al., 2024, Wang et al., 2025], achieving improved generation and explanation capabilities. However, most existing methods depend on static supervision, making them brittle under heavy obfuscation and difficult to scale to real-world scenarios. This limitation becomes particularly pronounced under industrial-grade code obfuscation. Among various techniques, virtual machine (VM)-based obfuscation is widely regarded as one of the strongest protections used in commercial software and real-world malware [Banescu et al., 2016, Blazytko et al., 2017]. By translating native instructions into proprietary virtual instruction sets executed by custom interpreters, VM-based obfuscators fundamentally reshape control flow, instruction semantics, and execution context [Salwan et al., 2018, Li et al., 2022]. The resulting binaries exhibit long execution traces, frequent decoy context switches, and complex one-to-many instruction mappings that are absent from standard training corpora [Coogan et al., 2011, Xu et al., 2018]. Two key challenges hinder LLMs’ effectiveness: obfuscated code often expands to hundreds of thousands of instructions, far exceeding the context window of most open-source LLMs, and acquiring fine-grained supervision is prohibitively expensive, especially for commercial black-box obfuscators. To address these challenges, we propose OASIF, an Obfuscation-Aware Self-evolving InstructionFollowing framework for understanding obfuscated assembly code. As illustrated in Fig. 1, OASIF integrates a token efficient assembly encoder with a pretrained code LLM through a lightweight projection module. We introduce a three-phase training, the first two phases establish stable assembly–language alignment and instruction following behavior via Feature Space Alignment and Instruction Fine-tuning. In the Online Self-Evolving Reinforcement Learning (RL) phase, evaluating its own outputs with hybrid structural and semantic rewards, OASIF enables continuous self-improvement on challenging obfuscated samples. Our main contributions are summarized as follows. • We propose OASIF, a framework that introduces a three-phase training featuring online self-evolving reinforcement learning to improve instruction following and comprehension under assembly obfuscation. • We develop an instruction-centric synthetic data generation engine that enables annotationefficient adaptation to obfuscated assembly code while maintaining training scalability. Additionally, we present OASIF-Bench, a comprehensive obfuscated assembly comprehension and instruction following benchmark. • On VMISA-Bench featuring three commercial VM obfuscators, OASIF consistently improves open-source backbones; specifically, Qwen2.5-Coder-Instruct-14B achieves Success Rate (SR) gains of +15.9, +5.8, and +16.9 pp on Code Virtualizer, Themida(v3.0.7), and VMProtect(v3.5), respectively, and improves the OASIF-Bench average score by +9.8. Furthermore, OASIF enhances general assembly comprehension across seven standard BCSD benchmarks while preserving domain-relevant capabilities on HumanEval, VulBench, and HumanEval-Decompile. (a) Data Engine Assembly Code Source Code (C/C++) Preprocessor Compiler
··· 46: 47: 48: 49: 50: 51: 52: 53: 54:
mov ecx, [ebp+var_4], xor ecx, ebp, call @__security_check_cookie@4, add esp, 74h, cmp ebp, esp, call __RTC_CheckEsp, mov esp, ebp, pop ebp, retn
(b) OASIF Framework
Obfuscator
Control Flow Graph
··· 26: mov eax, [rbp-38h], 27: mov ecx, eax, ··· 1794: sub rsp, 20h, 1795: call cs:__imp_WSACleanup, 1796: add rsp, 20h, 1797: mov dword ptr [rbp-38h], 0ED09D0ADh, 1798: jmp INSTR1815, 1799: mov rax, [rbp-30h], 1800: mov dword ptr [rbp-38h], 2160225Dh, 1801: jmp INSTR1815, 1802: mov rax, [rbp-30h], 1803: mov ecx, [rax], 1804: shl ecx, 1, 1805: mov rax, [rbp+8], 1806: mov [rax], ecx, 1807: mov rax, [rbp+8], 1808: mov ecx, [rax], 1809: sub rsp, 20h, 1810: call printIntLine, 1811: add rsp, 20h, 1812: mov dword ptr [rbp-38h], 3397DCD1h, 1813: jmp INSTR1815, 1814: mov dword ptr [rbp-38h], 97EEC6EAh, 1815: jmp INSTR26
Stage3: Online Self-Evolution RL Policy Optimization
What is the actual argument list passed to __execl in this assembly snippet? Does this assembly appear to be obfuscated; if so, which obfuscation technique(s) were used?
Obfuscated Assembly Code Optimizer
Stage2: Instruction Fine-tuning
Stage1: Feature Space Alignment
LLM Query
<inst_start> [assembly] <inst_end> [text] <EOS>
Assembly-Language Connector
Text Tokenizer
!𝒊 # 𝒚
!𝟏 𝒚
Unlabeled
!𝟐 𝒚
Majority
···
Voting
𝑹(! 𝒚𝟏, 𝒚) 𝒚
Semantic Reward Calculation
𝑹(! 𝒚𝟏, 𝒚) ···
𝑹(! 𝒚𝟏, 𝒚)
𝑺𝒆𝒎𝒂𝒏𝒕𝒊𝒄 𝑺𝒕𝒓𝒖𝒄𝒕𝒖𝒓𝒂𝒍 ···
labeled
𝒚 !𝑵
Structural Reward Calculation
Semantic Rewards Structural Rewards
Predictions Assembly Encoder
Text
Listen_socket read data using a listen socket (server side), execute command with execl.
Figure 1: Overview of the OASIF framework. (a) Data Engine generates paired normal/obfuscated assembly by compiling source code and applying obfuscation. (b) OASIF Framework aligns assembly features with a pretrained code LLM and trains in three stages. 2
2
Related Work
2.1 LLMs for Binary Analysis and Obfuscation Deep learning for binary analysis has progressed from static representation learning to LLM-enabled generative reasoning. Early work mainly adopts Masked Language Modeling (MLM) [Devlin et al., 2019] to capture instruction semantics. Asm2Vec [Ding et al., 2019] and CodeArt [Su et al., 2024] learn robust assembly embeddings, and CLAP [Wang et al., 2024] aligns assembly with natural language via contrastive learning. To enhance comprehension, Nova [Jiang et al., 2023] models assembly structures with hierarchical attention, and ASMA-Tune [Wang et al., 2025] further aligns assembly features with LLMs through structural semantic tuning. These methods strengthen code LLMs on low-information-density assembly inputs, yet robustness under commercial-grade obfuscation remains challenging. Obfuscation further increases difficulty due to substantial control-flow and instruction-level perturbations. DisasLLM [Rong et al., 2024] explores an LLM-driven strategy for analyzing executables. However, systematic evaluations [Tkachenko et al., 2025] show that although models can handle some common transformations, stronger techniques remain difficult, underscoring the need for more robust reasoning and adaptation. Meanwhile, the prevailing reliance on supervised fine-tuning is increasingly limited by the scarcity and cost of high-quality labeled data for obfuscated binaries. 2.2 RL and Self-Evolution in LLMs Recent progress in self-improvement enables models to enhance inference performance with minimal or no additional annotation [Wang et al., 2023, Luo et al., 2023, Yuan et al., 2024]. Selfconsistency [Zuo et al., 2025], for instance, uses majority voting and consistency signals to stabilize reasoning trajectories. While such paradigms have been shown to be effective in domains like mathematical reasoning and code generation [Guo et al., 2025], their practical instantiation for binary security tasks remains underexplored, especially where supervision is expensive and data distributions shift sharply under obfuscation. Our work targets this gap by introducing OASIF, a framework that integrates a multi-modal assembly encoder with a self-evolving reinforcement learning strategy to enhance instruction understanding and reasoning under obfuscation. Moving beyond purely static supervision, OASIF leverages unlabeled or weakly supervised data to iteratively refine instruction-following and comprehension under obfuscation while also improving generalization on standard assembly understanding tasks.
3
Dataset Engine
3.1
Binary Dataset Construction
Raw corpora. Following CLAP [Wang et al., 2024] and ASMA-Tune [Wang et al., 2025], we build a large-scale multi-source corpus from BinaryCorp-3M [Wang et al., 2022] and the Juliet Test Suite [Boland and Black, 2012], and further augment Juliet with obfuscated variants to better reflect real-world reverse-engineering settings. BinaryCorp-3M, collected from Arch Linux official repositories and AUR [Arc, 2025, AUR, 2025], contains 10,265 binaries and ∼3M functions across diverse software categories, supports gcc/clang and five optimization levels, and is among the largest corpora for binary code similarity detection (BCSD). Juliet provides 64,099 vulnerability-analysis test cases with accompanying vulnerability descriptions, and its semantic annotations have been validated for code understanding tasks [Taviss et al., 2024]. Compilation, disassembly, and annotation. For the BinaryCorp-3M training split, we parse instructions and CFGs with Capstone, reconstruct per-instruction jump metadata, sample across compilers and optimization levels, and retain functions with 10–3,000 instructions. For Juliet, we compile C/C++ sources with gcc under randomly selected optimization levels (-O0–-O3), disassemble the binaries with IDA Pro, extract function names and types from symbol-preserved binaries via IDA’s function identification, and derive function-level textual descriptions from vulnerability headers. Obfuscated variants. We further compile Juliet with Obfuscator-LLVM (OLLVM) [Junod et al., 2015] using instruction substitution (SUB), bogus control flow (BCF), and control-flow flattening (FLA). We disassemble x86 64 binaries with IDA Pro and rebase instruction addresses while keeping relative jump offsets [Wang et al., 2024, 2025] to preserve structural semantics and critical identifiers (e.g., function calls and variable names). The resulting raw assembly corpus contains 513,106 3
snippets: 212,117 from the BinaryCorp-3M training split, 79,920 from unobfuscated Juliet, and 221,069 from OLLVM-obfuscated Juliet, including 2,651 SUB, 58,578 FLA, 79,920 BCF, and 79,920 ALL (SUB+FLA+BCF) samples. Details and exact OLLVM flags are reported in Appendix A. 3.2
Instruction-Centric Synthetic Data Generation
Given an assembly corpus C = {ci }N i=1 , we use an LLM Mϕ to generate instruction-centric supervision via task-specific prompts. For a task type t ∈ T , the generator is defined as: G(c, t) = Mϕ (πt (c)) 7→ (Qt , At )
(1)
where πt (·) denotes a few-shot prompt for task t. We consider five task types T = {simp, detail, conv, reason, reason+ }, which correspond to simplified description, detailed description, multi-turn conversation, reasoning, and challenge-level reasoning, following the perceptionto-reasoning progression [Liu et al., 2024, Chen et al., 2024, Luo et al., 2023]. Three-phase training datasets. We organize all synthetic data into three stage-specific datasets: Dalign =
[
{G(c, simp) ⊕ c}
(2)
c∈Calign
[
{G(c, t) ⊕ c | t ∈ {detail, conv, reason}}
(3)
+ Drl = {(Q, A)}M j=1 , (Q, A) = G(c, reason ), c ∈ Crl
(4)
Dsft =
c∈Csft
where ⊕ concatenates c with the generated QA pair, and reason+ denotes challenge prompts requiring multi-step semantic inference. The instruction mixture in Dsft follows [Liu et al., 2024], whereas Drl requires complete reasoning steps and manual verification. Overall, D = {Dalign , Dsft , Drl } provides multi-granularity supervision for three-phase training, emphasizing obfuscation robustness and reasoning-intensive alignment. After stage-specific filtering and deduplication, Dalign , Dsft , and Drl contain 430,027 simp, 11,750 detail/conv/reason, and 4,244 manually verified reason+ instances, respectively. Details are provided in Appendix B.
4
Methodology
We present OASIF, an effective obfuscation-aware framework for improving LLMs in assembly instruction following and comprehension. As shown in Fig. 1, OASIF combines an assembly encoder, a lightweight projector, and a pretrained code LLM with a compact three-phase training pipeline. The training pipeline consists of three phases: feature-space alignment (Phase I), supervised instruction fine-tuning (Phase II), and Online Self-Evolving Reinforcement Learning with hybrid rewards (Phase III). Phases I–II establish stable assembly–language alignment and basic instruction-following ability, while Phase III further enhances the model through self-evolving RL, enabling continual refinement without relying on abundant ground-truth annotations for commercially obfuscated code. 4.1
Framework Structure
Token-Efficient Assembly Representation. Obfuscation typically inflates code length and disrupts local regularities, resulting in extremely long sequences that exceed the context window of most open-source LLMs. Consequently, given an obfuscated snippet and assembly codes c, directly serializing it into a textual sequence for LLMs is impractical. To address this challenge, we employ a dedicated assembly encoder Fenc to compress obfuscated assembly into a compact representation with a fixed-dimensional space Rdenc , and further align this representation with the linguistic space of the LLM via an MLP-based projection W : Rdenc → Rdllm , as shown in Eq. (5), ensuring compatibility with limited context windows while preserving the essential structural and semantic information of the original assembly code. Zc = W Fenc (c) , Fenc (c) ∈ Rdenc , Zc ∈ Rdllm
(5)
Instruction Decoding with Special Tokens. After obtaining the compact assembly representation Zc , we condition the LLM on both encoded assembly and language instructions to generate analyzes. To explicitly structure the input, we introduce special tokens instead of treating assembly and instructions as homogeneous text. In particular, <inst start> and <inst end> delimit each instruction instance, separating prompt components and stabilizing instruction-conditioned decoding. 4
4.2 Phases I & II: Supervised Knowledge Acquisition In this subsection, we describe the training procedures for the projector and the instruction following capability of the LLM. The first two training phases are intentionally designed to be lightweight, providing a solid and efficient initialization for the subsequent self-evolution RL stage. 4.2.1 Phase I: Feature Space Alignment In the first phase, we freeze the LLM and only optimize the projector to align assembly features with the semantic space of the LLM. The projector is trained using the simplified summary y generated by the frozen LLM from either the obfuscated or the original assembly representation Zc , as specified by the corresponding objective Eq. (6). This ensures that the assembly token embedding acquires a direct and consistent semantic interpretation within the frozen LLM. " Lalign = Ec∼Dalign −
L X
# log P (yi | Zc , y<i )
(6)
i=1
Dalign is the input distribution used for alignment, and L denotes the length of the output summary y. 4.2.2 Phase II: Instruction Fine-Tuning Subsequently, the projector and the LLM are jointly optimized on instruction following data while keeping the assembly encoder frozen. As formulated in Eq. (6), this phase maintains the same autoregressive language-modeling loss as Phase I. This training process enables the LLM to internalize the semantics of obfuscated assembly and attain a foundational understanding of obfuscated assembly code, thereby establishing a robust precursor for the subsequent self-evolution stage. 4.3 4.3.1
Phase III: Self-Evolving RL with hybrid rewards Self-Evolving RL
Phases I–II establish robust general behaviors, commercial obfuscation remains challenging. Given the limited context length of open-source LLMs and the impracticality of collecting human annotations for long obfuscated-code analyses, we adopt a self-evolving RL scheme to enhance comprehension of obfuscated assembly. For each obfuscated assembly snippet c and its corresponding instruction q, the model samples multiple candidate analyses, scores each candidate from multiple perspectives. These scores are then used to optimize the model itself. 4.3.2
Hybrid Rewards
To provide reliable learning signals under heavily obfuscated assembly code, we redesign the reward function as a combination of structural reward and semantic reward. Structural Awareness Reward. The structural reward encourages the model to accurately identify obfuscation techniques while strictly penalizing the misclassification of obfuscated code as benign (ORG). Specifically, the reward is set to 1 for a correct prediction of the specific obfuscation type, 0.2 for the binary detection of whether obfuscation exists, and 0 otherwise. Given the predicted class t̂, the ground-truth tgt , and the set of obfuscation types T , the reward Rstruc is formally defined as: if t̂ = tgt 1 Rstruc = 0.2 if t̂ ̸= tgt and (t̂, tgt ∈ T or t̂, tgt ∈ / T) 0 otherwise
(7)
Semantic Reward. Relying solely on the correctness of the final output yields a sparse learning signal. To provide denser supervision, we introduce a semantic reward Rsem over the generated reasoning process. The model evaluates its own reasoning steps against the original unobfuscated assembly and the reference metadata along five dimensions, i.e., helpfulness, relevance, accuracy, detail, and comprehensiveness, from which Rsem is computed. Total Reward. The overall reward is a combination of structural and semantic components, weighted by a hyperparameter λ and conditioned on the model’s generated response y: Rtotal = λRstruc (t̂, tgt ) + (1 − λ)Rsem (c, q, y)
(8)
4.3.3 Group Relative Policy Optimization We optimize our model using GRPO [Shao et al., 2024]. For each sample, the model scores its own K sampled trajectories to obtain rewards. We normalize rewards within each group and directly use the resulting standardized advantage in the GRPO objective: 5
1 JGRPO (θ) = E(c,q)∼Drl K
PK
k=1
h
i clip ππθθ (y(ykk|c,q) · Rk −mean(R) , 1 − ϵ, 1 + ϵ − β DKL (πθ ∥ πref ) |c,q) std(R)
(9)
old
Rk is the reward of the k-th trajectory and R = {R1 , R2 , . . . , RK } denotes the set of trajectory rewards in the group. πθ and πθold are the new and old policies, and πref is the reference policy.
5
Experimental Setup
5.1
Evaluation Benchmarks and Metrics
Commercial Black-box VM-based Obfuscator Comprehension. We evaluate on VMISA [Li et al., 2022], which tests whether models can recover the semantics of proprietary virtual instructions that are entirely unseen during training, making it a challenging out-of-distribution setting. It includes binaries protected by Code Virtualizer [Cod, 2019], Themida (v3.0.7) [The, 2019], and VMProtect (v3.5) [VMP, 2019]. We report the number of recovered k × 100%. Details instructions k from the virtual instruction set (ISA) and the success rate SR = |ISA| are provided in Appendix C. Obfuscation-Aware Assembly-Language Instruction Following. We introduce OASIF-Bench, the benchmark dedicated to LLM-based obfuscated assembly comprehension and instruction following. We randomly select 30 domain-diverse assembly snippets from the BinaryCorp-3M test split, ensuring no overlap with the training data, while covering cryptography, malware, and protocol implementations. Following Sec. 3.2, we expand them into 150 snippets using five types (ORG, BCF, FLA, SUB and ALL). The resulting dataset contains 450 expert-curated queries spanning conversation, description, and reasoning tasks. All queries and reference answers are independently analyzed and cross-validated by five binary-analysis experts. Following the evaluation protocol in LLaVA [Liu et al., 2024] and ASMA-Tune [Wang et al., 2025], GPT-5.4 serves as the evaluator and scores model outputs on a 1–10 scale across five dimensions (helpfulness, relevance, accuracy, detail, and comprehensiveness) relative to an expert-derived upper bound. Scores are normalized to a 0–100 scale for reporting, and each sample is scored over three independent trials to stabilize the estimate. Binary Code Similarity Detection (BCSD). For binary code similarity detection, we use seven established benchmarks widely adopted in prior BCSD research [Su et al., 2024, Jiang et al., 2023, Wang et al., 2025]: Curl, Coreutils, Binutils, ImageMagick, SQLite, OpenSSL, and Putty. Consistent with prior work, we report Recall@1 under varying pool sizes to measure the percentage of queries for which the ground-truth candidate is ranked first, and Mean Reciprocal Rank (MRR) as the average reciprocal rank of the correct candidate. Domain-Relevant Capability Preservation. To verify that OASIF preserves capabilities relevant to code generation and downstream binary-analysis tasks, we further evaluate on three non-overlapping benchmarks: HumanEval [Chen et al., 2021], VulBench [Gao et al., 2023], and HumanEvalDecompile [Tan et al., 2024]. We follow the official settings of each benchmark and report Pass@1 for HumanEval, Accuracy/F1/Precision/Recall for VulBench, and Compile Rate/Run Rate (Pass@1) for HumanEval-Decompile. Details of the dataset and optimization levels are provided in Appendix D. 5.2 Baselines We compare our framework against a representative suite of state-of-the-art LLMs: (1) Proprietary LLMs: GPT-5, GPT-4-Turbo, Claude-Sonnet-4-5, Claude-Opus-4-6, and Gemini-2.5-Pro. (2) Open-Source Code LLMs: Qwen2.5-Coder-Instruct (7B, 14B), Qwen3-Coder-Plus, DeepSeekCoder-V2-Lite-Instruct-16B [Zhu et al., 2024], DeepSeek-Coder-Instruct-6.7B, GPT-oss-120B. 5.3 Model Instantiation and Training Settings We instantiate OASIF using three representative backbones: Qwen2.5-Coder-Instruct-14B, Qwen2.5Coder-Instruct-7B, and DeepSeek-Coder-Instruct-6.7B. The architecture comprises three components: an Assembly Code Encoder (CLAP-ASM [Wang et al., 2024], 110M parameters), a Connector (singlelayer MLP, 30M parameters), and the LLM backbone. Dimension alignment adapts the projector output to each LLM’s hidden size (e.g., 3,584 for Qwen2.5-7B and 4,096 for DeepSeek-6.7B). Training is conducted in three stages. Phase I (Alignment) is trained for 1 epoch on 8 NVIDIA H20 GPUs (learning rate 1e-5, global batch size 1024). Phase II (SFT) is fine-tuned for 1 epoch on 8 NVIDIA H20 GPUs (learning rate 1e-5, global batch size 32). Phase III (RL) is optimized for 3 6
Table 1: Results on VMISA-Bench and OASIF-Bench. Best result per column among proprietary models and open-source models (including OASIF) are bolded. Model
VMISA-Bench (k (SR%)) Code Themida VMProtect Virtualizer v3.0.7 v3.5
OASIF-Bench ORG
BCF
FLA
SUB
ALL
Avg
Proprietary Large Language Models GPT-5
89 (64.5)
45 (32.6)
153 (64.6)
76.8
79.1
80.0
77.0
74.0
77.4
Claude-Sonnet-4-5
104 (75.4)
57 (41.3)
105 (44.3)
75.6
68.0
79.7
59.6
67.6
70.1
Claude-Opus-4-6
101 (73.2)
56 (40.6)
144 (60.8)
80.2
76.9
78.5
67.2
74.8
75.5
Gemini-2.5-Pro
98 (71.0)
41 (29.7)
183 (77.2)
69.0
66.4
64.2
72.0
59.8
66.3
Open-Source Code Large Language Models Qwen2.5-Coder-Instruct-7B
40 (29.0)
5 (3.6)
46 (19.4)
37.8
35.8
41.8
37.7
41.8
39.0
Qwen2.5-Coder-Instruct-14B
60 (43.5)
12 (8.7)
78 (32.9)
56.2
54.8
55.0
51.7
55.8
54.7
Qwen3-Coder-Plus
81 (58.7)
23 (16.7)
74 (31.2)
56.1
60.6
69.3
51.6
63.8
60.3
DeepSeek-Coder-Instruct-6.7B
32 (23.2)
7 (5.1)
52 (21.9)
37.1
38.4
39.6
30.6
30.4
35.2
DeepSeek-Coder-V2-Lite-Instruct-16B 74 (53.6)
22 (15.9)
95 (40.1)
58.0
52.0
56.0
50.0
51.0
53.4
GPT-oss-120B
14 (10.1)
92 (38.8)
73.6
68.7
72.0
64.7
68.4
69.5
42 (30.4)
OASIF (Ours) Qwen2.5-Coder-Instruct-7B-OASIF
46 (33.3) ↑4.3 8 (5.8) ↑2.2
Qwen2.5-Coder-Instruct-14B-OASIF 82 (59.4)
↑15.9
20 (14.5)
56 (23.6) ↑4.2
↑5.8
118 (49.8)
↑16.9
49.6 ↑11.8 38.6 ↑2.8 44.4 ↑2.6 46.4 ↑8.7 49.8 ↑8.0 45.8 ↑6.8 62.0 ↑5.8 61.7 ↑6.9 68.4 ↑13.4 64.8 ↑13.1 65.8 ↑10.0 64.5 ↑9.8
DeepSeek-Coder-Instruct-6.7B-OASIF 52 (37.7) ↑14.5 13 (9.4) ↑4.3 82 (34.6) ↑12.7 52.4 ↑15.3 35.0 ↓3.4 55.8 ↑16.2 40.9 ↑10.3 38.2 ↑7.8 44.5 ↑9.3
epochs on 32 NVIDIA H20 GPUs, with 8 rollouts per instruction, a temperature of 0.7, top-p 0.9, ϵ=0.2, β=0.04, and a maximum sequence length of Lmax =4096. λ is empirically set to 0.5 to balance the reward signals. 5.4 Implementation Details Inference is performed with a temperature of 0 and top-k of 1 to ensure deterministic outputs. Prompts and data used for evaluation are strictly excluded from the retrieval corpus generation. BCSD follows the established few-shot CoT protocol from prior work [Wang et al., 2025] to enable direct comparison of both general-purpose and specialized code LLMs. Under the deterministic decoding setting, small numerical differences are highly reproducible and are unlikely to be attributable to sampling variance.
6
Experiments
6.1 VMISA-Bench Evaluation Tab. 1 reports the performance of OASIF and baseline models on VMISA-Bench, which constitutes a challenging out-of-distribution setting since the virtual ISAs are entirely unseen during training. OASIF consistently improves open-source LLMs across all three commercial VM obfuscators, with larger gains on smaller backbones, and remains effective under different VM instruction transformation regimes. Across Code Virtualizer, Themida, and VMProtect, OASIF-equipped models exhibit clear and stable gains over their corresponding backbones. For example, Qwen2.5Coder-Instruct-14B improves from 43.5% to 59.4% SR on Code Virtualizer (+15.9), from 8.7% to 14.5% on Themida (+5.8), and from 32.9% to 49.8% on VMProtect (+16.9). Similar trends are observed for the 7B Qwen and DeepSeek-6.7B variants, indicating effectiveness across model scales and architectures, with diminishing returns as backbone capacity increases. The gains remain pronounced even on VMProtect, which combines one-to-many (O2M) and one-to-one (O2O) instruction mappings and semantically obfuscates context-switch instructions, requiring reasoning over multiple handler compositions for a single native instruction. Under this more complex regime, Qwen2.5-Coder-Instruct-14B-OASIF approaches or surpasses the success rates of some closed-source models on VMProtect despite using substantially fewer parameters in an open-source setting. These results indicate stronger obfuscation-aware comprehension rather than memorization of fixed mapping rules. By integrating token-efficient assembly representations with self-evolving reinforcement learning, OASIF improves generalization to unseen proprietary VM-based obfuscation and enhances compositional reasoning over virtual handlers in difficult OOD settings. 6.2 OASIF-Bench Evaluation Tab. 1 reports the performance of OASIF and baseline models on OASIF-Bench. 7