arXiv:2605.09961v1 [cs.CR] 11 May 2026
T OWARDS LLM-BASED A NALYSIS OF V IRTUALIZATION -O BFUSCATED C ODE THROUGH AUTOMATED DATA G ENERATION
Sangjun An Chungnam National University Daejeon, Republic of Korea [email protected]
Hyeyeon Park Chungnam National University Daejeon, Republic of Korea [email protected]
Seoksu Lee Chungnam National University Daejeon, Republic of Korea [email protected]
Yejin Son Chungnam National University Daejeon, Republic of Korea [email protected]
Eun-Sun Cho Chungnam National University Daejeon, Republic of Korea [email protected]
A BSTRACT Virtualization-based obfuscation produces extremely large and structurally complex binaries, posing challenges for LLM-based analysis due to input size limits and the need for large-scale labeled data. We address this by focusing on structural rather than full semantic analysis. Obfuscated binaries are decomposed into the largest semantically coherent units that fit within LLM constraints and are labeled according to their structural roles. We implement a static analysis framework to automate labeling and enable large-scale dataset generation. Our prototype shows strong performance on real-world virtualization obfuscators. Keywords Code Virtualization · Control Flow Graph · LLVM Pass · Natural Language Processing · Static Analysis
1
Introduction
Software obfuscation techniques [1] continue to evolve, and virtualization obfuscation is considered one of the most powerful approaches [2, 3, 4, 5]. It translates original instructions into virtual instructions executed by an interpreter, significantly complicating static analysis. As such techniques are increasingly used to evade malware detection, efficiently understanding the internal logic of virtualized binaries has become a critical challenge. Traditional static analysis struggles with virtualization, leading analysts to rely on dynamic approaches. However, dynamic analysis requires bypassing anti-reversing mechanisms and suffers from limited code coverage [15, 16, 17]. We propose a framework for statically identifying the core structures of virtualization-obfuscated binaries. Unlike prior CNN-based detection methods that primarily focus on binary classification (i.e., identifying the presence of obfuscation) through low-level patterns [14], our approach leverages Large Language Models (LLMs) to capture the contextual semantics of assembly instructions. This shift in perspective enables not just detection, but a granular structural identification of internal Virtual Machine (VM) components—such as dispatchers and handlers—which is essential for deep reverse engineering and de-obfuscation. Applying LLMs in this specialized domain, however, introduces two critical challenges: (1) Data Scarcity: High-quality, large-scale labeled datasets for virtualization structures are non-existent and costly to manually annotate. (2) Input Size Constraints: The sheer verbosity of virtualized binaries often exceeds the sequence length limits of standard LLM architectures. To address these issues, we propose an automated pipeline that leverages IR-level analysis to generate large-scale, high-fidelity training data. By segmenting assembly code into semantically coherent units, our framework satisfies
Figure 1: CFG before and after virtualization obfuscation.
LLM input constraints while preserving the functional relationships between instructions. The resulting model achieves high precision in identifying core virtualization structures and facilitates CFG-based visualization, providing analysts with actionable insights into the obfuscated execution flow. The main contributions of this work are as follows: • We design an automated labeling framework that overcomes the data scarcity problem by generating large-scale ground-truth datasets for virtualization analysis. • We develop a BERT-based identification model that achieves 99.8% accuracy in classifying key virtualization components (Dispatcher, Handler, etc.), even in optimized binaries. • We provide a visualization-aided analysis tool that reconstructs the logical execution flow of virtualized code, significantly reducing the manual effort required for reverse engineering.
2
Analysis of Virtualization Obfuscation Logic and Framework Design
2.1
Structural Characteristics of Virtualization Obfuscation
Virtualization obfuscation transforms original instructions into virtual instructions executed by an interpreter [5, 11], significantly altering the program’s control flow structure. Understanding its internal organization requires identifying three core components: • VPC (Virtual Program Counter): Points to the current position within the virtual instruction array. • Dispatcher: Reads the instruction referenced by the VPC and transfers control to the corresponding handler. • Handler: Implements fragments of the original logic and returns control to the dispatcher or another handler. 2
Fig. 1 illustrates the structural transformation under virtualization. The dispatcher serves as a central hub, while handlers implement the decomposed execution logic. Although dispatchers are commonly loop-based, recent obfuscators employ threaded designs to hinder detection. Our framework handles both structures. 2.2
Core Structure Identification Framework
Recovering the logic of virtualization-obfuscated binaries requires distinguishing VM-related blocks from mixed control-flow regions. We therefore design a framework composed of automated structural labeling, LLM-based block classification, and CFG visualization [10]. Our framework consists of three stages. First, ground-truth labels are automatically generated during obfuscation to enable large-scale dataset construction. Second, a BERT-based model is trained to learn contextual relationships between assembly instructions and classify block roles. Finally, the identified blocks are connected to reconstruct and visualize the virtualized execution flow. We define four core block types within virtualized regions: • Dispatch Start: Entry point of the virtualization loop. • Handler: Blocks implementing decomposed operations. • VM Start: Transition point from native to virtualized execution. • VM End: Exit point returning to native execution.
3
Automated Labeling and Dataset Construction
This section describes the automated extraction of structural roles from virtualization-obfuscated code and their conversion into training data. Direct analysis at the binary level is highly complex; therefore, we identify virtualization structures at an intermediate representation stage to enable scalable ground-truth labeling for large datasets. 3.1
Data Collection and Processing
To improve generalization, we constructed a dataset using diverse source programs and obfuscation configurations. Our framework consists of three stages. First, ground-truth labels are automatically generated during obfuscation to enable large-scale dataset construction. Second, a BERT-based model is trained to learn contextual relationships between assembly instructions and classify block roles. Finally, the identified blocks are connected to reconstruct and visualize the virtualized execution flow. 3.2
Automated Structural Labeling
We implement an automated pass that traverses the Control Flow Graph (CFG) and assigns structural roles to basic blocks. We design an automated structural labeling pass that analyzes control flows at the intermediate representation level to capture virtualization-specific control patterns. After evaluating multiple structural heuristics, we found that identifying blocks based on successor distribution provides the most stable criterion across different obfuscation settings. Algorithm 1: Dispatcher Identification candidate ← null, maxSuccs ← 0 for each BasicBlock BB in function F do n ← number of successors of BB if n > maxSuccs then maxSuccs ← n; candidate ← BB end if end for return candidate • Dispatch Start: The block with the maximum out-degree is identified as the dispatcher, as described in Algorithm 1. 3
• Handler: Blocks directly reachable from the dispatcher are labeled as handlers. • VM Start/End: The predecessor of the dispatcher is marked as VM Start, and exits from the virtualization region are marked as VM End. To preserve labels in the generated assembly, lightweight marker calls are inserted into identified blocks. Note that the proposed framework identifies virtualization structures at the intermediate representation (IR) level to enable scalable ground-truth labeling for large-scale datasets. Although this stage requires access to the obfuscation process, the resulting model is trained on raw assembly sequences. Consequently, once the training is complete, the model can be applied to black-box binaries where source code or IR is unavailable, fulfilling the practical requirements of malware analysis. 3.3
Data Preprocessing
Because LLMs impose strict token limits, entire obfuscated binaries cannot be processed as single inputs. We therefore segment assembly code into basic-block units and merge adjacent blocks sharing the same label to maintain semantic continuity. Oversized blocks are further subdivided. This strategy reduces input length by an average of 98.53%, enabling efficient use of the model’s token capacity while preserving structural semantics.
4
Experimental Results and Analysis
We developed a prototype implementation of the proposed framework to validate its feasibility. The prototype evaluates the structural labeling stage and the LLM-based classification performance. It first verifies whether automated structural analysis can correctly identify virtualization components generated by Tigress, then uses the resulting labels to fine-tune a BERT model and reconstruct CFG visualizations from the inferred results.
Figure 2: Overall framework of the proposed automated labeling and LLM-based analysis pipeline.
4.1
Structural Identification Results
We evaluated three Tigress dispatch modes (Switch, Direct, and Indirect), which significantly affect control-flow structure. As shown in Fig. 3, Switch employs a loop-switch dispatcher, whereas Direct and Indirect adopt threaded control transfers. Our framework successfully identified dispatch structures across all configurations. Interestingly, even in threaded implementations, compiler optimizations often introduce centralized hub blocks, enabling consistent detection. Table 1 summarizes identification results across three benchmark programs. Under -O0, all four core structures (VM Start, Dispatch, Handler, VM End) were correctly detected. Under higher optimization levels, branch merging occasionally obscured VM Start and VM End boundaries; however, Dispatch and Handler detection remained robust. Higher optimization levels occasionally obscure VM Start and VM End boundaries due to compiler-induced branch 4
Figure 3: Architectural comparison of VM structures based on dispatch options: Loop-Switch versus Threaded. merging. However, the identification of Dispatch and Handler blocks remains robust across all configurations. Since the primary goal of de-obfuscation is to understand the core execution logic—driven by dispatchers and handlers—our framework provides significant actionable insights even when boundary markers are optimized away. Table 1: Identification Results by Optimization and Dispatch Options Opt. -O0 Others Role switch direct indirect switch direct indirect VM Start ✓ ✓ ✓ X X X Dispatch ✓ ✓ ✓ ✓ ✓ ✓ Bubble Sort Handler ✓ ✓ ✓ ✓ ✓ ✓ VM End ✓ ✓ ✓ X X X VM Start ✓ ✓ ✓ X X X Dispatch ✓ ✓ ✓ ✓ ✓ ✓ Factorial Handler ✓ ✓ ✓ ✓ ✓ ✓ VM End ✓ ✓ ✓ X X X VM Start ✓ ✓ ✓ X X X Dispatch ✓ ✓ ✓ ✓ ✓ ✓ Fibonacci Handler ✓ ✓ ✓ ✓ ✓ ✓ VM End ✓ ✓ ✓ X X X
Algorithm
4.2
Model Performance
The dataset contains 24,010 samples across three dispatch options (Table 2 and 3). The BERT model was fine-tuned using multi-task learning to classify both dispatch types and block roles. We compared BertTokenizer and Palmtree. As shown in Table 4, training time was similar (approximately two hours), but BertTokenizer achieved significantly higher main-label accuracy (91.7% vs. 83.8%) while maintaining comparable sub-label performance (99.8% vs. 99.5%). Class-wise evaluation results are summarized in Tables 5 and 6. BertTokenizer achieved a macro F1-score of 0.998, slightly outperforming Palmtree (0.996). Both models exceeded 99% accuracy across most classes, indicating strong structural discriminability. To examine the effect of pre-training, we additionally evaluated models without pre-trained weights. The performance gap was minimal (approximately 0.001 in macro F1), suggesting that tokenizer characteristics have greater influence than pre-training initialization in this task in general. 5
Table 2: Data Distribution by Dispatch Option Dispatch Option Data Switch 7,996 Direct 8,002 Indirect 8,012 Total 24,010
Table 3: Detailed Dataset Composition for Experiments Category Single Sub-label Data per Main Label (6 sub-labels) Total Exp. Data (3 main labels) Train/Val Set (80%) Test Set (20%)
Quantity Remarks 7,000 Extracted per sub-label within a main label 42,000
7, 000 × 6
126,000
42, 000 × 3
100,800 25,200
Model training and tuning Final performance evaluation
Table 4: Performance Comparison between Tokenizers Training Main Label Sub Label Tokenizer Time Accuracy (%) Accuracy (%) BertTokenizer
2h 15m
91.7
99.8
Palmtree
2h 10m
83.8
99.5
Notably, BertTokenizer outperformed Palmtree (91.7% vs. 83.8%), while asm2vec showed comparable results and is omitted for brevity. We attribute this to domain mismatch: assembly embedding models are pre-trained on humanwritten binaries, whereas virtualization obfuscators generate repetitive and synthetic structural patterns. In contrast, BERT’s flexible subword tokenization better captures the textual fingerprints of obfuscation logic during fine-tuning. Table 5: Classification performance of BertTokenizer by class. Class HANDLER VM VM-START NON-VM VM-END DISPATCH-START Macro avg
Precision 0.9983 0.9969 0.9998 0.9905 1.0000 0.9998 0.9976
Recall 0.9993 0.9919 0.9995 0.9962 0.9990 0.9993 0.9975
F1-score 0.9988 0.9944 0.9996 0.9934 0.9995 0.9995 0.9975
Support 4,200 4,200 4,200 4,200 4,139 4,200 25,139
Table 6: Classification performance of Palmtree by class. Class HANDLER VM VM-START NON-VM VM-END DISPATCH-START Macro avg
4.3
Precision 0.9978 0.9922 0.9976 0.9957 0.9921 1.0000 0.9959
Recall 0.9919 0.9974 0.9988 0.9888 0.9990 0.9995 0.9959
F1-score 0.9949 0.9948 0.9982 0.9922 0.9955 0.9998 0.9959
Support 4,200 4,200 4,200 4,200 4,139 4,200 25,139
CFG Visualization
Finally, based on BERT inference results, we reconstructed a color-coded CFG (Fig. 4) to visualize virtualization execution structure. VM and non-VM regions are clearly separated, and dispatcher-handler relationships become readily interpretable. 6
Figure 4: Complete visualized CFG with color-coded virtualization components identified by BERT.
5
Conclusion
In this study, we proposed an LLM-based framework for automatically identifying and visualizing the execution structures of virtualization-obfuscated code. First, we established an automated structural labeling pipeline that enables large-scale dataset construction without manual intervention. This approach provides a practical methodology for generating reliable training data in virtualization obfuscation research. Second, we demonstrated high identification performance using a BERT-based model. By learning contextual relationships between assembly instructions, the model classified core components such as dispatchers and handlers with up to 99.8% accuracy, outperforming traditional rule-based detection approaches and adapting robustly to different dispatch configurations. Third, we enhanced analysis interpretability through CFG-based visualization of identified virtualization regions. By highlighting VM entry/exit points and dispatcher-handler relationships, the framework allows analysts to directly grasp the core execution logic of obfuscated programs. Although our experiments focused on Tigress, the proposed structure-oriented learning framework is extensible to other virtualization-based obfuscators such as VMProtect and Code Virtualizer. Future work will extend this approach beyond visualization toward semantic recovery and automated de-obfuscation of virtualized code. Our prototype and dataset will be made publicly available to support reproducible research.
References [1] C. Collberg, C. Thomborson, and D. Low, “A taxonomy of obfuscating transformations,” Department of Computer Science, University of Auckland, New Zealand, Tech. Rep. 148, 1997. [2] V. J. M. Salis and A. S. Sani, “A survey of code obfuscation techniques,” ACM Comput. Surv., vol. 53, no. 3, pp. 1–36, 2020. [3] N. Zhang, et al., “Inspecting Virtual Machine Diversification Inside Virtualization Obfuscation,” 2025 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA, 2025, pp. 3051–3069, doi: 10.1109/SP61157.2025.00071. [4] S. Li, et al., “Chosen-Instruction Attack Against Commercial Code Virtualization Obfuscators,” 29th Annual Network and Distributed System Security Symposium, NDSS 2022, San Diego, California, USA, April 24–28, 2022. [5] J. Teschke and T. Kurth, “Virtualization-based obfuscation: A survey,” in Proc. 12th Int. Conf. Availability, Rel. Secur. (ARES), 2017, pp. 1–10. [6] B. Kinder, “The tigress diversifying c compiler,” 2023. [Online]. Available: http://tigress.cs.arizona.edu/ [7] C. Lattner and V. Adve, “LLVM: A compilation framework for lifelong program analysis & transformation,” in Proc. Int. Symp. Code Gener. Optim. (CGO), 2004, pp. 75–86. 7
[8] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” 2018, arXiv:1810.04805. [Online]. Available: https://arxiv.org/abs/1810.04805 [9] B. Li, S. Feng, and G. Tan, “Palmtree: Learning an assembly language model for instruction embedding,” in Proc. 2021 ACM SIGSAC Conf. Comput. Commun. Secur. (CCS), 2021, pp. 3236–3251. [10] H. Pei, B. G. Chun, and S. Kim, “DeepBinDiff: Learning program-wide code representations for binary diffing,” in Proc. 27th Network Distrib. Syst. Secur. Symp. (NDSS), 2020. [11] G. Luan, S. Feng, and G. Tan, “Identifying virtualized code via deep learning,” in Proc. 36th IEEE/ACM Int. Conf. Autom. Softw. Eng. (ASE), 2021, pp. 1178–1189. [12] X. Wang, J. Yang, and Z. Lin, “BinBert: Binary code understanding with BERT,” in Proc. 32nd IEEE Int. Conf. Softw. Maint. Evol. (ICSME), 2022. [13] S. Vaswani et al., “Attention is all you need,” in Proc. 31st Int. Conf. Neural Inf. Process. Syst. (NIPS), 2017, pp. 5998–6008. [14] D. Yoo et al., “CNN based Virtualized Obfuscated Malware Detection Technique,” in Proc. Symposium of the Korean Institute of Communications and Information Sciences, 2024, pp. 637–638. [15] S. Park and Y. Park, “Analysis of anti-reversing functionalities of vmprotect and bypass method using pin,” KIPS Trans. Comp. Comm. Syst., vol. 10, no. 11, pp. 297–304, 2021. [16] D. Xu et al., “VMHunt: A verifiable approach to partially-virtualized binary code simplification,” in Proc. ACM CCS, 2018, pp. 442–458. [17] A. Kalysch et al., “VMAttack: Deobfuscating virtualization-based packed binaries,” in Proc. ARES, 2017, pp. 1–10.
8