Challenges and Future Directions in Agentic Reverse Engineering Systems

Challenges and Future Directions in Agentic Reverse Engineering Systems Salem Radey, Jack West, Kassem Fawaz

arXiv:2604.14317v1 [cs.CR] 15 Apr 2026

University of Wisconsin-Madison Madison, USA {salem.radey, jwwest, kfawaz}@wisc.edu

Abstract—Agentic systems built on large language models (LLMs) are increasingly being used for complex security tasks, including binary reverse engineering (RE). Despite recent growth in popularity and capability, these systems continue to face limitations in realistic settings. Cutting-edge systems still fail in complex RE scenarios that involve obfuscation, timing, and unique architecture. In this work, we examine how agentic systems perform reverse engineering tasks with static, dynamic, and hybrid agents. Through an analysis of existing agentic tool usage, we identify several limitations, including token constraints, struggles with obfuscation, and a lack of program guardrails. From these findings, we outline current challenges and position future directions for system designers to overcome from a security perspective.

1. Introduction Agentic reverse engineering (RE) systems are collections of agents aimed to solve a given security task [1]. Each RE agent performs selected tasks in response to the system’s needs at a given time. These tasks range from redteaming [2], blue teaming [3], anomaly detection [4], and many other security tasks [1]. One of the most necessary capabilities for almost all RE problems is binary RE. Binary RE is the process of defining a binary’s functionality using its compiled code. There are three methods in security research for which a binary can be fully explored: (1) static analysis, (2) dynamic analysis, and (3) hybrid analysis. Static analysis attempts to define the inner workings of a binary without executing the program itself [5]. Dynamic analysis, on the other hand, is the process of analyzing the binary while it is running. Finally, hybrid analysis combines static and dynamic analysis to analyze a binary. Modern security research [6], [7] that uses RE often employs hybrid analysis, as it captures almost all functionality of a given binary. Agents in a system are compartmentalized to perform one or more of the three binary types [8], [9], [2]. However, the technical details of how these systems implement these processes are opaque. There is no all-encompassing algorithm to RE an arbitrary binary [10]. Depending on how the binary is constructed, it could completely change the tool set and approach an agent must make. For example, Android apps require two different decompilers to fully examine a given

app across two different runtimes [11]. Therefore, understanding the technical capabilities of an agentic system will define what binaries it can or cannot RE. In this work, we aim to define the current technical details for how agentic systems perform binary RE and lay out future directions and existing challenges that must be overcome. To define the details, we analyze open-source libraries for agentic RE systems to determine precisely how these systems perform binary RE. We divide binary RE into three categories: static, dynamic, and hybrid analysis. We also discuss 6 challenges and limitations that agents currently experience. We find that agents that perform static analysis are limited in capability. There are two main challenges agents currently face: (1) obfuscation and (2) tokenization. Obfuscated binaries cause confusion and unneeded complexity. Several studies have shown that obfuscation affects agent analysis [12] or that agents avoid obfuscated binaries entirely [13]. However, in realistic security scenarios, obfuscated binaries are common [12]. Binaries, especially after decompilation, are token-heavy [14], [11]. Limited tokens constrain the coverage an agent may have. Dynamic analysis, on the other hand, exhibits its own set of issues. We identify three core challenges: (1) lack of dynamic analysis guardrails, (2) timeouts, and (3) reliance on emulation. Upon reviewing codebases of existing agentic systems [15], [2], [16], we found that they commonly allow models to execute any command the agent deems necessary. RE tools are extremely powerful pieces of software, often requiring root-level access into compartmentalized memory spaces [17], [18], [19]. Recent work has shown [20], [21] that it is possible to trick an agent into executing malicious code. Thus, we argue that it may be possible to craft a binary that tricks a model into executing malicious code. Adversarial programs commonly employ timeouts to prevent processes from being monitored [22]. Given the time and latency agents naturally have, we anticipate that binaries will employ strict timeouts to block any dynamic analysis attempts. Adversarial applications also load dynamic binaries that may be encrypted or downloaded from an offdevice source, further complicating the timing of an agent’s analysis. When performing safe dynamic analysis, experts commonly load malicious programs into virtual machines (VMs). Agents also analyze binaries within virtual envi-

ronments as well for safe analysis [15], [16]. However, unlike their human RE counterparts, agents can be tricked into executing malicious code [23]. Furthermore, several virtual environments for edge devices have virtualization issues [24]. Finally, hybrid analysis is when an RE performs both static and dynamic analysis. Hybrid analysis shares the core challenges of static and dynamic analyses but also exhibits unique issues. The main challenge we identify in our research is that hybrid agents rely on a human-in-the-loop or perform all reasoning internally [25], [2], [16], [15]. Both approaches have their own issues. For example, humanin-the-loop is slow and requires an active RE to execute the instructions. Whereas, completely automated systems reason independently and may be susceptible to adversarially crafted binaries. We focus on automation issues since the end goal of an agentic system is to remove human dependency. In this work, we propose several research directions for static analysis agents, informed by our expert understanding of reverse engineering. First, we propose tokenizing static analysis based on byte patterns to lower the number of tokens the model may see. We also posit that exploring alternative decompilation methods within an agentic system could optimize an agent’s understanding of a given binary. Both byte-pattern filtering and ML model binary translation may result in lower tokenization. Furthermore, we argue that obfuscation must be accounted for in agentic RE systems. Work has shown that obfuscation reduces an agent’s capabilities; therefore, we must incorporate a deobfuscation component into the pipeline. Regarding dynamic analysis, we also provide researchers with directions for future exploration to improve agent performance. Our analysis reveals that agentic systems designers rely heavily on the model’s judgment for executing commands. This lack of guardrails could lead to system hijacking by constructing a clever binary. Therefore, we present two avenues for research: (1) an adversarial direction to create a binary that takes control of the host system by utilizing the lack of guardrails, and (2) building guardrails for dynamic analysis. Another set of problems in dynamic analysis includes timeouts and the lack of sequential processing. Both issues may limit an agent’s ability to perform dynamic analysis. Thus, researchers should design specialized tools to overcome the limitations of sequential real-time data processing for agents. Finally, for hybrid agents, we propose a stronger reasoning methodology to prevent the execution of malicious code. Current systems lack a voting infrastructure and safety checks to prevent an agent from performing dynamic analysis. Hybrid systems are extremely vulnerable to malicious code as they interact with it. Thus, further verification methods across static and dynamic analysis should be explored.

2. Taxonomy In this section, we detail our taxonomy when discussing reverse engineering. Our focus will be on agents that can

analyze a binary on a given device. Meaning, the agent has full access to a binary and unlimited resources to perform any type of analysis.

2.1. Reverse Engineering We examine papers that attempt to automate reverse engineering using agents. We define reverse engineering as the process of iteratively interacting with an arbitrary computer program. Specifically, we focus on reverse-engineering a given binary across multiple architectures. Therefore, our analysis will exclude reverse engineering that attempts to exploit binaries off a device on which the given agent exists. Reverse engineering can be described into three different types: (1) static, (2) dynamic, and (3) hybrid analysis. Static analysis is the process of examining a binary without executing it on the device. Dynamic analysis, on the other hand, is analyzing a binary while it is running. Hybrid analysis combines static and dynamic analysis to evaluate a binary.

2.2. AI Agent and Agentic Scope We define an AI agent as an LLM designed to solve a task, equipped with the tools to do so [25]. LLM agents plan and execute their own individual decisions by observing environmental feedback [1]. Agentic systems are collections of agents aiming to work together to accomplish a goal. In this work, we examine an agentic system where the goal is to reverse engineer a given binary. Rather than defining a threat model, we define the capabilities and goals of an arbitrary agent. First, we assume that the agent has no access to the source code of a given binary. We argue that the source code of a binary greatly simplifies RE tasks and is unrealistic for real-world security analysis. Our focus will purely be on local binary analysis. Some RE tasks require networking capabilities, such as port scanning or network traffic analysis. We do not discuss any RE task that requires networking. We assume no restrictions on the size, target device, and capabilities for a given binary. In all agents and agentic systems, we observed that they all decompile binaries. Decompiled binaries attempt to translate the bytes into a more human-readable form. Tools such as Ghidra [26] and IDA [27] translate binary code into a higher-level language. Agents that study decompiled binaries first preprocess the binaries to make them easier to read. We do not restrict agentic systems to a set of tools; any decompilation method is valid.

3. Overview of Agents This section presents a technical analysis of how agents perform various types of binary reverse engineering. Our focus is on the technical capabilities of the agents rather than their overall performance or goal. Several surveys have explored the diverse applications of LLM agents [1], [28], [29], [30], but our work focuses on their use in binary analysis from a security perspective. The focus on technical

Agent Capabilities Static

Dynamic APK

Ghidra/IDA usage

wasm

C

Various languages

Binary debuggers

Shell commands

Preprocessing

Emulation

Hybrid

Wide tool choice

Autonomy

Figure 1. An overview of agent capabilities by type.

capabilities is intended to clarify how these agents perform binary reverse engineering. We segment agents into two categories: static analysis and hybrid analysis. Static analysis agents are common because static analysis can assist with a wide range of tasks beyond RE. Therefore, the community’s understanding of static analysis agents exceeds that of hybrid agents. Our literature review found that no agent focuses on dynamic analysis; they would perform hybrid analysis. Thus, our discussion pertaining to hybrid analysis will detail current dynamic analysis capabilities. An overview of these capabilities can be seen in Figure 1.

3.1. Static Analysis Agents Static analysis is an essential component of any binary reverse engineering pipeline and is, therefore, commonly performed by agents [8], [25], [21], [31], [32], [9], [33], [34], [35], [36], [14]. We found that agents perform static analysis on decompiled binaries. 3.1.1. Decompiled Code. Binary decompilation is a common practice in reverse engineering [37]. The purpose of decompilation is to provide the human reverse engineer a readable interpretation of a given binary. The human reverse engineer then reads and interprets the outputs to determine the program’s goal. We find that decompilation is most commonly performed using Ghidra [26] or IDA Pro [27]. However, some studies also seek to train LLM agents to perform decompilation. Tan et al. [12] introduce LLM4Decompile, the first series of open source LLM trained for decompilation. They introduce both LLM4Decompile-End models which do decompilation directly, as well as LLM4Decompile-Ref models which refine Ghidra output. LLM4Decompile-End uses ObjDump to disassemble binaries prior to decompilation. Likewise, Rong et al. [38] present DISAS LLM, an endto-end LLM disassembler for highly obfuscated executables.

The system performs an initial disassembly and then uses an LLM-based classifier to validate decoded instructions. To improve efficiency, DISAS LLM batches LLM queries and outputs the corrected disassembly after all instructions are checked. Unlike LLM4Decompile, DISAS LLM is end-toend and does not interact with traditional decompilers such as Ghidra or IDA Pro. Other works explore web runtimes and languages such as WebAssembly (WASM). She et al. [39] present WA D EC, an approach that converts Wasm binaries to the textual wat format and performs block-wise decompilation using a fine-tuned LLM. By segmenting functions into Wasm loop blocks and applying a unified variable renaming scheme, WaDec improves scalability and readability, producing decompiled code with minimal code inflation. Other agents turn to JADx [40] when reverse engineering Android dex and APK files. Qian et al. [11] introduce LAMD: an LLM-powered framework that utilizes key context extraction and tier-wise code reasoning to detect malware more accurately than standalone LLMs. The LAMD framework uses existing models, with the paper evaluating GPT-4o-mini. However, since GPT-4o-mini and many other LLM cannot process APKs directly, the authors use JADx to decompile the code as a pre-processing step before sending it to the LLM. Since Android APK files consist of both C and Java code [41], there is additional pre-processing overhead to account for the differing methods required to decompile each language. Most studies tend to use IDA Pro or Ghidra. Shang et al. [42] introduce BinMetric, a benchmark of 1000 questions pertaining to common real-world binary analysis tasks. To obtain data to feed the LLM, the authors compile and strip the source code, then use IDA Pro to decompile and disassemble the compiled binaries. Although they choose to use IDA Pro, the authors note the prevalence of tools such as Ghidra, Radare2 [17], and Clang Static Analyzer [5] as static analysis tools. However, they note that the tools require significant manual effort and expert knowledge to use. Additionally, the authors use the SrcML tool [43] to extract information from the source files and convert it into XML format as additional data.

3.2. Hybrid Analysis Hybrid analysis combines static and dynamic analysis to leverage the strengths of both approaches when reverse engineering. We define a hybrid analysis agent as an agent that can use tools for both static and dynamic analysis. All agents we observed that perform dynamic analysis do so in tandem with static analysis. In this section, we provide an overview of dynamic agent capabilities and discuss hybrid implementations. 3.2.1. Dynamic Analysis Agents. Dynamic analysis is the process of observing code behavior at runtime. Due to dynamic code loading [22], many behaviors of a program are not inferable through static analysis and are only observed

during runtime. To observe these behaviors, reverse engineers use tools for dynamic analysis, including debuggers and dynamic instrumentation. Dynamic analysis allows a reverse engineer to define complex behaviors using captured values, execution traces, and breakpoints. Despite the advantages of dynamic analysis, it remains more difficult to scale than static analysis. Currently, the interactive and adaptive components of dynamic analysis make it difficult for LLMs to perform independently. Therefore, human analysts must still account for a large portion of the work [44]. Additionally, pipelines used for dynamic analysis are tightly coupled to the code’s architecture, making portability a challenge [6], [22], [7]. An example tool used for dynamic analysis is the GNU Debugger (GDB) [45], [46], [47]. GDB allows for step-bystep instruction-level analysis of a binary. Beyond stepping through code, GDB also features disassembly, which can be done at runtime, decompilation, the ability to view the contents of registers, and breakpoints [48]. Zhuo et al. [49] present CTF-Dojo, an executable runtime designed to train cybersecurity agents using CTF challenges. CTFDojo builds upon EnIGMA+ [50], [46] for its agent, which implements dynamic reverse engineering capabilities using GDB. EnIGMA+ implements GDB only with the commands break, which adds a breakpoint, continue, which continues execution of the binary, stepi, which steps to a specific line, and debug_exec, which allows for arbitrary GDB command execution[15]. Disassembly and decompilation are handled separately with Ghidra. Likewise, Muzsai et al. [2] introduce HackSynth, an LLM agent capable of autonomous penetration testing. HackSynth’s architecture consists of a planner and summarizer that iteratively generate and interpret bash commands within a containerized Linux environment. Unlike CTFDojo, HackSynth uses any bash command available to the agent. 3.2.2. Hybrid Implementations. While dynamic analysis agents appear only in hybrid systems, there are many ways to design a hybrid agent. There are two methods to perform hybrid analysis: (1) human-in-the-loop and (2) fully autonomous. Human-in-the-loop systems execute the agents’ instructions and verify their feasibility and security. The humans in this process are expected to run the commands so that the agent can execute commands safely. Whereas autonomous systems are left completely to their own devices when performing analysis. Meaning, all of the reasoning for RE comes from the collection of agents communicating. Deng et al. [8] present P ENTEST GPT, an LLMdriven framework for automated pentesting. Similar to HackSynth, P ENTEST GPT comprises multiple agents. Whereas HackSynth used a planner and summarizer, P EN TEST GPT employs reasoning, generation, and parsing agents. P ENTEST GPT does not enable tool usage; rather, it operates with a human-in-the-loop strategy, in which a human expert serves solely as an executor, strictly following the LLM’s suggestions. P ENTEST GPT represents a human-

in-the-loop system whereas HackSynth is completely automated. Another case of an agentic hybrid approach is by Pasquini et al. [21], which creates MANTIS: a defense against LLM-driven cyberattacks. MANTIS operates autonomously, responds to detected anomalous interactions, and employs both active (dynamic) and passive (static) defenses. The static component analyzes the attacker’s goals, constraints, and control logic to misdirect them. The dynamic component instruments a counterattack by leading the adversarial LLM to open a reverse shell on the adversary’s machine.

4. Current Challenges Reverse engineers face a broad range of challenges that stem from the increasing complexity of modern software. Obfuscation techniques, large-scale code bases, and dynamic runtime behavior all serve to complicate RE. These methods fundamentally limit the understanding an analyst can glean from a program’s bytes, structure, and behavior. As with many technical disciplines, reverse engineering is continually evolving. As RE improves, so do anti-RE techniques hindering models.

4.1. Static Analysis Challenges Static reverse engineering comes with a set of inherent challenges that analysts face regardless of their expertise. Techniques such as code obfuscation through symbol stripping, variable name changes, control flow changes, and comment removal obscure code semantics and impede understanding for both human and LLM reverse engineers. Additionally, the code bases of reversing targets are often large, which requires additional time and understanding to rationalize. 4.1.1. Obfuscation. Just as in non-agentic reverse engineering, obfuscation remains a hurdle for agentic systems. For agentic systems, obfuscation provides new challenges. Since LLM-based agents rely on their ability to detect syntax patterns they have been trained to recognize, previously unseen or unconventional obfuscation techniques disrupt the agents’ ability to function optimally [12]. This increases the likelihood of misclassifications or an incorrect prioritization of code importance. Tan et al. [12] find that Control Flow Flattening (CFF) and Bogus Control Flow (BCF) both decrease the success rate of decompilation by over 70%, showing that even with common obfuscation techniques, there exists a drastic decrease in reversing accuracy. This implies that complex obfuscation techniques could drop accuracies even lower. As a result, many works dealing with binary code choose to forgo including obfuscated binaries altogether, since they pose challenges for LLMs [51], [42], [44]. However, closedsource real-world applications commonly employ obfuscation techniques, meaning that when given an obfuscated binary, these agents are likely to fail.

4.1.2. Tokenization. LLM-based agents operate with a bounded context depending on their token limit, which constrains the amount of decompiled code that can be considered at once [11]. The natural workaround for this is to segment the input into chunks that fit the token limit, then analyze them sequentially. However, this implementation fails to account for long control flows that span the entire code and for functions or variables defined outside the current section. This lack of context impairs the agent’s ability to properly analyze the code’s relationships as a whole. Udeshi et al. [14] propose a framework using LLMs to generate summarizations of binary diffs for malware detection in supply chain scenarios. The study was conducted on 104 binaries using five LLMs and required token usage ranging from 100M to 500M per model. This study represents a best-case scenario, where the authors removed exact matches from the diff, thereby reducing token counts. However, this removal may affect an agent’s ability to fully understand the context of a binary. Current work demonstrates that larger binaries require a balance between tokens and functionality.

4.2. Dynamic Analysis Challenges Dynamic analysis poses challenges for agentic reverseengineering systems from both architectural and tooling perspectives. Current works demonstrate an over-reliance on shell commands, which may constrain the agent’s ability to perform more diverse runtime analysis and, depending on the implementation, introduce security risks. Additionally, LLM-based agents are hindered by their prolonged reasoning times, which may impact time-sensitive interactions and lead reverse engineers to reach incorrect conclusions due to timeouts and other unintended behaviors. In addition, lowlevel emulation environments are hard to emulate, limiting the dynamic analysis an agent can perform on niche hardware. 4.2.1. Lack of Guardrails. Some agents lack guardrails for the commands they are allowed to execute. One example of this is CTF-Dojo by Zhuo et al. [49] which builds upon E NIGMA + [50]. E NIGMA + allows unrestricted dynamic analysis on a given binary. This means an attacker could purposefully trick the agent into executing unsafe commands on the device. Another example is Muzsai et al. [2], who developed HackSynth, an LLM-based pentesting agent tested on CTF challenges. Similar to CTF-Dojo, HackSynth allows commands to be input without validation. 4.2.2. Timeouts. A key challenge for executing dynamic analysis with agentic systems arises from the variable processing times of LLMs. LLMs utilize multi-step reasoning and planning, which may require an extended period of time to ’think’. This inconsistency in execution time can trigger timeouts [22] or miss key interactions altogether. Reverse engineers interacting with these systems may draw spurious

conclusions from these timeouts, assuming the agent is operating under proper conditions. 4.2.3. Reliance on Emulation. Almost all existing agentic systems perform analysis on desktops. These systems use Docker and/or QEMU to create virtual machines for arbitrary devices and architectures. However, several edge devices are notoriously hard to simulate [24]. Furthermore, some operating systems require multiple application components compiled with different compilers [11]. To our knowledge, edge device RE is not currently being explored due to emulation restrictions.

4.3. Hybrid Analysis Challenges Hybrid analysis shares challenges with static and dynamic analysis. However, hybrid systems exhibit one major challenge regardless of technical capabilities: system reasoning. Binaries are not definable deterministically [10]. Therefore, unchecked reasoning in agentic systems is more likely to lead to failure on RE tasks for complex applications. For example, if an agent hyperfocuses on a specific region of the binary, which may be obvious to a human, it could waste hours of compute time without any human intervention. Another issue found in hybrid systems is the reliance on a human-in-the-loop. Human-in-the-loop adds time to the RE process and limits the model’s capabilities while retaining security and guidance. Interestingly, the challenges of full automation and Human-in-the-loop mirror each other. In this section, we describe the challenges associated with automation, as the goal for the community is automation rather than a human-in-the-loop approach. 4.3.1. Never Ending Automation. As with any cat-andmouse chase in security, the attacker adapts. If agentic systems attempt to fully automate RE practices, we foresee adversarial binary development to maximize the agent’s processing time. For example, new obfuscation techniques could emerge that force agents to triage long execution chains to extract a single piece of information. Rather than optimizing for human deobfuscation, the adversary will likely obfuscate to maximize hindrance to agentic communication and time.

5. Future Directions Agentic RE is still young. Many systems and agents are still vulnerable. Secure systems, including agentic ones, require solutions for edge cases to prevent system compromise. Below, we outline several research directions to address existing challenges.

5.1. Static Analysis Directions Our analysis indicates that static analysis agents are the most common. Agentic systems have several methods for understanding and interacting with decompiled code from

a given binary. However, several challenges exist. In this section, we detail potential ways to improve static analysis agents from a security researcher’s perspective. 5.1.1. Raw Binary Analysis. Currently, no agents deal with raw binary inputs. This is by design, as agents work with natural languages rather than bits and bytes. However, adversaries can exploit reliance on decompilation tools to hinder agentic reverse engineering. Since Ghidra and IDA deploy their own decompilation algorithms, it is possible to exploit the decompilation process to hide functionality [52]. Qin et al. [52] demonstrate an attack where an adversary can design assembly instructions that create a semantic deviation. One of those types is disassembly deviation, where, depending on the compiler, a set of instructions may yield different results. Thus, future work should explore alternative methods for binary decompilation such that they avoid or detect decompilation evasion. 5.1.2. Add Deobfuscation to Agentic Pipelines. Obfuscation is a challenge in RE for both scalable [7] and manual methods [6]. Agentic frameworks are no different [4], [32], with many frameworks forgoing [32] or lightly deobfuscating [4]. Thus, we argue that research should be done to allow agents to perform deobfuscation. Deobfuscating a binary will likely increase the accuracy of malware detection for agentic systems, as the system will spend less time understanding instructions. 5.1.3. Clever Tokenization. We found that most agentic systems that perform decompilation using Ghidra/IDA feed the output directly to the agent [8]. The problem with feeding decompilations to agents is that they contain repetitive information (e.g., stack frame construction, local variable allocation) that is irrelevant to the high-level functionality. Humans actively ignore repetitive functionality and focus on core logic for understanding the binary. LLM agents, on the other hand, may not ignore irrelevant blocks of instructions, which can lead to large tokenized decompilations and limit their capabilities. Some works, such as Udeshi et al. [14], cleverly leverage the fact that the agent is likely aware of open-source projects and can remove irrelevant functionality. While a clever solution, if presented with a binary that the system is unaware of, it may fail.

5.2. Dynamic Analysis Directions Work has shown that manipulating agents performing dynamic analysis is possible [21]. We anticipate that these attacks will become more complex as researchers explore techniques to interfere with agentic RE. This section describes potential solutions to the challenges in dynamic analysis. 5.2.1. Lack of Guardrails. One common theme among the agents we review is a lack of guardrails on their capabilities when exploring a binary. Binary analysis tools are extremely powerful and may allow for a cleverly crafted binary to

take control of a system. We propose that researchers attack agents during the dynamic analysis phase, as the binary serves as a dynamic adversary and can adjust its behavior in real time. It is expected that an adversarial binary has situational awareness and can adapt to human RE efforts. 5.2.2. Timeouts. Adversarial binaries employ strict timeouts when executing critical tasks to prevent dynamic analysis [22]. Agents are more susceptible to timeouts than their human counterparts due to network latency and limited thinking time. A binary built to attack high-resource systems could wrap important functionality around strict thread timeouts that easily defeat thinking models. A solution to this problem is for the agent to control the time-estimation mechanisms in the observed binary. However, this solution requires precise function hooks and control over which time value the program observes. Timeouts offer a unique opportunity to develop agent-specific RE tools that build around natural weaknesses agents have. If a human RE built a generalized tool to control the clock based on an agent’s response time, the agent could automatically adjust the clock to account for its thinking time while the binary remains unaware. 5.2.3. Improving Emulation Environments. While the technology does not currently exist, as a community, we should build stronger emulation environments for agentic systems. Several components could hypothetically be simulated using Gem5 [53], enabling deeper security analysis of hardware systems. Therefore, we propose building a similar tool that evaluates binaries on lesser-known hardware components that are challenging to emulate. This hypothetical tool will allow for scalable IoT analysis by an agentic system.

5.3. Hybrid Analysis Directions We argue that agents designed for hybrid analysis are susceptible to all of the challenges above as well as being vulnerable to their own. 5.3.1. Protection from Infinite Tool Calls. We need to design protections from novel obfuscation schemes designed to impede hybrid agents. The goal of the hypothetical scheme would be to force an agent to waste its time potentially indefinitely. While this does not yet exist, we argue that it may in the near future and must be addressed to ensure robustness and convergence.

6. Conclusion In this work, we examine how agentic systems perform binary reverse engineering across static, dynamic, and hybrid settings. Additionally, we identify 6 challenges within the space. For static analysis, we identify challenges with obfuscation and tokenization. To address these challenges, we propose a future research attempt to perform raw binary

analysis with agents and to add deobfuscation to agentic pipelines. For dynamic analysis, we note a lack of guardrails, issues stemming from timeouts, and a reliance on emulation. We outline future directions for binary design defenses, clock-based timeout protection, and improved emulation environments. Finally, we describe how hybrid agents may lead to an infinite sequence of tool calls. Our solution is to design defenses against infinite-agentic RE such that future obfuscation techniques will always converge.

References [1]

[2]

[3]

[4]

[5] [6]

[7]

[8]

[9]

A. Shahriar, M. N. Rahman, S. Ahmed, F. Sadeque, and M. R. Parvez, “A survey on agentic security: Applications, threats and defenses,” arXiv preprint arXiv:2510.06445, 2025. L. Muzsai, D. Imolai, and A. Lukács, “Hacksynth: Llm agent and evaluation framework for autonomous penetration testing,” 2024. [Online]. Available: https://arxiv.org/abs/2412.01778 X. He, D. Wu, Y. Zhai, and K. Sun, “SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systems,” May 2025, arXiv:2505.24201 [cs]. [Online]. Available: http://arxiv.org/abs/2505.24201 X. Chen, A. Zhou, C. Ye, and C. Zhang, “Clearagent: Agentic binary analysis for effective vulnerability detection,” in Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages, 2025, pp. 130–137. “Clang Static Analyzer — clang-analyzer.llvm.org,” https://clanganalyzer.llvm.org/. J. West, L. Thiemt, S. Ahmed, M. Bartig, K. Fawaz, and S. Banerjee, “A picture is worth 500 labels: A case study of demographic disparities in local machine learning models for instagram and tiktok,” in 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024, pp. 369–387. J. Cao, F. Guo, and Y. Qu, “Jnfuzz-droid: a lightweight fuzzing and taint analysis framework for native code of android applications,” Empirical Software Engineering, vol. 30, no. 5, p. 113, 2025. G. Deng, Y. Liu, A. Robotics, A.-A.-U. Klagenfurt, P. Liu, Y. Li, T. Zhang, Y. Liu, A.-A.-U. Klagenfurt, and S. Rass, “PentestGPt: Evaluating and Harnessing Large Language Models for Automated Penetration Testing.” R. Ghosh, H.-M. v. Stockhausen, M. Schmitt, G. M. Vasile, S. K. Karn, and O. Farri, “CVE-LLM: Ontology-Assisted Automatic Vulnerability Evaluation Using Large Language Models,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 28, pp. 28 757–28 765, Apr. 2025. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/35139

[10] D. Engel, F. Verbeek, and B. Ravindran, “On the decidability of disassembling binaries,” in International Symposium on Theoretical Aspects of Software Engineering. Springer, 2024, pp. 127–145. [11] X. Qian, X. Zheng, Y. He, S. Yang, and L. Cavallaro, “Lamd: Context-driven android malware detection and classification with llms,” 2025. [Online]. Available: https://arxiv.org/abs/2502.13055 [12] H. Tan, Q. Luo, J. Li, and Y. Zhang, “Llm4decompile: Decompiling binary code with large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024, p. 3473–3487. [Online]. Available: http://dx.doi.org/10.18653/v1/2024.emnlp-main.203 [13] L. Dramko, D. Bölöni-Turgut, C. Le Goues, and E. Schwartz, “Quantifying and mitigating the impact of obfuscations on machinelearning-based decompilation improvement,” in International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 2025, pp. 244–266.

[14] M. Udeshi, V. S. C. Putrevu, P. Krishnamurthy, P. Anantharaman, S. Carrick, R. Karri, and F. Khorrami, “Binary Diff Summarization using Large Language Models,” Sep. 2025, arXiv:2509.23970 [cs]. [Online]. Available: http://arxiv.org/abs/2509.23970 [15] “Cyber-Zero/enigma-plus/config/commands/debug.sh at main · amazon-science/Cyber-Zero — github.com,” https://github.com/amazon-science/Cyber-Zero/blob/main/enigmaplus/config/commands/debug.sh. [16] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” 2024. [Online]. Available: https://arxiv.org/abs/2405.15793 [17] “GitHub - radareorg/radare2: UNIX-like reverse engineering framework and command-line toolset — github.com,” https://github.com/radareorg/radare2. [18] “Frida • A world-class dynamic instrumentation toolkit — frida.re,” https://frida.re/. [19] “dynamorio.org,” https://dynamorio.org/. [20] H. Triedman, R. Jha, and V. Shmatikov, “Multi-agent systems execute arbitrary malicious code,” arXiv preprint arXiv:2503.12188, 2025. [21] D. Pasquini, E. M. Kornaropoulos, and G. Ateniese, “Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLMdriven Cyberattacks,” Nov. 2024, arXiv:2410.20911 [cs]. [Online]. Available: http://arxiv.org/abs/2410.20911 [22] A. Afianian, S. Niksefat, B. Sadeghiyan, and D. Baptiste, “Malware dynamic analysis evasion techniques: A survey,” ACM Computing Surveys (CSUR), vol. 52, no. 6, pp. 1–28, 2019. [23] H. Triedman, R. Jha, and V. Shmatikov, “Multi-agent systems execute arbitrary malicious code,” 2025. [Online]. Available: https://arxiv.org/abs/2503.12188 [24] W. Zhou, L. Zhang, L. Guan, P. Liu, and Y. Zhang, “What your firmware tells you is not how you should emulate it: A specificationguided approach for firmware emulation,” in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2022, pp. 3269–3283. [25] X. Shen, L. Wang, Z. Li, Y. Chen, W. Zhao, D. Sun, J. Wang, and W. Ruan, “Pentestagent: Incorporating llm agents to automated penetration testing,” in Proceedings of the 20th ACM Asia Conference on Computer and Communications Security, 2025, pp. 375–391. [26] “GitHub - NationalSecurityAgency/ghidra: Ghidra is a software reverse engineering (SRE) framework — github.com,” https://github.com/NationalSecurityAgency/ghidra. [27] Hex-Rays, https://hex-rays.com/ida-pro. [28] J. P. A. Yaacoub, H. N. Noura, O. Salman, and G. Pujolle, “Large language models: applications, limitations, challenges, and recommendations in cybersecurity, digital forensics, and ethical hacking,” Annals of Telecommunications, Nov. 2025. [Online]. Available: https://doi.org/10.1007/s12243-025-01134-9 [29] S. Ren and S. Chen, “Large Language Models for Cybersecurity Intelligence, Threat Hunting, and Decision Support.” [30] X. Hu, Z. Fu, S. Xie, S. H. H. Ding, and P. Charland, “SoK: Potentials and Challenges of Large Language Models for Reverse Engineering,” Sep. 2025, arXiv:2509.21821 [cs]. [Online]. Available: http://arxiv.org/abs/2509.21821 [31] L. Hu, G. Chen, X. Shang, S. Cheng, B. Wu, G. Li, X. Zhu, W. Zhang, and N. Yu, “CompileAgent: Automated Real-World RepoLevel Compilation with Tool-Integrated LLM-based Agent System.” [32] G. Chen, H. Sun, D. Liu, Z. Wang, Q. Wang, B. Yin, L. Liu, and L. Ying, “ReCopilot: Reverse Engineering Copilot in Binary Analysis,” May 2025, arXiv:2505.16366 [cs]. [Online]. Available: http://arxiv.org/abs/2505.16366

[33] H. Kong, D. Hu, J. Ge, L. Li, T. Li, and B. Wu, “VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework,” Jan. 2025, arXiv:2501.13411 [cs]. [Online]. Available: http://arxiv.org/abs/2501.13411 [34] X. Lin, J. Zhang, G. Deng, T. Liu, T. Zhang, Q. Guo, and R. Chen, “IRCopilot: Automated Incident Response with Large Language Models,” Oct. 2025, arXiv:2505.20945 [cs]. [Online]. Available: http://arxiv.org/abs/2505.20945 [35] T. Y. Zhuo, D. Wang, H. Ding, V. Kumar, and Z. Wang, “Training Language Model Agents to Find Vulnerabilities with CTF-Dojo,” Sep. 2025, arXiv:2508.18370 [cs]. [Online]. Available: http://arxiv.org/abs/2508.18370 [36] F. Balassone, V. Mayoral-Vilches, S. Rass, M. Pinzger, G. Perrone, S. P. Romano, and P. Schartner, “Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs,” Oct. 2025, arXiv:2510.17521 [cs]. [Online]. Available: http://arxiv.org/abs/2510.17521 [37] Y. Wang, X. Xu, X. Zhu, X. Gu, and B. Shen, “Salt4decompile: Inferring source-level abstract logic tree for llm-based binary decompilation,” 2025. [Online]. Available: https://arxiv.org/abs/2509.14646 [38] H. Rong, Y. Duan, H. Zhang, X. Wang, H. Chen, S. Duan, and S. Wang, “Disassembling obfuscated executables with llm,” 2024. [Online]. Available: https://arxiv.org/abs/2407.08924 [39] X. She, Y. Zhao, and H. Wang, “Wadec: Decompiling webassembly using large language model,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 481–492. [Online]. Available: https://doi.org/10.1145/3691620.3695020 [40] Skylot, “GitHub - skylot/jadx: Dex to Java decompiler — github.com,” https://github.com/skylot/jadx. [41] “Application fundamentals — App architecture — Android Developers — developer.android.com,” https://developer.android.com/guide/components/fundamentals. [42] X. Shang, G. Chen, S. Cheng, B. Wu, L. Hu, G. Li, W. Zhang, and N. Yu, “Binmetric: A comprehensive binary analysis benchmark for large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2505.07360 [43] J. I. Maletic and M. L. Collard, “Exploration, analysis, and manipulation of source code using srcml,” May 2015. [Online]. Available: http://dx.doi.org/10.1109/ICSE.2015.302 [44] Z. L. Basque, S. Doria, A. Soneji, W. Gibbs, A. Doupé, Y. Shoshitaishvili, E. Losiouk, R. Wang, and S. Aonzo, “Decompiling the synergy: An empirical study of human–llm teaming in software reverse engineering.” [45] “GDB: The GNU Project Debugger https://www.sourceware.org/gdb/.

—

sourceware.org,”

[46] T. Y. Zhuo, D. Wang, H. Ding, V. Kumar, and Z. Wang, “Cyber-zero: Training cybersecurity agents without runtime,” 2025. [Online]. Available: https://arxiv.org/abs/2508.00910 [47] T. Abramovich, M. Udeshi, M. Shao, K. Lieret, H. Xi, K. Milner, S. Jancheska, J. Yang, C. E. Jimenez, F. Khorrami, P. Krishnamurthy, B. Dolan-Gavitt, M. Shafique, K. Narasimhan, R. Karri, and O. Press, “Enigma: Interactive tools substantially assist lm agents in finding security vulnerabilities,” 2025. [Online]. Available: https://arxiv.org/abs/2409.16165 [48] O. L. . C. LLC, “Debuggers - CTF Handbook — ctf101.org the gnu debugger (gdb),” https://ctf101.org/reverse-engineering/what-is-gdb/, 2024. [49] T. Y. Zhuo, D. Wang, H. Ding, V. Kumar, and Z. Wang, “Training language model agents to find vulnerabilities with ctf-dojo,” 2025. [Online]. Available: https://arxiv.org/abs/2508.18370

[50] “GitHub - amazon-science/Cyber-Zero: Cyber-Zero: Training Cybersecurity Agents Without Runtime — github.com,” https://github.com/amazon-science/Cyber-Zero, 2025. [51] X. Shang, S. Cheng, G. Chen, Y. Zhang, L. Hu, X. Yu, G. Li, W. Zhang, and N. Yu, “How far have we gone in binary code understanding using large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2404.09836 [52] S. Qin, C. Zhang, K. Chen, and Z. Li, “idev: Exploring and exploiting semantic deviations in arm instruction processing,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, pp. 580–592. [53] “gem5: The gem5 https://www.gem5.org/.

simulator

system

—

gem5.org,”