arXiv CSopen access

Guarding Organizations Against Malware Risk: A Novel Graph-Based Malware Detection Method

Guarding Organizations Against Malware Risk: A Novel Graph-Based Malware Detection Method Yinan Gao School of Management, Fudan University, [email protected]

Jiarong Xu School of Management, Fudan University, [email protected]

Xiaohang Zhao School of Information Management and Engineering, Shanghai University of Finance and Economics, [email protected]

Xiao Fang Lerner College of Business and Economics, University of Delaware, [email protected]

arXiv:2607.26634v1 [cs.CR] 29 Jul 2026

Abstract The digitalization of business processes and IT infrastructures has expanded organizations’ exposure to cybersecurity risks, which have made cybersecurity an increasingly important research area in Information Systems (IS). Among these risks, malware has emerged as one of the most pervasive and destructive threats to organizational digital systems. To mitigate malware risks, byte-based machine learning (ML) methods are widely used to detect malware, but they remain vulnerable to malware exhibiting evasive behaviors. Such malware can evade detection by strategically manipulating raw bytes to confuse the detector while preserving its harmful functionality. Program graphs offer a promising alternative by representing software execution behavior rather than raw bytes, thereby reducing the influence of byte manipulations inserted into non-executed regions. Graph-based methods build on such program graphs, but existing approaches still face two limitations: they do not explicitly identify cohesive groups of basic blocks (i.e., short sequences of instructions) which jointly realize meaningful program behaviors, and they do not learn sufficiently expressive program graph representations for accurate malware detection. To address these limitations, we propose MalGuard, a graph-based malware detection method for supporting organizational malware risk management. MalGuard introduces two methodological innovations: an operational role identification approach and a program graph representation learning method. The former identifies these cohesive groups of basic blocks as operational roles, enabling the detector to capture program behaviors that may not be visible from isolated basic blocks. The latter learns expressive program graph representations by modeling interactions among operational roles, preserving sparse but critical malicious signals, and capturing the hierarchical structure of program graphs. Through extensive experiments on a real-world software dataset, we show that MalGuard outperforms state-of-the-art byte-based and graph-based malware detection methods and yields economic value by reducing the expected cost of undetected malware.

Keywords: Cybersecurity; computational design science; malware detection; program graph; graph neural network 1

1 INTRODUCTION The digitalization of business processes and IT infrastructures has enabled organizations to improve operational efficiency and generate value from data-driven decision making (Melville et al., 2004; Zheng et al., 2012; Mao et al., 2023). By moving more critical activities onto interconnected digital technologies, however, this transformation has expanded organizations’ exposure to cybersecurity risks (Li et al., 2023). Cybersecurity has therefore become a central theme in Information Systems (IS) research (Cram et al., 2019; Abbasi et al., 2021; Samtani et al., 2022; Clement and Arce, 2025). In particular, IS scholars are increasingly adopting the computational design science paradigm (Rai, 2017; Fang et al., 2025) to develop methods that help organizations mitigate cybersecurity risks, such as phishing threat (Abbasi et al., 2021) and hacker exploits (Ampel et al., 2024). Among these risks, malware (malicious software) has emerged as one of the most pervasive and destructive threats to digital systems in organizations (Kim and Kim, 2014; Yu et al., 2009). Malware refers to software programs intentionally designed to infiltrate, disrupt, or damage computer systems (Gandotra et al., 2014).1 The malware threat landscape remains large and active, with tens of millions of new malware instances reported over the past 12 months.2 This growing threat has resulted in substantial financial losses for organizations. A prominent example is the SolarWinds supply-chain malware attack in 2020, which compromised nearly 18,000 organizations worldwide and caused estimated damages exceeding $10 billion.3 Reflecting these consequences, research in IS has increasingly examined malware as both an organizational risk and a critical technical challenge for effective detection (Guo et al., 2016; Dey and Lahiri, 2026; Ebrahimi et al., 2025). To mitigate malware risks, organizations increasingly rely on automated machine learning (ML)

1

In this paper, we focus on malware in the Windows Portable Executable (PE) file format, which accounts for over 97% of all malware (Ling et al., 2022). We use the terms “software” and “programs” interchangeably. 2 See https://portal.av-atlas.org/malware (last accessed on Jul 23, 2026). 3 See https://www.gao.gov/blog/solarwinds-cyberattack-demands-significant-federal-and-private-sector-responseinfographic (last accessed on Jul 23, 2026).

2

… 0x4006E0 0x4006F0 0x400600 0x400610 0x400620 0x400630 0x400640 0x400650 0x400660 0x400670 0x400680 0x400690 0x4006A0 0x4006B0 0x4006C0 0x4006D0 0x4006E0 0x4006F0 0x400700 0x400710 0x400720 0x400730 0x400740

00 10 00 00 10 00 00 00 00 10 00 00 10 00 00 00 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0C 00 00 00 00 60 00 00Adversarial 00 02 00 benign-looking 00 00 12 00 00 byte 00 10 00 00 00 00 00 00 00 00 00sequences 00 40 00 00 42 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00 80 28 00 00 10 00 00 00 48 00 00 00 02 00 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 08 A4 08 00 00 00 20 00 00 00 0A 00 00 00 02 00 00 00 00 00 00 00 00 00 00 0D 00 00 00 20 00 00 60 2E 72 73 72 63 00 00 00 A8 05 00 00 00 40 00 00 00 06 00 00 00 0c 00 00 00 00 00 00 00 00 00 00 E4 20 00 00 C0 06 00 00 09 00 00 00 01 00 00 06 4C 28 00 00 4F 00 00 00 00 40 00 00 A8 05 00 00 00 00 00 00 00 00 00 00 Core 00 00malicious 00 00 00 00 00 00 payload 60 00 00 0C 00 00 00 A4 27 00 00 1C 00 00 00 00 00 00 00 A6 00 00 00 08 20 00 00 00 48 00 00 00 00 00 00 00 40 00 00 40 2E 72 65 6C 6F 63 00 00 00 00 00 00 00 00 00 00 20 00 00 00 08 00 00 00 00 00 00 00 00 00 00 00 2E 74 65 78 74 00 00 00 00 00 00 00 00 00 00 00 50 20 00 00 80 00 00 00 00 09 00 00 40 00 00 11 00 00 20 00 00 09 00 00 00 20 00 00 01 00 05 00 00 10 00 00 00 00 70 00

control flow

non-executed filtered out regions ×

0x40067c: push ebp 0x400680: mov ebp, esp 0x4006ac: ret 0x400780: mov eax, 1 0x400782: shl eax, 8 0x400784: add eax, 0x20 0x400888: jmp 0x000896

(e.g., legacy header, section slack)

executed regions

operational role

illustrative instructions in a basic block

…

(a) Raw byte representation

Figure 1

basic block

executed regions

(b) Program graph representation (taking CFG as an example)

Comparison of raw byte and program graph representations of malware with evasive behaviors.

methods to detect malware before it can enter or operate within organizational environments. A large body of this research focuses on byte-based malware detection methods, which process the raw bytes of software as input and use ML models to learn byte-level patterns for distinguishing malware from benign software (Raff et al., 2018; Fleshman et al., 2019; Saha et al., 2024). However, such methods remain vulnerable to malware exhibiting evasive behaviors (Afianian et al., 2019; D’Elia et al., 2020). In particular, adversaries can modify raw bytes so that malware evades detection without compromising its harmful functionality. This is often achieved by injecting adversarially crafted benign-looking byte sequences into non-executed regions while leaving the core executed malicious payload intact (Kirat et al., 2014; D’Elia et al., 2020; Ebrahimi et al., 2025). As illustrated in Figure 1(a), these inserted adversarial benign-looking byte sequences (highlighted in red) can obscure the core malicious payload (highlighted in yellow) in the raw byte representation. Consequently, byte-based detection methods may be misled, resulting in the misclassification of evasive malware as benign (Afianian et al., 2019; D’Elia et al., 2020). For organizations, such detection failures mean that malware may bypass cyber defenses and remain active within operational environments. Graph-based malware detection methods address this challenge by shifting the representation of software from raw bytes to execution behavior modeled in program graphs. These program graphs 3

are then fed into graph neural networks (GNNs) to learn software representations for detection (Yan et al., 2019; Ling et al., 2022; Feng et al., 2024; Kargarnovin et al., 2024). Figure 1 illustrates why this representation is more robust to malware with evasive behaviors. Raw byte representations include both executed and non-executed byte regions, which means that byte-based detectors analyze the entire byte regions, including injected benign-looking byte sequences. In contrast, program graphs represent the program’s execution paths rather than its raw bytes. As shown in the resulting program graph of Figure 1(b), ordinary code that is executed is mapped to regular graph nodes, and the core malicious payload is mapped to graph node(s) with malicious signals, whereas injected benign-looking bytes in non-executed regions have no corresponding graph nodes or edges and are filtered out. The program graph therefore preserves executed behavior while excluding non-executed injected bytes. A common form of program graph is the control flow graph (CFG), where each node represents a basic block and each edge represents a possible control flow transition between basic blocks. A basic block is a short sequence of instructions that runs sequentially without internal branches or jumps; the inset in Figure 1(b) provides an illustrative example of a basic block. By abstracting execution flows rather than raw bytes, program graphs preserve executed behavior while excluding non-executed injected bytes, thereby providing robustness against evasive behaviors. However, existing graph-based methods struggle to accurately detect malware when analyzing program graphs, leaving critical challenges unresolved. First, malicious behavior often does not reside in an individual basic block (node). As illustrated by the yellow blocks in Figure 1(b), one block may reserve memory, another may prepare a request to the operating system, and a subsequent block may update a system setting. Examined in isolation, these blocks may appear routine: reserving memory, setting up a system request, or updating a system configuration. Yet when executed together along the control flow path, they can reveal a malicious privilege escalation operation, in which the malware attempts to obtain higher-level permissions than it is allowed

4

to have. We refer to such a cohesive group of basic blocks that work together toward a shared computational objective as an operational role (Allen, 1970; Bruschi et al., 2006). Identifying these operational roles is crucial for malware detection because it enables the detector to capture malicious behaviors that emerge only through coordinated execution. Existing graph-based methods typically learn representations from individual basic blocks or their local neighborhoods. Yet, operational roles are defined by shared computational objectives rather than by isolated basic blocks or fixed neighborhoods. Consequently, these methods fail to explicitly model operational roles, and often overlook malicious signals that become apparent only when several routine-looking basic blocks are examined together. Second, identifying operational roles alone is not sufficient; an effective detector must also learn expressive representations of program graphs for accurate malware detection. Addressing this challenge is nontrivial because program graphs exhibit three salient characteristics. The first characteristic is that operational roles may interact across the graph. For example, one operational role may collect user credentials, while another may check these credentials to determine whether access to restricted system resources should be allowed. Such interactions among roles are informative for understanding program behavior because they reveal higher-level workflows, such as authentication and access control. However, existing graph-based methods typically do not explicitly model interactions among operational roles. Furthermore, malicious signals for detecting malware are often sparse. A large program graph consists mostly of routine parts, such as initialization or simple computation, while only a small fraction carries malicious signals. Existing graph-based methods tend to combine information from different parts of the graph uniformly, both when GNNs aggregate information from neighboring nodes and when node representations are pooled to form the final software representation. In these steps, rare malicious signals may be mixed with routine neighboring signals or outweighed by representations of routine nodes. As a result, sparse but

5

important malicious signals can become less visible to the detector. Besides, program graphs are structurally hierarchical, as software is organized at multiple levels. Basic blocks form the internal execution flow of each function, and functions further call one another to shape the behavior of the broader program. A graph-based method therefore needs to model this hierarchy rather than treating the program as a flat graph that ignores these levels. Together, these characteristics call for an effective program graph representation learning method that can model complex interactions among operational roles, preserve sparse malicious signals as graph information is aggregated and pooled for detection, and capture the hierarchical structure of software. In response to the above challenges, we propose MalGuard, a novel graph-based malware detection method to support malware risk management. MalGuard is designed to overcome two critical limitations of existing graph-based ML methods: overlooking the operational roles in program graphs and falling short in learning expressive program graph representations for malware detection. To achieve this, we first design an operational role identification approach that uncovers cohesive groups of basic blocks functioning together toward shared computational objectives, i.e., operational roles. By identifying these operational roles, our method captures node groups that can reflect meaningful program behaviors, including behaviors that may indicate malicious activity. Second, we propose a program graph representation learning method for malware detection, featuring three key innovations: a coarsening layer to model complex interactions among operational roles, attention-based message passing and gated pooling layers to preserve rare but critical malicious signals as graph information is aggregated and pooled for malware detection, and a hierarchical learning framework to capture the hierarchical structure of program graphs and obtain the final software representation. Extensive experiments on a real-world software dataset collected for this study demonstrate that MalGuard outperforms state-of-the-art malware detection methods while yielding substantial economic value by reducing the financial cost of undetected malware.

6

2 LITERATURE REVIEW In this section, we first review the broader IS cybersecurity literature and discuss malware threat. We then review existing malware detection methods and highlight the key novelties of our study. 2.1 IS Cybersecurity Research and Malware Threat Our study is situated in IS cybersecurity research, an important area that examines cybersecurity as a critical organizational and technical challenge. IS cybersecurity research can be broadly organized into three categories. One stream examines the behavioral dimension of cybersecurity, focusing on how individuals and organizations perceive and respond to cybersecurity threats (Bulgurcu et al., 2010; Moody et al., 2018; Cram et al., 2019). A second stream uses economic modeling to examine how cybersecurity risks influence firms’ market outcomes and economic consequences, thereby generating managerial insights for firms’ cybersecurity risk management (Clement and Arce, 2025; Temizkan et al., 2017; August et al., 2022). A third stream, more relevant to our study, adopts the computational design science paradigm to develop technical methods that help organizations mitigate cybersecurity risks, such as phishing threat, hacker exploits, and cyber threats associated with hacker assets (Abbasi et al., 2021; Samtani et al., 2022; Ampel et al., 2024). For example, Abbasi et al. (2021) address phishing as a cybersecurity threat by proposing a phishing funnel model to predict user susceptibility to phishing websites. Samtani et al. (2022) and Ampel et al. (2024) both focus on hacker exploits as cybersecurity threats and develop technical methods to analyze and label exploits for proactive cyber defense. In addition, Ebrahimi et al. (2022) study cyber threats associated with hacker assets, such as hacking tools and malicious source code shared on the dark web, and introduce cross-lingual cybersecurity analytics for detecting such assets across international dark web platforms. These studies demonstrate the value of computational design science IS research in responding to different cyber threats. Among the cybersecurity risks, malware (malicious software) creates a critical challenge for 7

organizational risk management. Malware is intentionally developed by cybercriminals to achieve harmful objectives, typically targeting enterprise computer systems and leading to substantial economic and societal damage (Kolter and Maloof, 2004). Given these profound implications, IS scholars have increasingly focused on the economic impacts and detection mechanisms of malware. Within this domain, one line of IS research focuses on the economic and organizational implications of malware, aiming to explain how malware affects market competition and organizational processes (Kim and Kim, 2014; Guo et al., 2016; August et al., 2022). For instance, Kim and Kim (2014) study the malware resolution process of an antivirus software provider and show how prior experience affects the efficiency of resolving malware problems. Guo et al. (2016) further analyze how network structures dictate the propagation speed and scale of malware. In addition, ransomware has received increasing attention as a major form of malware risk, with recent studies examining its implications for risk interdependence and externalities across firms (August et al., 2022; Dey and Lahiri, 2026). Another line of IS research focuses on developing technical methods for malware detection (Ebrahimi et al., 2025). Ebrahimi et al. (2025) propose the RADAR framework, which utilizes deep reinforcement learning to develop an adversarially robust malware detection method. Together, these studies establish malware as an important cybersecurity topic in IS research and highlight the need for effective technical artifacts to support organizational malware risk management. To ground this research in a prevalent organizational malware threat, we focus on Portable Executable (PE) malware, which has emerged as one of the most common and persistent cybersecurity threats (Ye et al., 2017). PE malware targets programs in the PE format, the standard file format for Windows applications (Microsoft, 2025). Because Windows remains widely used in consumer and enterprise computing, PE software represents a high-value target for attackers. In addition, the structured and extensible nature of the PE format provides attackers with multiple opportunities

8

to manipulate file contents (Pietrek, 2002). These characteristics make PE malware a persistent threat to organizations, supporting a wide range of malicious activities such as data theft, system disruption, and ransomware deployment (August et al., 2022; McIntosh et al., 2024; Dey and Lahiri, 2026). More importantly, the challenge of PE malware is further amplified by its increasing use of evasive behaviors. Malware with evasive behaviors deliberately modifies its byte file to evade detection while preserving harmful functionality (Christodorescu and Jha, 2003; Kirat and Vigna, 2015; D’Elia et al., 2020; Ling et al., 2023). These evasion techniques are designed to mislead ML-based detectors by injecting adversarial or benign-looking byte sequences into non-executed regions of the file, such as DOS headers, slack space, or appended regions (Demetrio et al., 2021; Suciu et al., 2019; Kreuk et al., 2018; Nisi et al., 2021). As illustrated in Figure 1(a), such injected bytes (in red) can obscure the core malicious payload (in yellow) in the raw byte representation while leaving the executed malicious behavior intact. These evasive behaviors make malware increasingly difficult to detect, posing a serious threat to malware detection systems. Our study contributes to IS cybersecurity research by proposing a graph-based malware detection method that provides inherent robustness to malware with evasive behaviors and more accurately identifies malware by effectively learning from program structures and malicious signals, thereby supporting organizational malware risk management. 2.2 Malware Detection Methods To identify and mitigate malware threat, a variety of malware detection methods have been developed. These methods can be broadly categorized into four main types based on how they represent software for detection: signature-based methods, feature-based ML methods, byte-based ML methods, and graph-based ML methods. Among these, signature-based and feature-based methods are traditional detection approaches. Signature-based methods match predefined patterns from known malware (Sung et al., 2004), while feature-based ML methods train classifiers on handcrafted soft-

9

ware features (Mohaisen et al., 2015; Anderson and Roth, 2018); however, both approaches struggle to detect malware with evasive behaviors, and their reliance on fixed signatures or handcrafted features limits their ability to capture more complex behavioral patterns of malware (Ye et al., 2007; Anderson and Roth, 2018). Recent malware detection research has increasingly shifted toward byte-based and graph-based ML methods. Accordingly, we focus on these two approaches and review them in detail below. Byte-based ML malware detection.

Byte-based methods operate directly on the raw byte

sequences of programs and learn malware-related patterns from the original bytes. These methods commonly adopt convolutional neural networks (CNNs) to capture local byte-level patterns in executables (Yan et al., 2018). A prominent example is MalConv (Raff et al., 2018), which embeds each byte sequence into a continuous vector space, applies gated convolutional layers to learn local patterns, and uses temporal max-pooling to extract the most salient features across the file. MalConv demonstrates strong effectiveness in malware detection and remains a state-of-the-art benchmark for byte-based approaches. Despite their effectiveness in detecting conventional malware, MalConv and other byte-based models remain vulnerable to malware exhibiting evasive behaviors (Ling et al., 2023). This vulnerability arises because these methods process the entire PE file, including non-executed regions where adversarial perturbations or benign-looking byte sequences can be injected (Kreuk et al., 2018; Demetrio et al., 2019, 2021; Suciu et al., 2019; Nisi et al., 2021). To mitigate this issue, several extensions of MalConv have been proposed. For instance, Fleshman et al. (2019) impose non-negative constraints on the final layer’s weights to reduce the model’s sensitivity to the manipulated regions, preventing injected bytes from undermining malware predictions. Saha et al. (2024) propose an ensemble-based strategy DRSM, which divides the input byte sequence into multiple segments, applies a pre-trained MalConv model to each segment independently, and

10

aggregates the resulting predictions through majority voting. This design improves robustness by reducing over-reliance on any single region of the file. Although these methods alleviate the impact of evasive behaviors, the use of constraints, region-wise ensemble mechanisms, or adversarial training may compromise detection accuracy on conventional malware (Saha et al., 2024). Graph-based ML malware detection. Graph neural networks (GNNs) have become a widely used framework for learning representations from graph-structured data (Wu et al., 2020; Feng et al., 2017). Using this capability, graph-based ML methods detect malware by representing software as program graphs, which are then fed into GNNs for detection. Compared to byte-based methods, these methods offer two key advantages. First, by encoding control structures and function call interactions in program graphs, graph-based ML approaches can better capture a software instance’s underlying execution logic (Reps, 1998; Alfred et al., 2007). Second, these approaches are inherently more robust to malware exhibiting evasive behaviors. Because program graphs model the execution logic of software, benign-looking byte sequences injected into non-executed regions are excluded from the graph and thus have limited influence on detection outcomes (Bilot et al., 2024). Figure 1(b) shows a program graph of malware with evasive behaviors, which preserves the executed malicious functionality (in yellow) while naturally removing the adversarial benign-looking byte sequences that are never executed. Several recent studies have advanced graph-based ML methods to model program graphs for malware detection. For example, Yan et al. (2019) propose MAGIC, which represents software as a control flow graph (CFG) to explicitly capture execution paths. A GNN is then applied to learn structural patterns from the CFG. Chen et al. (2023) introduce DeepCall, which constructs a call graph (CG) from each software, where nodes represent functions and edges denote calling relationships, and applies a GNN to learn from the CG. Furthermore, Kargarnovin et al. (2024) explore CGs for malware detection and impose a non-negative weight constraint to enhance the

11

robustness of GNNs against evasive behaviors, inspired by earlier non-negative techniques for byte-based methods (Fleshman et al., 2019). Nevertheless, these approaches only focus on a single structural view of program graph, either CFG or CG. To overcome this limitation, Ling et al. (2022) propose MalGraph, which integrates CFGs and CG into a unified hierarchical structure and employs a hierarchical framework to learn from the structure. However, existing graph-based ML methods do not fully exploit program graphs for malware detection. First, meaningful behaviors in CFGs often emerge not from individual nodes but from groups of nodes working together toward a shared computational objective, which we refer to as an operational role (Balsamo et al., 2004). These roles may directly correspond to malicious intent, such as privilege escalation. Yet existing graph-based methods typically learn from individual nodes or from their local neighborhoods, without explicitly identifying operational roles. This creates a mismatch because an individual basic block often captures only a short sequence of instructions rather than a meaningful behavior, while local neighborhoods do not group nodes according to whether they jointly carry out the same computational objective. As a result, these methods may miss malicious behavior that emerges only when several routine-looking blocks are examined together. Second, existing methods still fall short in learning expressive program graph representations for malware detection. Effective detection requires software representations that can jointly capture complex interactions among operational roles, preserve sparse but critical malicious evidence during GNN aggregation and pooling for detection, and model the hierarchical structure between CFGs and the CG. Existing GNN architectures are limited in this context, which underscores the need for a more effective graph representation learning approach for malware detection. Research gaps.

Our literature review suggests that mainstream malware detection methods,

including byte-based and graph-based approaches, suffer from inherent limitations, as summarized in Table 1. Byte-based ML methods treat programs as raw bytes, overlooking the execution

12

Table 1

Comparison of our proposed method with existing malware detection methods. Robust to evasive behaviors

Operational role identification

Modeling role interactions

Preserving sparse malicious signals

Modeling hierarchical structure

× ✓ ✓

× × ×

× × ×

× ✓ ✓

× × ×

MAGIC (Yan et al., 2019) MalGraph (Ling et al., 2022) Graph-based DeepCall (Chen et al., 2023) Mal2GCN (Kargarnovin et al., 2024)

✓ ✓ ✓ ✓

× × × ×

× × × ×

× × × ✓

× ✓ × ×

Ours

✓

✓

✓

✓

✓

Category

Methods

Byte-based

MalConv (Raff et al., 2018) NonNeg (Fleshman et al., 2019) DRSM (Saha et al., 2024)

logic that is critical for understanding program behavior. Moreover, these methods are vulnerable when detecting malware exhibiting evasive behaviors because they rely on raw byte patterns that adversaries can strategically manipulate while preserving malware’s harmful functionality. Graphbased ML methods are more robust to evasive behaviors because they shift the basis of detection from such surface-level artifacts to execution behavior modeled in program graphs. Since program graphs are constructed from possible execution paths, adversarial manipulations inserted into non-executed regions have limited impact on the resulting representation. However, important research gaps remain. First, existing graph-based methods do not explicitly identify operational roles, which are groups of basic blocks that jointly carry out a shared computational objective. This limitation is important because malicious behavior may not be visible in individual blocks or their local neighborhoods, but may become apparent only when several routine-looking blocks are examined together as part of the same operational role. Second, existing graph-based methods leave the challenge of learning expressive program representations for malware detection. Effective representations must simultaneously capture interactions among operational roles, preserve sparse but critical malicious signals during aggregation and pooling, and model hierarchical program structures. Existing architectures are not designed to satisfy these requirements simultaneously. To address these gaps, we propose two methodological components. First, we develop an operational role identification approach that infers semantically cohesive groups of basic blocks corresponding to shared computational objectives. Second, we propose a program graph represen-

13

tation learning method for malware detection that can simultaneously model interactions among operational roles, preserve sparse malicious evidence for accurate malware detection, and capture hierarchical program structure. Together, these two components constitute the key methodological contributions of this study. 3 PROBLEM FORMULATION AND PRELIMINARIES In this section, we first formally define the malware detection problem. We then introduce preliminaries on program graphs, which serve as the input of our proposed malware detection method. 3.1 Problem Definition Consider a set of software instances denoted by R, where each instance ri ∈ R is assigned a binary label yi ∈ {0, 1}. Specifically, yi = 0 indicates that the software is benign, whereas yi = 1 denotes that it is malicious. Formally, we define the research problem as follows. Definition 1 (Malware Detection Problem). Let R = {ri } denote a set of software instances, where each instance belongs to one of two classes: benign or malicious. The objective of malware detection is to learn a classification function f : ri → yi ,

yi ∈ {0, 1},

(1)

that can accurately identify malicious instances (yi = 1), including those with evasive behaviors. 3.2 Hierarchical Program Graph Representing software in a form that is inherently robust to evasive behaviors is a prerequisite for addressing the malware detection problem. Program graphs provide such a representation by abstracting a program’s potential execution logic. Because program graphs are constructed from possible execution paths rather than raw bytes, manipulations inserted into non-executed regions, a common evasion strategy, have limited impact on the resulting graph (Peng et al., 2025). Program execution is complex because software is organized hierarchically. Software is not simply a flat sequence of instructions; instead, a software program consists of multiple functions, 14

block 1 block 2 block 3

Func1 (0x000058)

block 5 block 4 block 6

0x00067c: push ebp 0x000680: mov ebp, esp 0x0006ac: ret 0x000780: mov eax, 1 0x000782: shl eax, 8 0x000784: add eax, 0x20 0x000888: jmp 0x000896

block instructions

block 7

Func2 (0x0000B2)

Func3 (0x0002D2)

Func4 (0x0005a4)

block 8

Func5 (0x00067c)

… block 1

Software

block 2

Func6 (0x000806)

Func7 (0x000c42)

block 3 block 5

0x000b02: push ebx 0x000b0c: mov ebx, ebp 0x000b14: call 0x000b00 0x000b19: pop ebx 0x000b1a: jmp 0x000b20

block instructions

block 4

block 6

Inter-function level: Call Graph (CG)

Figure 2

Intra-function level: Control Flow Graphs (CFGs)

Illustration of the hierarchical program graph for a software instance.

and functions may call one another during execution. These inter-function relationships shape the behavior of the overall program and are represented by a Call Graph (CG). At the same time, within each function, instructions are grouped into multiple basic blocks, and the intra-function control flow structure among basic blocks is represented by a CFG. Figure 2 illustrates such a hierarchical structure in a program graph for an example software instance. We therefore represent each software instance as a hierarchical program graph that combines CFGs for individual functions with a CG connecting these functions. In this hierarchy, CFGs constitute the intra-function (lower) level and the CG constitutes the inter-function (higher) level. This representation captures both how execution proceeds within each function and how functions interact across the program. Formally, let GCG denote the CG among the functions in a software instance, with VCG denoting its set of function nodes and |VCG | the number of functions. Let GCFG = {G1 , G2 , . . . , G|VCG | } denote the collection of CFGs, where Gm is the CFG corresponding to function node vm ∈ VCG . The hierarchical program graph is defined as H = (GCG , GCFG ).

(2)

Next, we detail the construction procedure of a hierarchical program graph. At the inter-function level, the software is transformed into a CG, which captures interactions

15

between functions, with nodes representing functions and edges indicating call instructions between caller functions and their corresponding callees (Ryder, 1979). We define the CG as follows: Definition 2 (Call Graph (CG)). Given a software instance, its CG is defined as GCG = (VCG , ECG ). Here, VCG is the set of nodes, with each node vm ∈ VCG representing a function contained in the software. ECG is the set of edges, with each edge (vm , vn ) ∈ ECG denoting a function call between the caller function vm and the callee function vn . At the intra-function level, each function is further represented as a CFG. A CFG specifies the internal control flow of a function, where nodes correspond to basic blocks and edges represent control flow transitions between them (Ryder, 1979). A basic block is a straight-line sequence of instructions that executes sequentially from a single entry point to a single exit point, without internal control flow transfers such as branches, jumps, or returns. Each basic block is further enriched with statistical attributes extracted from its instructions. CFG can be defined as follows: Definition 3 (Control Flow Graph (CFG)). Given the function represented by node vm ∈ VCG , its CFG is defined as Gm = (Vm , Am , Xm ), where Vm is the set of basic block nodes, Am ∈ {0, 1}|Vm |×|Vm | is the adjacency matrix, and Xm ∈ R|Vm |×dx is the node attribute matrix. Each node vmi ∈ Vm represents the i-th basic block within function m, and |Vm | is the total number of basic blocks within the function. For the adjacency matrix, (Am )ij = 1 indicates the existence of a control flow from basic block vmi to basic block vmj , and (Am )ij = 0 otherwise. The i-th row of the node attribute matrix, (Xm )i , encodes statistical attributes derived from the instruction content of basic block vmi , including the total number of instructions, the counts of different instruction types, constant values, and other statistical characteristics.4 The constructed hierarchical program graph provides a unified abstraction of software execution behavior by capturing both inter-function interactions and intra-function control flows. By design, 4

When focusing on a single CFG and no ambiguity arises, we simplify the notation to G = (V, A, X) and write vi ∈ V , Aij , and Xi for basic block nodes, adjacency entries, and node attributes, respectively.

16

Call Graph (CG)

CFG with operational subgraphs

Control Flow Graph (CFG)

Result

Intra-function modeling

…

Software

Hierarchical program graph

Figure 3

…

Inter-function modeling

Operational role identification approach

Program graph representation learning method

Framework overview of MalGuard.

evasive manipulations inserted into regions that do not appear on execution paths have limited impact on the resulting hierarchical program graph, making it a suitable input for the method proposed in the next section. 4 PROPOSED METHOD In this section, we introduce MalGuard, a novel graph-based method for addressing the malware detection problem. MalGuard builds on hierarchical program graphs, which represent software execution behavior and provide robustness against malware with evasive behaviors. MalGuard introduces two methodological innovations. First, an operational role identification approach infers latent operational roles in each CFG and extracts the corresponding operational subgraphs (Section 4.1). Second, a program graph representation learning method for malware detection learns software representations for accurate malware detection by modeling interactions among operational roles, preserving sparse malicious signals, and capturing the hierarchical structure of program graphs (Section 4.2). Section 4.3 summarizes the training algorithm. Figure 3 provides a framework overview of the proposed method. 4.1 Operational Role Identification A salient characteristic of the hierarchical program graph is that within each CFG, meaningful execution behaviors often arise not from individual basic blocks (nodes) but from groups of basic blocks working together toward a shared computational objective (Allen, 1970; Bruschi et al., 2006). We refer to such a semantically cohesive group of basic blocks as having a shared operational role 17

(short for role). Unlike a single basic block that only performs elementary operations and thus contributes just a fragment of an overall behavior, this collaborative structure among basic blocks with a shared role captures a more holistic execution behavior that emerges from the joint activity of multiple blocks. More importantly, it is particularly crucial in malware detection. Malicious behaviors are typically realized through the coordinated execution of interrelated blocks (Cesare and Xiang, 2010). Therefore, identifying operational roles of basic blocks is essential for accurately modeling CFGs and uncovering malicious patterns. However, these roles are not directly observable or labeled in practice, and manual identification is both labor-intensive and impractical at scale. To address this, we propose a novel operational role identification module that treats operational roles as latent variables, enabling them to be inferred from the observed CFG without relying on explicit supervision. We first overview the module architecture and then elaborate each component in turn. 4.1.1 Overall Architecture Inspired by latent variable approaches that infer hidden constructs from observable data (Zheng and Pavlou, 2010; Wang et al., 2024), we represent operational roles as latent variables associated with program nodes. Formally, given a CFG G = (V, A, X), each basic block (node) vi ∈ V is associated with a latent variable zi ∈ RK encoding its probabilistic affiliation with one or more operational roles. Each dimension zik corresponds to the degree to which vi is associated with the kth operational role. A notable feature of CFG is that a single basic block may contribute to multiple roles simultaneously. For example, in a ransomware, a single basic block might simultaneously encrypt a user’s file (a malicious role) and check whether it is the last file in a folder (a control role) (Alfred et al., 2007). To accommodate this multi-role assignment of a basic block, we model zi using a multivariate logit-normal distribution (Mead, 1965; Aitchison and Shen, 1980; Aitchison, 1982), ensuring that each element zik lies within the range (0, 1). This probabilistic formulation

18

Reconstructed structure

Learning objective

Input

Inference model

Generative model Latent variable: operational roles

Figure 4

The architecture of the operational role identification module.

supports soft yet interpretable role assignments and allows each block to participate in multiple roles to varying extents.5 To infer the latent operational roles from observed CFG, we adopt a variational inferencegenerative architecture as illustrated by Figure 4: the inference model maps each basic block to a distribution over a low-dimensional latent space, capturing its probabilistic association with latent roles; the generative model then attempts to reconstruct the original CFG structure based on the sampled latent role assignments, encouraging the learned roles to preserve meaningful executionrelated semantics. This reconstruction-based training objective allows the model to learn latent roles that are operationally meaningful and aligned with execution semantics. In the following, we detail the design of the inference model, the generative model, and the training objective. 4.1.2 Inference Model The inference model aims to infer a latent variable of each basic block that captures its association with multiple latent operational roles. Specifically, the inference model maps the adjacency matrix A and node attribute matrix X of the CFG into a variational posterior distribution qϕ (Z|A, X) where |V |

Z = {zi }i=1 is the set of latent variables. Following prior works (Kipf and Welling, 2016; Grover et al., 2019; Li et al., 2020), we adopt a mean-field approximation for the variational distribution by assuming that the latent variables are conditionally independent given the input graph. In accordance

5

Compared to conventional Gaussian priors commonly used in existing works (Kipf and Welling, 2016; Grover et al., 2019), the multivariate logit-normal distribution is better suited for reflecting latent roles in real-world software behaviors.

19

with the aforementioned logit-normal design of each latent vector zi , we model each qϕ (zi | A, X) as a multivariate logit-normal distribution: qϕ (Z|A, X) =

|V | Y

qϕ (zi |A, X),

(3)

i=1

qϕ (zi |A, X) = LogitNormal(µi , Σi ), Σi = diag(σ 2i ),

(4)

where LogitNormal(µi , Σi ) is the density function of the logit-normal distribution specified by the mean vector µi ∈ RK and the covariance matrix Σi ∈ RK×K , with the latter being a diagonal matrix filled by the element-wise square of the standard deviation vector σ i ∈ RK . This formulation guarantees that zik ∈ (0, 1) for k = 1, 2, ...K. In plain words, µi quantifies the expected logit vector of role strengths for node vi , while σ i captures the uncertainty in these estimates across different roles. To obtain µi and σ i , we apply a GNN that encodes both structural information A and node attributes X simultaneously: µ, σ = GNNϕ (A, X),

(5)

where GNNϕ is a GNN layer specified by learnable parameters ϕ. This layer outputs a matrix of size |V | × 2K, with the first K columns extracted as matrix µ ∈ R|V |×K , while the remaining K columns as σ ∈ R|V |×K . We then treat the i-th row of µ and σ as µi and σ i respectively. To enable gradient-based optimization, we adopt the reparameterization trick (Kingma and Welling, 2014; Rezende et al., 2014) to draw instances from qϕ (zi |A, X) in a way that is differentiable with respect to learnable parameters. Specifically, once obtaining µi and σ i , we sample the latent variable zi as follows: e zi = µi + σ i ⊙ ϵi , where ϵi ∼ N (0, I),

(6)

zik = sigmoid(e zik ), k = 1, 2, ..., K,

(7)

where ⊙ denotes element-wise multiplication, ϵi ∈ RK is a random noise vector drawn from the standard multivariate normal distribution N (0, I), and sigmoid(z) = 1/(1 + exp(−z)). The

20

sigmoid function ensures zik ∈ (0, 1), allowing it to be interpreted as the probability of role k being assigned to node vi . This reparameterization-based sampling procedure faithfully instantiates a distribution that is multivariate logit-normal. The resulting latent variables Z then serve as a semantically meaningful and probabilistically grounded representation of the operational roles played by the basic blocks of the given CFG. 4.1.3 Generative Model To guide the learning of latent operational roles toward reflecting meaningful execution behavior, we design a generative model pθ (A|Z) that reconstructs the original CFG structure based on the latent roles of its constituent basic blocks. The underlying intuition is that accurate reconstruction requires the latent roles to encode execution-related semantics. In particular, this generative model is responsible for generating the adjacency matrix A, which encodes the presence or absence of edges (i.e., control flow relationships), based solely on the latent role assignment information Z. Since the CFG structure is composed entirely of edges between basic blocks, generating the graph reduces to deciding, for every node pair (vi , vj ), whether Aij equals one or zero, given their corresponding nodes’ latent roles zi and zj . Therefore, the generative model defines the conditional probability of the full adjacency matrix as a product of the probabilities of individual matrix entries: pθ (A|Z) =

Y

pθ (Aij = 1 | zi , zj )

Aij =1

Y

pθ (Aij = 0 | zi , zj ).

(8)

Aij =0

To instantiate this model, we need to specify how the probability of edge presence or absence (i.e., pθ (Aij = 1 | zi , zj ) and pθ (Aij = 0 | zi , zj )) is determined from the latent roles. Conventional edge generation strategies assume that similar latent variables should imply high likelihood of edge formation (Kipf and Welling, 2016; Grover et al., 2019; Li et al., 2020). However, such an assumption does not necessarily hold in the case of CFGs. A distinctive property of CFGs is that nodes with similar or even identical latent roles may be located far apart in the graph without direct connections (Musgrave et al., 2024). This is common for reusable routines such as error21

handling or control-check blocks, which are invoked repeatedly at different points during program execution (Haq and Caballero, 2021). For instance, an error-handling routine may occur in multiple branches of a software yet remain disconnected in the CFG. This property challenges the validity of conventional edge generation strategies in CFGs. As such, we require a more flexible edge generation strategy, one that incorporates role similarity as a useful signal while still allowing for the absence of edges between nodes with similar or identical roles. In this regard, we introduce a novel edge generation strategy with a learnable gate, which is explicitly designed to capture the structural particularity of CFGs. Specifically, for edge presence, we follow the intuition that nodes with highly similar roles are more likely to be connected. Edge absence, however, is more nuanced: two nodes may be disconnected either because they serve different roles, or because, despite having similar roles, they lie on distinct branches in the control flow and therefore are not directly connected in the CFG. To model these two cases of edge absence, we introduce a learnable gate M that adaptively determines whether an absent edge should be explained by role dissimilarity or by the fact that two nodes with similar roles are in distinct branches in the control flow. Formally, the edge generation likelihoods are defined as:  
  pθ (Aij = 1 | zi , zj ) = sigmoid sim(zi , zj ) ,

(9)


  pθ (Aij = 0 | zi , zj ) = sigmoid Mij (1 − sim(zi , zj )) + (1 − Mij )C ,
Mij = sigmoid MLPθ (zi , zj ) ,

(10)

where sim(zi , zj ) = z⊤ i zj /(∥zi ∥2 ∥zj ∥2 ) denotes the similarity between the latent roles zi and zj , and MLPθ is an MLP (Multi-Layer Perceptron) layer specified by learnable parameters θ. The constant C provides a fallback likelihood for edge absence when two nodes have similar roles but are not directly connected, for example because they lie on different branches of the control flow. To modulate this effect, we introduce a gating matrix M ∈ R|V |×|V | , where each entry Mij controls how the absence score is explained. When Mij is large, edge absence is primarily attributed to role 22

dissimilarity. When Mij is small, the fallback term C becomes dominant, allowing the model to account for unconnected node pairs with similar roles. 4.1.4 Learning Objective To train the model, our objective is to ensure that the learned latent roles are capable of reconstructing the original CFG structure as accurately as possible. Formally, given the node attribute matrix X, we aim to maximize the marginal log-likelihood of the observed adjacency matrix A under the generative model pθ (·): Z log pθ (A|X) = log

pθ (A|Z)p(Z|X)dZ,

(11)

where pθ (A | Z) models the likelihood of reconstructing the graph structure given Z, and p(Z|X) = Q|V |

6 i=1 p(zi |X) is the prior distribution of latent variables with p(zi |X) = LogitNormal(µ0 , Σ0 ).

However, the integral over Z in Equation (11) is intractable, making direct optimization of the marginal log-likelihood computationally infeasible. To address this, we derive the evidence lower bound (ELBO) as a tractable surrogate to the intractable marginal log-likelihood using variational inference and Jensen’s inequality (Jensen, 1906; Jordan et al., 1999; Kingma and Welling, 2014; Rezende et al., 2014):
log pθ (A|X) ≥ LELBO (ϕ, θ; G) = Eqϕ (Z|A,X) log pθ (A|Z) − KL qϕ (Z|A, X) ∥ p(Z|X) . {z } | {z } | reconstruction loss

(12)

KL divergence

The reconstruction loss term in Equation (12) encourages the generative model to accurately reconstruct the adjacency matrix A from the latent variables Z. Here, the expectation is taken over the variational distribution qϕ (Z|A, X), which is intended to approximate the true posterior pθ (Z | A, X). In practice, the expectation is approximated by drawing a limited number of instances from qϕ (Z|A, X) based on the procedure defined by Equation (7). The Kullback-Leibler (KL) divergence term in Equation (12) regularizes the variational distribution qϕ (Z|A, X) to be close to the prior p(Z|X). Since both distributions factorize over nodes, this KL term decomposes as 6

We have implicitly assumed that p(A|Z, X) = p(A|Z), that is, A is conditionally independent of X given Z, which is common in the literature (Kingma and Welling, 2014). In practice, (µ0 , Σ0 ) can be set as (0, I).

23

P|V |

i=1 KLi , where each KLi is defined as:

1
|Σ0 |  ⊤ −1 KLi = KL qϕ (zi |A, X) ∥ p(zi |X) = tr Σ−1 . (13) 0 Σi + (µ0 − µi ) Σ0 (µ0 − µi ) − K + log 2 |Σi |

In summary, the learning objective is to maximize the ELBO with respect to both the variational parameters ϕ and the generative parameters θ: LELBO (ϕ, θ; G).

(14)

4.1.5 Operational Subgraph An operational subgraph is a connected group of basic blocks in a CFG that are strongly associated with the same operational role. We extract operational subgraphs after estimating the latent roles of basic blocks, so that the identified roles can be explicitly incorporated into subsequent graph learning. Specifically, for each node vi ∈ V in a CFG, we have the estimated latent role zi ∈ RK , where zik denotes the probability that vi is associated with role k.7 To identify operational subgraphs, we first determine the set of nodes strongly affiliated with each role. For each role k ∈ {1, . . . , K}, we define a node set Vk consisting of all nodes whose assignment probability to role k exceeds a predefined threshold τ ∈ (0, 1): Vk = {vi ∈ V | zik > τ }.

(15)

This thresholding operation allows a basic block to be associated with multiple roles while ensuring that only nodes with sufficiently high probability are assigned to that role. Next, for each Vk , we extract the induced subgraph G[Vk ] from the original CFG by including all edges in G that connect pairs of nodes within Vk . Since G[Vk ] may consist of multiple disconnected components, we further decompose it into its connected components. Each connected component is then treated as an individual operational subgraph corresponding to role k, and the resulting collection is denoted as OSk . By repeating this procedure for all K roles, we obtain a final set of operational subgraphs 7

In practice, zik can be computed in accordance with Equation (7) by fixing the noise vector ϵi as a zero vector to obtain deterministic values for subsequent computations.

24

v1

v6

v7

0.34 0.93 0.21 0.39

v5

0.26 0.84 0.08 0.16 0.19 0.23 0.67 0.78

v8

0.25 0.75 0.06 0.69

v 10

z 8 0.40 0.34 0.37 0.78 z 9 0.63 0.22 0.09 0.36 z 10 0.85 0.19 0.05 0.23

G = (V,A,X ) v9 |V |=10

CFG with 10 basic blocks

Figure 5

threshold τ = 0.5

…

v4

Node assignment

role1 role2 role3 role4

z 1 0.91 0.35 0.12 0.07 z 2 0.78 0.68 0.33 0.10 z 3 0.38 0.41 0.80 0.13

v2 v3

Inference model

V 1 ={v 1,v 2,v 9,v 10} V 2 ={v 2,v 4,v 5,v 7} V 3 ={v 3,v 6} V 4 ={v 6,v 7,v 8}

𝒪𝒮={OS1,OS2,OS3,OS4,OS5} S=5 v1

OS1 v2 v3 OS3 v6 OS4

v7

v5

v8 v9 OS5 v 10

Extract the induced subgraph from G

Operational role identification module

v4

OS2

1

0

0

0

0

1

1

0

0

0

0

0

1

0

0

0

1

0

0

0

0

1

0

0

0

0

0

1

1

0

0

1

0

1

0

0

0

0

1

0

0

0

0

0

1

0

0

0

0

1

CFG with 5 operational subgraphs

Illustrative example of the operational role identification process.

OS from the CFG G: OS =

K [

OSk = {OS1 , OS2 , · · · , OSS },

(16)

k=1

where S is the total number of extracted operational subgraphs across all roles from CFG G. We can represent the assignment of nodes to these operational subgraphs by a binary assignment matrix B ∈ {0, 1}|V |×S , where Bis = 1 indicates that node vi in CFG G belongs to the s-th operational subgraph OSs , and Bis = 0 otherwise. Figure 5 illustrates a numeric example of the operational role identification process in MalGuard. Starting from a CFG G with |V | = 10 basic blocks, the inference model estimates the latent role variables Z with K = 4 latent roles. Based on a threshold of τ = 0.5, nodes are grouped into role-specific subsets, from which operational subgraphs (S = 5) are derived according to Equation (16). The resulting binary assignment matrix B ∈ {0, 1}10×5 encodes the membership relations between basic blocks and operational subgraphs, linking each node to its corresponding operational subgraph for subsequent graph learning. 4.2 Program Graph Learning for Malware Detection This section introduces a program graph representation learning method for malware detection. It learns software representations by modeling interactions among operational roles, preserving sparse but critical malicious evidence, and capturing hierarchical program structure. Figure 6 illustrates 25

Coarsening layer

Attention-based Gated pooling layer message passing layers

OS1 OS1 OS3 OS3

Gated pooling layer

hCG

f

OS3 OS2 OS2

CFG with operational subgraphs

OS1

Message passing layers

OS2

…

OS4 OS4

OS4

OS5 OS5

OS5

… CFG Graph representation

Coarsened graph (Asuper,X super)

Intra-function modeling

CG with node attributes

Software representation

Inter-function modeling

Figure 6 The overall framework of the program graph representation learning method for malware detection.

the overall framework. Specifically, the method captures hierarchical program structure by modeling software at both the intra- and inter-function levels. At the intra-function level, each function is represented as a CFG and encoded to model interactions among operational roles while preserving sparse but critical malicious signals within the function. The resulting function representations are then propagated to the CG. At the inter-function level, the CG is modeled to capture interactions among functions and to emphasize function-level signals that are most informative for malware detection, yielding the final software representation. We next describe the intra-function modeling, inter-function modeling, and learning objective. 4.2.1 Intra-Function Modeling The goal of intra-function modeling is to learn a representation for each function by encoding its CFG. This module consists of three key layers: (i) a coarsening layer, which converts the CFG from a graph of basic blocks into a graph of operational subgraphs, thereby enabling the model to capture interactions among operational roles; (ii) a stack of attention-based message passing layers, which aggregate information among operational subgraphs with learnable attention weights, allowing the model to preserve sparse but critical malicious signals from being diluted by routine neighbors during GNN aggregation; and (iii) a gated pooling layer, which combines operational subgraph rep-

26

resentations into a function representation through a learnable gate, allowing operational subgraphs carrying evidence relevant to malware detection to contribute more to the function representation. Specifically, the coarsening layer first abstracts the CFG of basic blocks into a coarsened graph of operational subgraphs, which captures operational subgraphs and their complex interactions and facilitates later learning of meaningful program behavior. Given a CFG G = (V, A, X) and its assignment matrix B ∈ {0, 1}|V |×S that represents the assignment of |V | nodes (in CFG G) to S operational subgraphs, we construct the coarsened graph as follows. Each operational subgraph is treated as a super-node that captures the collective role of its constituent basic blocks. Edges between super-nodes encode execution dependencies between operational subgraphs. Formally, the coarsened graph is represented by a weighted adjacency matrix Asuper ∈ RS×S and a super-node attribute matrix X super ∈ RS×d : Asuper = B ⊤ AB, Xssuper =

X

(17)


 ReLU Wsuper · AGG Xvj , ∀vj ∈ NOSs (vi ) ∪ {vi } ,

(18)

vi ∈OSs

where AGG is a permutation invariant aggregation function such as summation or average, NOSs (vi ) represents the set of neighboring basic blocks of basic block node vi in the subgraph OSs , and Wsuper is a learnable weight matrix. Equation (17) combines the original adjacency matrix A and the assignment matrix B as a coarsened adjacency matrix Asuper . This operation compresses the original |V | basic blocks into S super-nodes, and aggregates edges between basic blocks belonging to different operational subgraphs into edges between their corresponding super-nodes. Each entry Asuper reflects the number of control flow edges between basic block nodes in OSs and OSt , thereby st encoding their execution dependencies.8 Equation (18) takes the operational subgraph OSs and the attributes of its constituent basic blocks as input, and applies a single-layer GNN specified by Wsuper to generate the representation of this subgraph. The resulting representation of OSs serves 8

This can be seen from the fact that Asuper = st

P|V | P|V | i=1

j=1 Bis Aij Bjt =

27

P

i∈OSs

P

j∈OSt Aij .

as the attribute vector Xssuper for super-node vssuper , providing a compact summary of the subgraph semantics. With the coarsened graph (Asuper , X super ), the next step is to update each operational subgraph representation by aggregating information from its neighbors. Message passing GNNs are a natural choice here, owing to their expressive power to model graph-structured data (Kipf and Welling, 2017; Hamilton et al., 2017). A common practice in standard message passing is to uniformly aggregate information from all neighbors when updating a node representation. However, this practice is not ideal for malware detection and can even undermine detection accuracy. In practice, malware typically contains only a small fraction of operational subgraphs that carry malicious cues, while the majority reflect common behaviors (Alam et al., 2015). Consider the case where a malicious operational subgraph is adjacent to common ones; its signal would be overwhelmed during aggregation with common neighbors, causing the model to overlook these sparse yet critical malicious patterns. This calls for a more selective message passing mechanism that can adaptively assign different importance to neighbors during aggregation. To this end, we introduce an attention-based message passing layer to model the relative importance of different neighboring super-nodes during message passing. This design is inspired by the graph attention mechanism (Veličković et al., 2018), where learnable attention weights determine the contribution of each neighbor when updating a super-node representation. Specifically, the representation of super-node vssuper is updated by aggregating information from its neighboring super-nodes vtsuper ∈ N (vssuper ), with each neighbor’s influence reflected by the learnable attention (l)

weight αst . This process can be formulated as: gs(l) = ReLU (0)



(l) (l) (l−1) Wrole · AGG αst gt , ∀vtsuper ∈ N (vssuper )



,

(l)

(19)

where gs = Xssuper , Wrole ∈ Rd×d is a learnable weight matrix at layer l, and N (vssuper ) = {vtsuper | Asuper > 0} ∪ {vssuper } denotes the neighboring super-nodes of vssuper in the coarsened graph. st 28

(l)

The attention weight αst is learned from the representations of the focal super-node vssuper and (l−1)

its neighbor vtsuper (i.e., gs

(l−1)

and gt

), thereby reflecting the relative influence of neighbor vtsuper

on vssuper . For example, one possible outcome is that when vssuper carries malicious signal while vtsuper (l)

reflects common behaviors, the learned αst becomes lower, preventing the malicious signal in vssuper from being overwhelmed by common behaviors. Moreover, we incorporate the edge weights Asuper st into the attention computation, ensuring that execution dependencies observed more frequently exert greater influence during message passing. Formally, the attention weight is defined as:   h i (l−1) (l−1) ⊤ Asuper exp LeakyReLU a W g W g s attn attn st t (l) i .   h (20) αst = P (l−1) (l−1) super ⊤ W super super Asr W g exp LeakyReLU a g attn r attn s vr ∈N (vs ) In this equation, each super-node representation is first linearly transformed by a shared weight matrix Wattn ∈ Rd×d . The transformed representations are then concatenated (denoted by ∥) and projected onto a scalar score by a learnable attention vector a ∈ R2d . This score is weighted by the edge weight Asuper and passed through a LeakyReLU activation. Finally, the normalization ensures st that

P

super

vt

super

∈N (vs

(l) super . ) αst = 1, making the attention weights comparable across the neighbors of vs (L)

By stacking L layers of attention-based message passing, we obtain the final representation gs

for

each operational subgraph OSs ∈ OS. Finally, the gated pooling layer is designed to combine the operational subgraph representations into a unified CFG representation through a learnable fusion gate. Unlike traditional mean or sum pooling, which treats all operational subgraph representations equally and may therefore dilute the contribution of rare but decisive subgraphs for malware detection, the fusion gate learns how much each operational subgraph should contribute to the final CFG representation. Specifically, the fusion gate β = (β1 , β2 , . . . , βS )⊤ ∈ RS is computed from the representations of operational subgraphs and serves as a set of learnable weights that suppress redundant or noisy signals while amplifying rare but decisive ones. This design ensures that the final CFG representation f ∈ Rd preserves the

29

most informative semantics for malware detection. Formally, the process can be expressed as: (L)
⊤ X exp WCFG gs (L) , (21) f= βs gs , βs = P (L)
⊤ g exp W r CFG OSs ∈OS OSr ∈OS where WCFG ∈ Rd is a learnable vector. The CFG representations obtained from all functions are then stacked to form FCFG ∈ R|VCG |×d . 4.2.2 Inter-Function Modeling Given the CG with node attribute matrix FCFG , we apply a stack of message passing layers to model how functions interact through the call structure and collectively realize global software behaviors. At the l-th layer of the GNN, the representation of a function node vm is updated by aggregating information from its neighbors and itself through an aggregation function AGG, followed by a linear transformation and a nonlinear activation: 
 (l) (l−1) h(l) = ReLU W · AGG h , ∀v ∈ N (v ) ∪ {v } , n m m m n func (l)

(22)

(0)

where hm represents the representation of function node vm at layer l, hm is the m-th row of FCFG , (l)

N (vm ) is the set of neighboring function nodes of vm in the CG, and Wfunc ∈ Rd×d is a learnable weight matrix. By stacking L message passing layers, we obtain the final function representations (L)

hm for vm ∈ VCG . Then, we adopt a gated pooling layer similar to the one used when performing intra-function modeling. This layer learns importance scores over function nodes, allowing functions carrying signals relevant to malware detection to contribute more to the final software representation. Specifically, the fusion gate γ = (γ1 , γ2 , . . . , γ|VCG | )⊤ ∈ R|VCG | assigns an importance weight to each function node. The software representation hCG is then computed as: hCG =

X vm ∈VCG

(L)

γm h(L) m ,

γm = P

⊤ exp WCG hm

vn ∈VCG exp

(L)

⊤ WCG hn

,

(23)

where γm is the learned importance weight assigned to function node vm , and WCG ∈ Rd is a learnable vector.

30

4.2.3 Learning Objective Finally, the software representation hCG is passed through an MLP layer parameterized by ψ, followed by a sigmoid activation, to produce the malware detection probability of the input software: ŷ = sigmoid (MLPψ (hCG )) .

(24)

A higher value of ŷ indicates a greater likelihood that the software is malicious. The malware detection task can be formulated as a binary classification problem, where the objective is to determine whether a given software is malicious or benign. To this end, the framework is trained by minimizing the binary cross-entropy loss between the predicted probability ŷ and the ground-truth label y ∈ {0, 1}: LBCE (Θ) = − (y log ŷ + (1 − y) log(1 − ŷ)) ,

(25)

where Θ = {Wsuper , Wrole , Wattn , Wfunc , WCFG , WCG , a, ψ} are the trainable parameters. 4.3 Overall Algorithm We integrate the components introduced above into a two-stage training procedure. The first stage performs operational role identification, which infers latent roles for basic blocks and extracts operational subgraphs. The second stage performs program graph learning, which learns software representations from hierarchical program graphs for malware detection. The overall training procedure of MalGuard is summarized in Algorithm 1. Stage 1, operational role identification, corresponds to lines 2–13. In this stage, MalGuard trains the latent role identifiall cation module over mini-batches of CFGs sampled from GCFG . For each mini-batch, it computes

the parameters of the latent role distribution (line 4), infers latent role variables Z for each CFG (line 5), evaluates the batch objective −LELBO (line 6), and updates ϕ and θ by gradient descent (line 7). After training, the inferred role affiliations are used to extract operational subgraphs and construct the assignment matrix B for each CFG (lines 10–13). Stage 2, program graph learning for

31

Algorithm 1 The Overall Training Procedure of MalGuard Input: A labeled set of software R Output: Learned model parameters ϕ, θ, Wsuper , Wrole , Wattn , Wfunc , WCFG , WCG , a, ψ all denote the 1: Construct the hierarchical graph H = (GCG , GCFG ) for each software instance in R, let GCFG collection of all CFGs from these hierarchical program graphs // Stage 1: Operational Role Identification 2: for epoch1 ∈ 1 : Nepoch1 do ▷ Nepoch1 : number of epochs in Stage 1 all 3: for each mini-batch of CFGs sampled from GCFG do 4: Compute µ, σ for each CFG in the mini-batch ▷ Equation (5) 5: Compute Z for each CFG in the mini-batch ▷ Equation (7) 6: Compute batch loss −LELBO over the mini-batch ▷ Equation (14) 7: Update ϕ and θ via gradient descent on the batch loss 8: end for 9: end for 10: for each CFG G = (V, A, X) ∈ Gall do 11: Extract operational subgraphs OS = {OS1 , . . . , OSS } 12: Construct B ∈ {0, 1}|V |×S 13: end for // Stage 2: Program Graph Learning for Malware Detection 14: for epoch2 ∈ 1 : Nepoch2 do ▷ Nepoch2 : number of epochs in Stage 2 15: for each software instance in R do 16: Retrieve its hierarchical program graph H = (GCG , GCFG ) 17: for each CFG G = (V, A, X) ∈ GCFG do 18: Retrieve OS and B associated with CFG G 19: Compute Asuper = B ⊤ AB and X super ▷ Equation (17), (18) (l) (l) 20: Compute αst , gs for l ∈ 1 : L ▷ Equation (19), (20) 21: Compute and store the CFG representation f ▷ Equation (21) 22: end for 23: Stack stored CFG representations as FCFG for GCG 24: Compute hCG ▷ Equation (22), (23) 25: Compute ŷ = sigmoid (MLPψ (hCG )) ▷ Equation (24) 26: Compute loss LBCE ▷ Equation (25) 27: Update Wsuper , Wrole , Wattn , Wfunc , WCFG , WCG , a, ψ with gradient descent on LBCE 28: end for 29: end for 30: return ϕ, θ, Wsuper , Wrole , Wattn , Wfunc , WCFG , WCG , a, ψ

malware detection, corresponds to lines 14–29. In this stage, MalGuard learns a representation of the hierarchical program graph for each software instance for malware detection. The intra-function modeling includes the coarsening layer, attention-based message passing layers, and gated pooling layer, which together produce a CFG-level representation for each function (lines 19–21). The inter-function modeling then applies message passing layers and a gated pooling layer over the CG to obtain the software-level representation hCG (line 24). Finally, MalGuard computes the malware prediction ŷ (line 25), evaluates the binary cross-entropy loss LBCE (line 26), and updates

32

the program graph learning parameters by gradient descent (line 27). The computational complexity analysis of MalGuard can be found in Appendix A. 5 EXPERIMENTS This section provides a comprehensive evaluation of the proposed method on a real-world, manually collected software dataset. We first describe the experimental setup in Section 5.1. Then, we evaluate the malware detection performance of MalGuard against various benchmarks in Section 5.2 and conduct ablation studies to evaluate the contribution of key components in MalGuard in Section 5.3. Section 5.4 provides a case study to demonstrate the interpretability of our model, and Section 5.5 presents an economic value analysis. 5.1 Experimental Setup Dataset. We manually constructed a dataset of malware and benign software from diverse sources, ensuring that it is both up-to-date and categorically diverse. For malware, we collected all newly released instances between February 14, 2024, and October 4, 2024 from VirusShare.com9 , a widely used repository that is continuously updated with new malware from multiple sources and provides researchers with access to live malware. These malware instances span a wide range of families, including trojans, ransomware, worms, and others, thereby reflecting the diversity of real-world malware. For benign software, we sourced applications from several reputable freeware websites, including Wiki DLL10 , NirSoft11 , and CNET12 , which collectively offer diverse benign software across different domains. To ensure the reliability of ground truth labels, all instances were crossverified with VirusTotal13 , an industry-standard platform that provides reports of malware detection outcomes from over 70 commercial antivirus engines. Each collected instance was compared against VirusTotal reports, and instances showing inconsistencies with their ground-truth labels 9

https://virusshare.com/ https://wikidll.com/ 11 https://www.nirsoft.net/ 12 https://download.cnet.com/ 13 https://virustotal.com/ 10

33

were excluded to reduce potential mislabeling. In total, 24 such instances were identified and removed from the dataset. To further reflect real-world class imbalance, where benign software is typically more prevalent, we adopted a 1:5 ratio of malicious to benign instances. This results in the final dataset containing 1,428 malware instances and 7,342 benign instances. Benchmark methods. We benchmarked our method against representative ML malware detection methods. First, we compared our method with byte-based methods, which operate directly on the raw byte sequences of the program. The comparison with these methods demonstrates the advantage of graph-based methods in defending against malware with evasive behaviors. • MalConv (Raff et al., 2018), which is a prominent state-of-the-art benchmark that embeds byte sequences into a vector space and applies a gated convolutional network for detection. • NonNeg (Fleshman et al., 2019), which is a variant of MalConv designed to detect evasive malware by constraining the final layer weights to be non-negative. • DRSM (Saha et al., 2024), another MalConv variant that improves robustness against evasive behaviors by dividing input bytes into segments, applying a pre-trained MalConv model to each segment, and combining the results through a voting strategy. We further evaluated our method against graph-based ML malware detection methods that detect malware by representing software as program graphs to capture execution logic and mitigate evasive behaviors. This comparison demonstrates the added value of our method in modeling program behaviors more effectively, thereby enabling more accurate malware detection. • GCN (Kipf and Welling, 2017), GAT (Veličković et al., 2018), GIN (Xu et al., 2019), which are standard GNN baselines applied to hierarchical program graphs for malware detection. • MAGIC (Yan et al., 2019), which leverages programs’ CFGs and applies DGCNN (Zhang et al., 2018) to these graphs for malware detection. • DeepCall (Chen et al., 2023), which builds on programs’ CGs and also employs DGCNN for

34

malware detection. • Mal2GCN (Kargarnovin et al., 2024), which leverages programs’ CGs and incorporates a non-negative weight constraint on GNN to enhance robustness against evasive malware. • MalGraph (Ling et al., 2022), which exploits hierarchical program graphs and introduces hierarchical graph learning for detection. Evaluation metrics. Considering the inherent class imbalance in malware detection, evaluation requires metrics that go beyond overall accuracy. We therefore adopted three evaluation metrics: F1-score, AUC (the area under the receiver operating characteristic curve, or ROC curve), and AUPRC (the area under the precision–recall curve). In all cases, malware instances were treated as the positive class and benign software as the negative class. Specifically, the F1-score is the harmonic mean of Precision and Recall, where Recall measures the proportion of malware correctly identified and Precision measures the proportion of predicted malware that are truly malicious, i.e., F1 = 2(Precision·Recall)/(Precision+Recall). AUC is computed as the area under the ROC curve, which plots the true positive rate against the false positive rate at varying thresholds. It reflects the probability that a randomly chosen malware instance receives a higher detection score than a benign one. AUPRC is computed as the area under the Precision-Recall curve, which plots precision against recall at varying thresholds and is particularly informative under class imbalance. The dataset was split into training, validation, and testing sets using an 8:1:1 ratio for model evaluation. Implementation details.

We conducted all experiments on a single NVIDIA GeForce RTX

3090 GPU with 24GB memory. All models were implemented in Python 3.8 with PyTorch library. Specifically, when implementing our method, we set the hidden dimension d to 128, the number of GNN layers L to 2, all MLPs were composed of 2 layers, and all aggregation functions AGG(·) were implemented as the mean operator. In the operational role identification approach, the fallback likelihood for edge absence C was fixed at 0.3, the role assignment threshold τ was set to 0.5, and

35

we adopted a standard multivariate logit-normal prior with µ0 = 0 and Σ0 = I. For optimization, we employed the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 1 × 10−3 and a batch size of 64. Training was performed with early stopping based on the validation loss to mitigate overfitting. To construct program graphs, we disassembled all software instances in the dataset using the Python program analysis framework angr14 , extracting CGs and CFGs. The original node attributes of CFG were initialized with 13 statistical features, including the numbers of call, transfer, arithmetic, logic, compare, move, and termination instructions, as well as counts of data declarations, total instructions, string constants, integer constants, in-degree, and out-degree. For fair comparisons, the hyperparameters of all baseline models were tuned to their optimal settings. MalConv, NonNeg, MAGIC, DeepCall, and Mal2GCN were strictly re-implemented following their original papers. The GNN-based baselines (GCN, GAT, and GIN) were implemented using the PyG library. For DRSM and MalGraph, we utilized the official implementations released by the authors. Details can be found in Appendix B. 5.2 Malware Detection Performance We first evaluated the effectiveness of MalGuard on malware detection. Table 2 reports the results on our collected dataset. Notably, MalGuard consistently achieves the best performance, with relative improvements of 5.16% in F1-score, 3.44% in AUC, and 3.91% in AUPRC compared with the best-performing benchmark. All improvements achieved by MalGuard are statistically significant (p < 0.05). Our method’s superiority over byte-based methods arises from its ability to model programs as program graphs, which effectively capture execution logic and provide inherent robustness against evasive malware. The performance gap between our method and other graphbased methods underscores the importance of explicitly identifying and incorporating operational roles, while jointly modeling hierarchical program structure and preserving sparse malicious signals.

14

https://angr.io/

36

Table 2

Malware detection performance of our method compared with byte-based and graph-based benchmarks. The best and second-best performances are highlighted in bold and underline, respectively. All values are reported as means with standard deviations in parentheses. Relative improvements of MalGuard over benchmarks are reported.

Category

Method MalConv

Byte-Based

NonNeg DRSM GCN GAT GIN

Graph-Based

MAGIC DeepCall Mal2GCN MalGraph

MalGuard

F1-score 87.03∗∗∗ (0.53) 80.50∗∗∗ (2.35) 87.21∗∗ (1.15) 82.90∗∗ (2.60) 84.47∗∗∗ (1.51) 81.25∗∗∗ (2.18) 83.47∗∗∗ (2.44) 87.77∗∗∗ (1.72) 77.33∗∗∗ (1.60) 82.76∗∗ (3.47) 92.30 (0.73)

Improvement by MalGuard 6.06% 14.66% 5.84%

AUC 89.27∗∗∗ (0.40) 84.16∗∗∗ (1.68) 90.06∗∗ (1.38)

11.53%

87.89∗ (2.80) 90.75∗ (1.16) 87.67∗∗ (2.60) 89.01∗∗ (1.80) 91.44∗∗∗ (0.87) 81.86∗∗∗ (0.99) 87.49∗ (3.68)

–

94.59 (0.70)

11.34% 9.27% 13.60% 10.58% 5.16% 19.36%

Improvement by MalGuard 5.96% 12.39% 5.03%

AUPRC 89.67∗∗ (0.48) 85.66∗∗ (1.64) 89.37∗∗∗ (0.59)

8.12%

85.50∗∗∗ (1.52) 85.76∗∗∗ (1.55) 83.52∗∗∗ (1.11) 85.25∗∗∗ (2.06) 89.21∗∗ (1.69) 82.87∗∗∗ (1.33) 85.78∗∗ (2.18)

–

93.18 (0.63)

7.62% 4.23% 7.89% 6.27% 3.44% 15.55%

Improvement by MalGuard 3.91% 8.78% 4.26% 8.98% 8.65% 11.57% 9.30% 4.45% 12.44% 8.63% –

Note. Statistical significance is marked as * (p < 0.05), ** (p < 0.01), and *** (p < 0.001).

More specifically, existing graph-based benchmarks commonly overlook operational roles and fail to adequately capture malicious signals. GCN, GIN, and GAT serve as standard GNN baselines that are not tailored for malware detection. MAGIC, DeepCall and Mal2GCN are specifically designed for malware detection, but they model programs either as CFGs or CGs and thus cannot exploit hierarchical program structure. MalGraph models programs as hierarchical graphs, but it fails to disentangle operational roles and overlooks malicious signals, resulting in weaker performance. We further evaluated MalGuard and the benchmark methods under more challenging conditions by injecting evasive behaviors into the malware instances in our test set with six representative evasion techniques. Results indicate that MalGuard achieves improvements over all benchmarks, particularly byte-based methods, underscoring its strong capability to defend against evasive malware. Detailed results are provided in Appendix C.

37

Table 3

Ablation study results. ∆ values show the relative performance drop observed after removing each component from MalGuard. Method

F1-score (∆)

AUC (∆)

AUPRC (∆)

MalGuard

92.30

94.59

93.18

86.69 (↓6.08%) 86.95 (↓5.80%) 89.78 (↓2.73%) 90.77 (↓1.66%) 90.56 (↓1.89%)

89.12 (↓5.78%) 90.31 (↓4.52%) 92.57 (↓2.14%) 92.80 (↓1.89%) 92.78 (↓1.91%)

87.39 (↓6.21%) 88.06 (↓5.49%) 90.40 (↓2.98%) 91.65 (↓1.64%) 91.34 (↓1.97%)

MalGuard-OR MalGuard-Hier MalGuard-CL MalGuard-Attn MalGuard-GP

5.3 Ablation Studies We conducted ablation studies to evaluate the contribution of key components in MalGuard. Specifically, we assessed five variants of our method. (1) MalGuard-OR removes the operational role identification module in our method, operating without both the role identification and the role-based coarsening layer. (2) MalGuard-Hier overlooks the hierarchical program structure and learns only from CFGs, producing the final software representation by directly pooling CFG representations without inter-function modeling. (3) MalGuard-CL removes the coarsening layer and instead incorporates the role information as an additional node attribute in CFGs. (4) MalGuard-Attn removes the attention-based message passing layers and instead adopts the standard message passing as GCN (Kipf and Welling, 2017). (5) MalGuard-GP removes the gated pooling layer and instead uses sum pooling as GIN (Xu et al., 2019). The results are summarized in Table 3. We observe that all variants perform worse than MalGuard, underscoring the effectiveness of each component. Notably, removing the operational role identification (MalGuard-OR) results in the most severe degradation, highlighting the necessity of identifying and leveraging operational roles in malware detection. The performance decrease of MalGuard-Hier demonstrates the importance of jointly modeling intra- and inter-function levels in the hierarchical program graph. Moreover, the performance drops observed for MalGuard-CL, MalGuard-Attn, and MalGuard-GP indicate the effectiveness of the coarsening layer, attentionbased message passing layer, and gated pooling layer in modeling interactions among operational

38

roles, preserving malicious evidence while modeling the hierarchical graph structure. 5.4 Case Study To demonstrate how MalGuard identifies semantically meaningful operational roles and provides interpretable insights into malware behavior, we conduct a case study on a representative malware instance, “MemoryDiagnostic.exe”. Specifically, our objective is to examine whether the operational roles uncovered by MalGuard correspond to meaningful behavioral patterns that align with expert analysis, and whether such abstractions can reveal latent malicious intent that is not observable at the level of individual basic blocks. The analyzed instance is a prototypical Trojan that deceptively masquerades as a legitimate system diagnostic tool. Its objective is to lurk within an infected system, perform extensive environmental reconnaissance, and subsequently download more destructive payloads such as info-stealers, while potentially propagating to other hosts on the same network. We focus on a specific function with the entry address 0x40210a, which receives a high importance weight in Equation (23) during the inter-function modeling stage of MalGuard, indicating its importance in the overall detection decision. Through the operational role identification module, MalGuard automatically uncovered a series of operational subgraphs within this function. As illustrated in Figure 7, one identified operational subgraph, highlighted by the red dashed box, forms a long and nearly linear execution chain. Our manual reverse engineering reveals that this identified operational subgraph corresponds to a customized memory mutation routine, which likely serves as a preparatory decoding stage prior to malicious payload execution. Importantly, when examined at the level of individual nodes, these basic blocks appear largely benign, as they are dominated by arithmetic operations on obfuscated constants and routine memory updates. For example, the block at 0x402244 mainly contains ordinary operations such as subtracting values from memory locations and writing updated

39

… 0x40218e -------------------------lea edx, [ebx] add byte ptr [eax], al add dword ptr [eax], ebp sbb byte ptr [eax], al add byte ptr [edx], cl

0x40219a -------------------------add byte ptr [eax], al or bh, byte ptr [edx] sub ecx, dword ptr [ebx] add byte ptr [ebx + 0x1a], dh

0x4021b3 -------------------------push es add byte ptr [eax], al sbb ebp, dword ptr [eax] push ss add byte ptr [eax], al or dl, byte ptr [edi + ebp*2] sbb eax, dword ptr [eax] add byte ptr [edx], cl add byte ptr [eax], al sub byte ptr [ecx], al add byte ptr [eax], al sub ecx, dword ptr [edx] fidivr word ptr [ebp - 0x22] jb 0x402244

0x4021a4 -------------------------add byte ptr [eax], al or al, byte ptr [eax + 0x4000006] add byte ptr [esi + 6], bh add byte ptr [eax], al add al, 0xd0

0x402244 -------------------------add cl, byte ptr [edx] sub eax, dword ptr [ecx] add byte ptr [esi], al sub al, byte ptr [eax] add dword ptr [eax + eax], ebx add byte ptr [ecx], al add byte ptr [eax + eax + 0x3700ba0a], cl xchg esi, eax add byte ptr [eax], al add byte ptr [edx], al add byte ptr [eax + eax + 0x1700f165], cl add byte ptr [eax], al add byte ptr [ecx], al …… sub eax, dword ptr [eax] push es sub al, byte ptr [eax] add byte ptr [ebx], dl xor byte ptr [ecx], al add byte ptr [eax + eax], cl add byte ptr [eax], al or dword ptr [eax], eax add byte ptr [ecx], dl add byte ptr [edx], al sub byte ptr [edi], ah add byte ptr [eax], al or cl, byte ptr [edx] sub eax, dword ptr [eax] push es sub dl, byte ptr [ebx] xor byte ptr [ecx], al add byte ptr [eax], dl

0x4022dd -------------------------add byte ptr [eax], al add byte ptr [edx], cl add byte ptr [eax], al adc dword ptr [eax], eax rol byte ptr [0x28020000], 1 push ss add byte ptr [eax], al or cl, byte ptr [edx]

One of the identified operational subgraphs

0x4022f0 -------------------------sub eax, dword ptr [eax] push es sub dl, byte ptr [ebx] xor byte ptr [ecx], al add byte ptr [eax + eax], cl add byte ptr [eax], al or eax, dword ptr [eax] add byte ptr [ecx], dl add byte ptr [edx], al sub byte ptr [eax], ch add byte ptr [eax], al or cl, byte ptr [edx] sub eax, dword ptr [eax] push es

0x402321 -------------------------add byte ptr es:[eax], al or cl, byte ptr [edx] sub eax, dword ptr [eax] push es sub al, byte ptr [eax] add byte ptr [ebx], dl xor byte ptr [ecx], al add byte ptr [eax + eax], cl add byte ptr [eax], al or dword ptr [eax], eax add byte ptr [ecx], dl add byte ptr [edx], al sub byte ptr [edi], ah …… add byte ptr [eax], ah add byte ptr [eax], al add byte ptr [eax + eax], cl add byte ptr [ecx], dl add byte ptr [edx], al mov word ptr [esi], es add byte ptr [eax], al sbb edx, dword ptr [esi + edi*8] add dword ptr [ebx], ecx pop es sub al, 0xa sub byte ptr [ecx], al add byte ptr [eax], al

0x402397 --------------------------

0x4023ab --------------------------

or ch, byte ptr [ebx] or byte ptr [ebx], ch add eax, 0x2b0a0200 add dword ptr [eax], eax push es sub dl, byte ptr [ebx] xor byte ptr [edx], al add byte ptr [edx], dl add byte ptr [eax], al

pop es add byte ptr [eax], al adc dword ptr [eax], eax add edx, dword ptr [edx] add dh, bh

…

…

Figure 7 Operational subgraph identified by MalGuard in a case study. The figure shows a partial CFG of the function located at entry address 0x40210a in the malware instance MemoryDiagnostic.exe. The red dashed box highlights an operational subgraph identified by MalGuard.

bytes back to memory. Similarly, the block at 0x402321 performs simple arithmetic and bytelevel memory updates, including adding a constant to a register and updating memory through instructions. Viewed in isolation, these instructions do not directly indicate a clear malicious action. However, it is precisely their coordinated execution that gives rise to suspicious behavior. Across the 40

identified operational subgraph, these blocks repeatedly read values from memory, transform them through arithmetic operations, and write the transformed values back to memory. This coordinated pattern suggests that the program is progressively modifying a memory region, which is consistent with a decoding or unpacking routine used before the actual payload is executed. Such routines are commonly suspicious in malware analysis because they allow malicious code to remain hidden in the file and only reveal its executable form at runtime, making it harder for detectors to identify the harmful behavior before execution. By aggregating these nodes into a coherent operational subgraph, MalGuard is able to capture this latent intent, which would be overlooked by methods focusing solely on local patterns. This case illustrates that malicious behavior is often not localized within individual nodes but emerges from coordinated execution patterns across multiple nodes in the program graph. By shifting the analytical focus from individual nodes to semantically meaningful operational roles, MalGuard enables more reliable and interpretable malware detection. Such interpretability is particularly valuable in organizational settings, where security analysts require transparent and actionable evidence to support incident response, auditing, and risk assessment. 5.5 Economic Value Analysis We further examined the economic value of our proposed method, MalGuard, in the context of organizational cybersecurity. Modern organizations face increasing exposure to malware attacks, which exploit vulnerabilities in computer systems to steal sensitive data, disrupt critical operations, or paralyze systems for ransom payments. Such incidents often escalate into large-scale security breaches that incur substantial economic losses. To mitigate these threats, enterprises often deploy proactive malware detection solutions designed to identify and block malicious software before it can cause harm. Nevertheless, due to inherent limitations in detection capabilities, no method is perfectly effective. Some malware instances are mistakenly classified as benign (i.e., negative)

41

Table 4

Economic value analysis of malware detection methods. Cost reductions indicate the estimated savings from replacing each baseline method with MalGuard. Method

FNR

Cost Reduction by MalGuard

Reduction Rate

MalGuard

9.77%

-

-

No Detection MalConv NonNeg DRSM GCN GAT GIN MAGIC DeepCall Mal2GCN MalGraph

100.00% 20.93% 31.32% 18.91% 22.33% 15.35% 21.86% 19.53% 15.50% 35.97% 23.41%

$2,358,568.89 $1,393,774.69 $1,798,552.54 $1,263,433.17 $1,470,274.84 $950,218.38 $1,445,685.26 $1,306,306.79 $966,319.03 $1,903,962.81 $1,523,037.39

90.23% 53.32% 68.81% 48.33% 56.25% 36.35% 55.31% 49.97% 36.97% 72.84% 58.27%

instances and remain active within the system. Accordingly, the risk posed by such undetected malware is quantified by the false negative rate (FNR) of the deployed model (Ye et al., 2017). Reducing the FNR is therefore critical to lessening the economic impact of cyberattacks, highlighting both the practical necessity and the economic benefit of more effective detection methods. More precisely, we let Costbaseline denote the annual malware-related economic loss under the baseline detection method, and Costours denote the loss under our proposed MalGuard for the same organizational setting. Assuming that the total cost is approximately proportional to the number of false negatives (i.e., malware mistakenly classified as benign), we have: Costours = Costbaseline ·

FNRours , FNRbaseline

(26)

where FNRbaseline is the false negative rate of the baseline model, and FNRours is that of MalGuard. Consequently, the cost reduction achieved by MalGuard compared to the deployed baseline method is given by: ∆Cost = Costbaseline − Costours   (27) FNRours = Costbaseline · 1 − . FNRbaseline According to Accenture’s cybercrime study, malware attacks incur an average annual cost of $2,613,952 per company.15 We thus set Costbaseline to $2,613,952. Table 4 presents the FNRs of 15

See https://www.accenture.com/content/dam/accenture/final/a-com-migration/pdf/pdf-96/accenture-2019-costof-cybercrime-study-final.pdf (last accessed on Jul 23, 2026).

42

different baseline detection methods along with the corresponding relative cost reductions that would be realized if these baseline methods were replaced with MalGuard. In addition, we include a No Detection case, representing a situation in which the focal organization adopts no malware detection method (i.e., FNR = 100%). As shown, MalGuard achieves the lowest FNR among all approaches, leading to substantial reductions in economic cost. In particular, compared to the best-performing byte-based baseline (MalConv) and the best-performing graph-based baseline (DeepCall), our method achieves cost reductions of 53.32% and 36.97%, respectively. These results highlight that the superior detection performance of MalGuard contributes to substantial economic value by more effectively detecting malware. 6 CONCLUSION 6.1 Summary and Contributions Malware remains a major cybersecurity threat to organizations. Although organizations commonly rely on ML malware detection methods to mitigate malware risks, existing approaches still face important limitations. Byte-based methods learn from raw byte sequences and are therefore vulnerable to evasive behaviors. Graph-based methods provide a more robust alternative by representing software through program graphs, but they often overlook operational roles and fail to learn sufficiently expressive program graph representations for accurate malware detection. To address these limitations, we propose MalGuard, a novel graph-based malware detection method for enhancing malware risk management. Specifically, we first introduce an operational role identification approach that infers latent roles in CFGs and extracts the corresponding operational subgraphs. Building on the identified operational roles, we further develop a program graph representation learning method that models interactions among operational roles, preserves sparse malicious signals, and captures the hierarchical structure of program graphs for malware detection. Extensive experiments on a real-world software dataset show that MalGuard outperforms representative ML malware detection

43

methods and reduces the expected cost of undetected malware for organizations. Our study contributes to the extant literature in two ways. First, our study belongs to the computational design science paradigm in the IS field, which aims to solve important business and societal problems and makes significant methodological contributions to the IS literature (Padmanabhan et al., 2022; Fang et al., 2025; Li et al., 2021). In this regard, we focus on malware risk management, a critical cybersecurity challenge faced by organizations, and propose a novel graph-based malware detection method to improve organizations’ ability to identify malware before it causes operational and financial damage. Specifically, the key innovations of MalGuard, operational role identification approach and program graph learning method for malware detection, constitute the core methodological contributions of this work. Second, our study contributes to the broad IS cybersecurity literature (Galbreth and Shor, 2010; Ebrahimi et al., 2025). By modeling the characteristics of program graphs and software execution, MalGuard enables organizations to better detect malware, including malware exhibiting evasive behaviors. In doing so, our study extends IS cybersecurity research by demonstrating how MalGuard can be used to support organizational malware risk management and reduce the economic consequences of cyber threats. 6.2 Implications and Future Work Beyond detection performance, our study also offers several practical implications. First, by adopting our proposed method, software users can screen potentially malicious software before it enters their IT infrastructures, thereby substantially reducing the expected costs associated with undetected malware. This capability is particularly important for public-facing and criticalinfrastructure organizations, such as financial institutions, transportation operators, and public utility providers. In these cases, a successful malware infection may not only disrupt internal operations but also interrupt essential services, compromise sensitive citizen or customer data, and generate cascading economic and social consequences. Therefore, our method provides

44

organizations with a practical ex ante cost-effective mechanism that helps reduce the broader organizational and societal costs of malware incidents. Second, our study provides implications for cybersecurity governance and policy. Specifically, malware exhibiting evasive behaviors is increasingly common in practice and is intentionally designed to bypass automated detectors while preserving harmful functionality. Cybersecurity governance should therefore move beyond static checklists and encourage the adoption of detection mechanisms that explicitly account for evasive and adaptive threats. Furthermore, our study has broader societal implications. In increasingly digitized societies, public trust in digital technologies is essential for the adoption of IT-enabled services. Malware incidents can undermine this trust by heightening users’ perceived security risks, privacy concerns, and distrust toward digital platforms and institutions. By improving the detection of malicious software, our method helps reduce the likelihood and impact of such incidents, thereby strengthening societal trust in digital environments. This enhanced trust may further promote the diffusion of IT-enabled applications, thereby contributing to social and economic development. Our study opens up multiple directions for future research. First, adapting graph-based malware detection to continuously evolving malware threat presents a valuable future direction. In realworld cybersecurity environments, malware authors often develop new evasive behaviors over time, which may reduce the effectiveness of detectors trained on currently observed malware instances. Therefore, future studies could extend MalGuard to support continuous adaptation under emerging threats by incorporating concept drift detection (Lu et al., 2018), cross-domain learning (Li and Tuzhilin, 2021), or adversarial training (Madry et al., 2018). Second, improving the method’s scalability and efficiency for practical enterprise deployment presents another important direction. In real organizational settings, malware detection systems need to scan large volumes of files with minimal delay. More scalable and efficient approaches could further enhance the practical value of the proposed method in organizational cybersecurity operations.

45

REFERENCES Abbasi, A., Dobolyi, D., Vance, A., and Zahedi, F. M. (2021). The phishing funnel model: A design artifact to predict user susceptibility to phishing websites. Information Systems Research, 32(2):410–436. Afianian, A., Niksefat, S., Sadeghiyan, B., and Baptiste, D. (2019). Malware dynamic analysis evasion techniques: A survey. ACM Computing Surveys, 52(6):1–28. Aitchison, J. (1982). The statistical analysis of compositional data. Journal of the Royal Statistical Society: Series B (Methodological), 44(2):139–160. Aitchison, J. and Shen, S. M. (1980). Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261–272. Alam, S., Traore, I., and Sogukpinar, I. (2015). Annotated control flow graph for metamorphic malware detection. The Computer Journal, 58(10):2608–2621. Alfred, V. A., Monica, S. L., and Jeffrey, D. U. (2007). Compilers principles, techniques & tools. pearson Education. Allen, F. E. (1970). Control flow analysis. ACM Sigplan Notices, 5(7):1–19. Ampel, B. M., Samtani, S., Zhu, H., and Chen, H. (2024). Creating proactive cyber threat intelligence with hacker exploit labels: A deep transfer learning approach. MIS Quarterly, 48(1):137–166. Anderson, H. S. and Roth, P. (2018). EMBER: an open dataset for training static PE malware machine learning models. arXiv preprint arXiv:1804.04637. August, T., Dao, D., and Niculescu, M. F. (2022). Economics of ransomware: Risk interdependence and large-scale attacks. Management Science, 68(12):8979–9002. Balsamo, S., Di Marco, A., Inverardi, P., and Simeoni, M. (2004). Model-based performance prediction in software development: A survey. IEEE Transactions on Software Engineering, 30(5):295–310. Bilot, T., El Madhoun, N., Al Agha, K., and Zouaoui, A. (2024). A survey on malware detection with graph representation learning. ACM Computing Surveys, 56(11):1–36. Bruschi, D., Martignoni, L., and Monga, M. (2006). Detecting self-mutating malware using controlflow graph matching. In Proceedings of the Third International Conference on Detection of Intrusions and Malware and Vulnerability Assessment, pages 129–143. Springer. Bulgurcu, B., Cavusoglu, H., and Benbasat, I. (2010). Information security policy compliance: an empirical study of rationality-based beliefs and information security awareness. MIS Quarterly, 34(3):523–548. Cesare, S. and Xiang, Y. (2010). Classification of malware using structured control flow. In Proceedings of the Eighth Australasian Symposium on Parallel and Distributed ComputingVolume 107, pages 61–70. Citeseer. Chen, Y., Feng, Y., Wang, C., Zhao, J., and Liu, Q. (2023). DeepCall: A fast and robust malware classification system with DGCNN and function call graph. In 2023 26th International Conference on Computer Supported Cooperative Work in Design, pages 1402–1408. IEEE. Christodorescu, M. and Jha, S. (2003). Static analysis of executables to detect malicious patterns. In Proceedings of the 12th USENIX Security Symposium, pages 169–186. Clement, N. and Arce, D. (2025). Dynamics of shared security in the cloud. Information Systems Research, 36(2):916–943. Cram, W. A., D’arcy, J., and Proudfoot, J. G. (2019). Seeing the forest and the trees: A meta-

46

analysis of the antecedents to information security policy compliance. MIS Quarterly, 43(2):525–553. Demetrio, L., Biggio, B., Lagorio, G., Roli, F., and Armando, A. (2019). Explaining vulnerabilities of deep learning to adversarial malware binaries. In Proceedings of the Third Italian Conference on Cyber Security, volume 2315 of CEUR Workshop Proceedings. Demetrio, L., Coull, S. E., Biggio, B., Lagorio, G., Armando, A., and Roli, F. (2021). Adversarial EXEmples: A survey and experimental evaluation of practical attacks on machine learning for Windows malware detection. ACM Transactions on Privacy and Security, 24(4):1–31. Dey, D. and Lahiri, A. (2026). “extortionality” in ransomware attacks: A microeconomic study of extortion and externality. Information Systems Research, 37(1):20–43. D’Elia, D. C., Coppa, E., Palmaro, F., and Cavallaro, L. (2020). On the dissection of evasive malware. IEEE Transactions on Information Forensics and Security, 15:2750–2765. Ebrahimi, M., Chai, Y., Samtani, S., and Chen, H. (2022). Cross-lingual cybersecurity analytics in the international dark web with adversarial deep representation learning. MIS Quarterly, 46(2). Ebrahimi, R., Chai, Y., Li, W., Pacheco, J., and Chen, H. (2025). RADAR: A framework for developing adversarially robust cyber defense AI agents with deep reinforcement learning. MIS Quarterly, 49(4):1385–1416. Fang, X., Hu, P. J., Chau, M., and Chen, H. (2025). Computational design science: A critical information systems research area contributing to artificial intelligence and data science. Available at SSRN 5455094. Feng, P., Gai, L., Yang, L., Wang, Q., Li, T., Xi, N., and Ma, J. (2024). DawnGNN: Documentation augmented Windows malware detection using graph neural network. Computers & Security, page 103788. Feng, X., Wu, S., Zhou, W., and Quan, M. (2017). Efficient locality weighted sparse representation for graph-based learning. Knowledge-Based Systems, 121:129–141. Fleshman, W., Raff, E., Sylvester, J., Forsyth, S., and McLean, M. (2019). Non-negative networks against adversarial attacks. In AAAI-2019 Workshop on Artificial Intelligence for Cyber Security. Galbreth, M. R. and Shor, M. (2010). The impact of malicious agents on the enterprise software industry. MIS Quarterly, 34(3):595–612. Gandotra, E., Bansal, D., and Sofat, S. (2014). Malware analysis and classification: A survey. Journal of Information Security, 5(2):56. Grover, A., Zweig, A., and Ermon, S. (2019). Graphite: Iterative generative modeling of graphs. In Proceedings of the 36th International Conference on Machine Learning, pages 2434–2444. Guo, H., Cheng, H. K., and Kelley, K. (2016). Impact of network structure on malware propagation: A growth curve perspective. Journal of Management Information Systems, 33(1):296–325. Hamilton, W. L., Ying, R., and Leskovec, J. (2017). Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1025–1035. Haq, I. U. and Caballero, J. (2021). A survey of binary code similarity. ACM Computing Surveys, 54(3):1–38. Jensen, J. L. W. V. (1906). Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta Mathematica, 30(1):175–193. Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. (1999). An introduction to variational

47

methods for graphical models. Machine Learning, 37:183–233. Kargarnovin, O., Sadeghzadeh, A. M., and Jalili, R. (2024). Mal2GCN: A robust malware detection approach using deep graph convolutional networks with non-negative weights. Journal of Computer Virology and Hacking Techniques, 20(1):95–111. Kim, S. H. and Kim, B. C. (2014). Differential effects of prior experience on the malware resolution process. MIS Quarterly, 38(3):655–678. Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations. Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In International Conference on Learning Representations. Kipf, T. N. and Welling, M. (2016). Variational graph auto-encoders. In Proceedings of the NIPS Workshop on Bayesian Deep Learning. Kipf, T. N. and Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations. Kirat, D. and Vigna, G. (2015). Malgene: Automatic extraction of malware analysis evasion signature. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 769–780. Kirat, D., Vigna, G., and Kruegel, C. (2014). {BareCloud}: Bare-metal analysis-based evasive malware detection. In 23rd USENIX Security Symposium (USENIX Security 14), pages 287–301. Kolter, J. Z. and Maloof, M. A. (2004). Learning to detect malicious executables in the wild. In Proceedings of the tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 470–478. Kreuk, F., Barak, A., Aviv-Reuven, S., Baruch, M., Pinkas, B., and Keshet, J. (2018). Deceiving end-to-end deep learning malware detectors using adversarial examples. arXiv preprint arXiv:1802.04528. Li, J., Yu, J., Li, J., Zhang, H., Zhao, K., Rong, Y., Cheng, H., and Huang, J. (2020). Dirichlet graph variational autoencoder. In Proceedings of the 34th International Conference on Neural Information Processing Systems, volume 33, pages 5274–5283. Li, P. and Tuzhilin, A. (2021). Dual metric learning for effective and efficient cross-domain recommendations. IEEE Transactions on Knowledge and Data Engineering, 35(1):321– 334. Li, W. W., Leung, A. C. M., and Yue, W. T. (2023). Where is IT in information security? the interrelationship among IT investment, security awareness, and data breaches. MIS Quarterly, 47(1):317–342. Li, Z., Ge, Y., and Bai, X. (2021). What will be popular next? predicting hotspots in two-mode social networks. MIS Quarterly, 45(2):925–966. Ling, X., Wu, L., Deng, W., Qu, Z., Zhang, J., Zhang, S., Ma, T., Wang, B., Wu, C., and Ji, S. (2022). MalGraph: Hierarchical graph neural networks for robust Windows malware detection. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications, pages 1998–2007. IEEE. Ling, X., Wu, L., Zhang, J., Qu, Z., Deng, W., Chen, X., Qian, Y., Wu, C., Ji, S., Luo, T., et al. (2023). Adversarial attacks against Windows PE malware detection: A survey of the state-of-the-art. Computers & Security, 128:103134. Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., and Zhang, G. (2018). Learning under concept drift: A

48

review. IEEE Transactions on Knowledge and Data Engineering, 31(12):2346–2363. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations. Mao, S., Dewan, S., and Ho, Y.-J. (2023). Personalized ranking at a mobile app distribution platform. Information Systems Research, 34(3):811–827. McIntosh, T., Susnjak, T., Liu, T., Xu, D., Watters, P., Liu, D., Hao, Y., Ng, A., and Halgamuge, M. (2024). Ransomware reloaded: Re-examining its trend, research and mitigation in the era of data exfiltration. ACM Computing Surveys, 57(1):1–40. Mead, R. (1965). A generalised logit-normal distribution. Biometrics, 21(3):721–732. Melville, N., Kraemer, K., and Gurbaxani, V. (2004). Information technology and organizational performance: An integrative model of IT business value. MIS Quarterly, 28(2):283–322. Microsoft (2025). PE and COFF format. Accessed: 2025-02-21. Mohaisen, A., Alrawi, O., and Mohaisen, M. (2015). Amal: high-fidelity, behavior-based automated malware analysis and classification. Computers & Security, 52:251–266. Moody, G. D., Siponen, M., and Pahnila, S. (2018). Toward a unified model of information security policy compliance. MIS Quarterly, 42(1):285–311. Musgrave, J., Campan, A., Messay-Kebede, T., and Kapp, D. (2024). Empirical network structure of malicious programs. Advances in Artificial Intelligence and Machine Learning, 4(1):1959– 1976. Nisi, D., Graziano, M., Fratantonio, Y., and Balzarotti, D. (2021). Lost in the loader: The many faces of the Windows PE file format. In Proceedings of the 24th International Symposium on Research in Attacks, Intrusions and Defenses, pages 177–192. Padmanabhan, B., Fang, X., Sahoo, N., and Burton-Jones, A. (2022). Machine learning in information systems research. MIS Quarterly, 46(1):iii–xix. Peng, H., Yu, Z., Zhao, D., Ding, Z., Yang, J., Zhang, B., Han, J., Zhang, X., Ji, S., and Zhong, M. (2025). Evading control flow graph based GNN malware detectors via active opcode insertion method with maliciousness preserving. Scientific Reports, 15(1):9174. Pietrek, M. (2002). An in-depth look into the win32 portable executable file format, part 2. MSDN Magazine, March. Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro, B., and Nicholas, C. K. (2018). Malware detection by eating a whole exe. In Workshops at the thirty-second AAAI conference on artificial intelligence. Rai, A. (2017). Editor’s comments: Diversity of design science research. MIS Quarterly, 41(1):iii– xviii. Reps, T. (1998). Program analysis via graph reachability. Information and Software Technology, 40(11-12):701–726. Rezende, D. J., Mohamed, S., and Wierstra, D. (2014). Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, pages 1278–1286. Ryder, B. G. (1979). Constructing the call graph of a program. IEEE Transactions on Software Engineering, pages 216–226. Saha, S., Wang, W., Kaya, Y., Feizi, S., and Dumitras, T. (2024). DRSM: De-randomized smoothing on malware classifier providing certified robustness. In International Conference on Learning Representations.

49

Samtani, S., Chai, Y., and Chen, H. (2022). Linking exploits from the dark web to known vulnerabilities for proactive cyber threat intelligence: An attention-based deep structured semantic model. MIS Quarterly, 46(2):911–946. Suciu, O., Coull, S. E., and Johns, J. (2019). Exploring adversarial examples in malware detection. In 2019 IEEE Security and Privacy Workshops (SPW), pages 8–14. IEEE. Sung, A., Xu, J., Chavez, P., and Mukkamala, S. (2004). Static analyzer of vicious executables (SAVE). In Proceedings of the 20th Annual Computer Security Applications Conference, pages 326–334. Temizkan, O., Park, S., and Saydam, C. (2017). Software diversity for improved network security: Optimal distribution of software-based shared vulnerabilities. Information Systems Research, 28(4):828–849. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (2018). Graph attention networks. In International Conference on Learning Representations. Wang, Y., Ge, Y., Li, Z., Li, L., and Chen, R. (2024). M3rec: A context-aware offline metalevel model-based reinforcement learning approach for cold-start recommendation. ACM Transactions on Information Systems, 42(6):1–27. Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Yu, P. S. (2020). A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4–24. Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2019). How powerful are graph neural networks? In International Conference on Learning Representations. Yan, J., Qi, Y., and Rao, Q. (2018). Detecting malware with an ensemble method based on deep neural network. Security and Communication Networks, 2018(1):7247095. Yan, J., Yan, G., and Jin, D. (2019). Classifying malware represented as control flow graphs using deep graph convolutional neural network. In 2019 49th annual IEEE/IFIP International Conference on Dependable Systems and Networks, pages 52–63. IEEE. Ye, Y., Li, T., Adjeroh, D., and Iyengar, S. S. (2017). A survey on malware detection using data mining techniques. ACM Computing Surveys, 50(3):1–40. Ye, Y., Wang, D., Li, T., and Ye, D. (2007). IMDS: Intelligent malware detection system. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1043–1047. Yu, W., Zhang, N., Fu, X., and Zhao, W. (2009). Self-disciplinary worms and countermeasures: Modeling and analysis. IEEE Transactions on Parallel and Distributed Systems, 21(10):1501–1514. Zhang, M., Cui, Z., Neumann, M., and Chen, Y. (2018). An end-to-end deep learning architecture for graph classification. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, volume 32. Zheng, Z., Fader, P., and Padmanabhan, B. (2012). From business intelligence to competitive intelligence: Inferring competitive measures using augmented site-centric data. Information Systems Research, 23(3-part-1):698–720. Zheng, Z. and Pavlou, P. A. (2010). Toward a causal interpretation from observational data: A new bayesian networks method for structural models with latent variables. Information Systems Research, 21(2):365–391.

50

APPENDICES Appendix A. Complexity Analysis Details The time complexity of MalGuard mainly involves three components: identifying operational roles, extracting operational subgraphs, and learning program graph representations for malware detection. Let |R| denote the number of software instances. For each software instance, let F denote the average number of functions, and mCG denote the average number of edges in a CG. For each CFG, let n and m denote the average number of nodes and edges, respectively. Let S denote the average number of extracted operational subgraphs per CFG, and let m e denote the average number of edges in the coarsened graph. Let K denote the number of latent operational roles, d denote the hidden dimension, and L denote the number of message passing layers. • Identifying operational roles. In Stage 1, MalGuard trains the operational role identification module over all CFGs. For each CFG, the inference model requires O(md + ndK) time to compute the latent role distribution. The reconstruction objective evaluates node-pair relationships in the CFG, requiring O(n2 K) time. After identifying latent operational roles, MalGuard extracts operational subgraphs by thresholding node-role assignments and finding connected components for each role, which requires O(K(n + m)) time for each CFG. Since operational subgraph extraction is performed only once and is much smaller than the cost of identifying latent operational roles, it can be ignored. Therefore, the per-CFG complexity of Stage 1 is O(md + ndK + n2 K). Across all |R|F CFGs, the per-epoch training complexity of Stage 1 is O (|R|F (md + ndK + n2 K)). • Learning program graph representations. In Stage 2, MalGuard learns program representations for malware detection. For intra-function modeling, the coarsening layer constructs the coarsened graph structure and computes super-node attributes with a cost of O(nd2 ) for each CFG. The attention-based message passing layers over the coarsened graph require A1

O(L(Sd2 + md)) e time. The gated pooling layer aggregates super-node representations into a CFG-level representation, with a cost of O(Sd). This term is lower order relative to the cost of other layers and is therefore omitted from the dominant complexity expression. Since each software instance contains F CFGs, the total intra-function modeling cost for each software instance is O(F (nd2 + L(Sd2 + md))). e For inter-function modeling, message passing over the CG requires O(L(F d2 + mCG d)) time, and the gated pooling layer introduces only a lower-order cost of O(F d). Therefore, the Stage 2 complexity across |R| software instances is O (|R| [F nd2 + F L(Sd2 + md) e + L(F d2 + mCG d)]) = O (|R| [F nd2 + F L(Sd2 + md) e + LmCG d]) , since S ≥ 1. Summing these components, the overall time complexity of MalGuard per epoch is O(|R|[F (md+ ndK + n2 K) + F nd2 + F L(Sd2 + md) e + LmCG d]). Since K, d, and L are fixed hyperparameters, the complexity is mainly determined by the number and size of program graphs.

A2

Appendix B. Detailed Experiment Settings We provide the detailed implementation settings and baseline configurations used in our experiments. All models were implemented in Python 3.8, the implementation details are summarized in Table 5. Specifically, MalConv, NonNeg, MAGIC, DeepCall, and Mal2GCN were strictly re-implemented following their papers. For DRSM and MalGraph, we utilized the official implementations released by their authors. The detailed hyperparameter settings of the baselines are summarized in Table 6. Table 5

Summary of model implementations.

Operation

Implementation Details

Official Link

Coding Environment

Python with PyTorch library

https://pytorch.org/

MalConv (Raff et al., 2018), NonNeg (Fleshman et al., 2019)

Re-implemented in PyTorch following their original model architectures

https://pytorch.org/

DRSM (Saha et al., 2024)

Implemented using the official source code

https://github.com/ShoumikSaha/DRSM

GCN (Kipf and Welling, 2017), GAT (Veličković et al., 2018), GIN (Xu et al., 2019)

Implemented with PyG library

https://pyg.org/

MAGIC (Yan et al., 2019), DeepCall (Chen et al., 2023), Mal2GCN (Kargarnovin et al., 2024)

Re-implemented in PyG following their original model architectures

https://pyg.org/

MalGraph (Ling et al., 2022)

Implemented using the official source code

https://github.com/ryderling/MalGraph

Table 6

Summary of baseline hyperparameter configurations.

Baseline

Hyperparameter Configurations

MalConv (Raff et al., 2018)

Maximum input length D = 2 MB; byte embedding dimension = 8; number of convolutional filters = 128; filter width = 500 bytes; stride = 500; temporal max pooling.

NonNeg (Fleshman et al., 2019)

MalConv backbone; maximum input length D = 2 MB; final-layer weights constrained to be nonnegative; weight decay = 10−3 .

DRSM (Saha et al., 2024)

MalConv backbone; maximum input length D = 2 MB; number of ablation windows n = 8; final prediction obtained by majority voting.

GCN (Kipf and Welling, 2017)

Two-layer GCN for CFG encoding; two-layer GCN for CG encoding; hidden dimension = 128; global mean pooling.

GAT (Veličković et al., 2018)

Two-layer GAT for CFG encoding; two-layer GAT for CG encoding; hidden dimension = 128; number of attention heads = 4; global mean pooling.

GIN (Xu et al., 2019)

Two-layer GIN for CFG encoding; two-layer GIN for CG encoding; each GIN layer uses a two-layer MLP with ReLU activation; hidden dimension = 128; global mean pooling.

MAGIC (Yan et al., 2019)

Two graph convolution layers on the CFG with hidden dimension = 128; two 1D convolution layers with kernel size = 5 and padding = 2; global mean pooling.

DeepCall (Chen et al., 2023)

Two graph convolution layers on the CG with hidden dimension = 128; two 1D convolution layers with kernel size = 5 and padding = 2.

Mal2GCN (Kargarnovin et al., 2024)

Two GCN layers with hidden dimension = 128; global mean pooling; final-layer weights constrained to be non-negative.

MalGraph (Ling et al., 2022)

Two-layer GraphSAGE for CFG encoding; two-layer GraphSAGE for CG encoding; hidden dimension = 128; dropout rate = 0.2; global mean pooling.

A3

Appendix C. Extended Experiments on Evasive Techniques To further evaluate MalGuard under more challenging conditions, we modify the original malware instances in our test set by injecting evasive behaviors. Specifically, we employed six different evasion techniques to inject evasive behaviors: FGSM Append (Kreuk et al., 2018), Slack Append (Suciu et al., 2019), DOS Extension (Demetrio et al., 2021), DOS Partial Modification (Demetrio et al., 2019), DOS Full Modification (Demetrio et al., 2021), and Header Field Modification (Nisi et al., 2021). Each technique was applied to the malware instances in the test set, after which models were evaluated on the evasive datasets. Results show that MalGuard achieves substantial gains over byte-based methods: for example, compared with MalConv, it improves AUC by 25.07% and AUPRC by 38.99%, underscoring its strong capability to defend against malware employing evasive behaviors. Table 7

Comparison of AUC and AUPRC under different evasion techniques. Values are reported as AUC/AUPRC. Evasive Technique

Category

Method

FGSM Append

Slack Append

DOS Extension

DOS Full Modification

DOS Partial Modification

Header Field Modification

Byte-Based

MalConv NonNeg DRSM

67.76/56.51 72.58/72.31 77.05/73.45

79.37/67.85 71.64/71.49 80.93/79.92

67.85/55.55 65.18/56.77 79.92/54.88

72.58/69.93 78.24/58.06 80.46/68.24

74.16/68.02 86.93/79.84 82.74/74.49

88.76/77.17 81.64/72.58 90.32/76.63

GCN GAT GIN MAGIC DeepCall Mal2GCN MalGraph

89.28/88.21 91.65/88.04 89.42/85.21 92.06/88.66 93.72/92.45 82.81/81.36 93.95/91.13

90.09/85.86 91.34/88.98 91.69/87.20 91.12/88.02 92.52/91.97 81.01/80.65 85.14/85.45

89.25/86.77 91.11/85.05 89.42/85.21 87.36/84.18 91.94/89.27 82.48/83.04 85.28/86.31

89.28/88.21 90.76/86.51 89.46/84.38 91.19/88.33 92.20/89.27 83.09/82.23 91.29/86.59

84.95/86.02 93.83/88.65 89.90/86.13 91.12/88.02 90.93/88.34 81.01/80.65 85.73/85.64

90.09/85.86 91.34/88.98 91.69/87.20 89.49/85.51 91.94/89.27 82.21/85.37 85.28/86.31

95.60/92.54

93.83/92.49

93.15/90.76

93.03/91.13

93.89/90.63

93.03/91.13

Graph-Based

MalGuard

A4

Appendix D. Notations Table 8 summarizes the main notations used throughout the paper. Unless otherwise stated, G = (V, A, X) denotes a single CFG under the simplified notation introduced in Section 3.2. Table 8

Summary of notations.

Notation

Description

R = {ri } yi , ŷi H = (GCG , GCFG ) GCG = (VCG , ECG ) vm ∈ VCG GCFG = {G1 , . . . , G|VCG | } Gm = (Vm , Am , Xm ) G = (V, A, X) K zi ∈ RK µi , σ i OS = {OS1 , . . . , OSS } B ∈ {0, 1}|V |×S Asuper , X super (l) gs (l) αst β f (l) hm γ hCG

Set of software instances Ground-truth label and predicted malware probability of ri Hierarchical program graph of a software instance CG of a software instance Function node corresponding to the m-th function Set of CFGs associated with function nodes in the CG CFG corresponding to function node vm Simplified notation for a single CFG when no ambiguity arises Number of latent operational roles Latent operational role vector of basic block node vi Mean and standard deviation vectors of the logit-normal posterior for zi Set of operational subgraphs extracted from a CFG Assignment matrix between basic block nodes and operational subgraphs Adjacency matrix and node attribute matrix of the coarsened graph Representation of OSs at layer l Attention weight from OSt to OSs at layer l Fusion gate over operational subgraphs in intra-function modeling CFG-level representation obtained from intra-function modeling Representation of function node vm at layer l Fusion gate over function nodes in inter-function modeling Final software representation obtained from inter-function modeling

A5