arXiv CSopen access

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs Yacong Gu

Tsinghua University; Tsinghua University-QI-ANXIN Group JCN Beijing, China [email protected]

arXiv:2606.30373v1 [cs.CR] 29 Jun 2026

Yingyuan Pu

QI-ANXIN Technology Research Institute Beijing, China [email protected]

Wenjie Zhu

QI-ANXIN Technology Research Institute Beijing, China [email protected]

Lingyun Ying

QI-ANXIN Technology Research Institute Beijing, China [email protected]

Xiaoxue Huang

QI-ANXIN Technology Research Institute Beijing, China [email protected]

Zidong Zhang

QI-ANXIN Technology Research Institute Beijing, China [email protected]

Jiawei Zhou

QI-ANXIN Technology Research Institute Beijing, China [email protected]

Donghong Sun

Tsinghua University Beijing, China [email protected]

Abstract

CCS Concepts

AI-powered Applications (AI-Apps), hosted on platforms such as Hugging Face, are democratizing access to pre-trained models through online inference and fine-tuning services. While lowering AI adoption barriers, these platforms introduce an unexplored attack surface, as AI-Apps are often developed by untrusted parties with weak isolation and misconfigured security settings. In this paper, we present the first systematic security analysis of AIApps across three leading platforms. To structure our investigation, we map the AI-App lifecycle to established risk taxonomies (e.g., OWASP), identifying five threat categories and ten attack vectors ranging from generic web flaws to high-impact architectural issues. Our analysis reveals critical failures including broken access control, insecure resource reuse, insufficient input validation, and sensitive data exposure. Notably, we uncover three novel architectural vulnerabilities inherent to platform design and demonstrate how traditional issues (e.g., world-readable logs) are uniquely amplified in this ecosystem. To assess real-world impact, we develop an analysis framework Insightor and apply it to over 970,000 public AI-Apps. Alarmingly, we find thousands of apps leaking credentials, hundreds containing input injection vulnerabilities that allow arbitrary code execution, and tens harboring embedded backdoors—indicating active exploitation. We have responsibly disclosed all findings to the affected platforms and developers.

• Security and privacy;

Haixin Duan

Tsinghua University Beijing, China [email protected]

Keywords AI-App Security, Model Hub Security, AI Supply Chain Security

1

Introduction

With the rapid development of artificial intelligence, numerous PreTrained Models (PTMs) have emerged. Platforms such as Hugging Face now offer AI-powered Applications (AI-Apps, called Space on Hugging Face) that enable streamlined model access. Unlike conventional workflows requiring local deployment and complex environment setup to run models, AI-Apps are cloud services that integrate models with runtime environments, inference APIs, and web interfaces. This approach simplifies deployment for developers while enabling end-users to perform inference and fine-tuning through browsers or APIs without specialized hardware. AI-Apps have gained remarkable traction. As of December 2025, Hugging Face hosts over 930,000 AI-Apps, while platforms like Replicate serve major companies including Meta, OpenAI, and Google. For example, Meta’s Llama AI-Apps have processed over 600 million inference requests on Replicate [26]. Unfortunately, AI-Apps introduce new attack surfaces as they can be created and accessed by less-trusted users. Improper configuration and insufficient isolation may allow adversaries to inject malicious payloads into victim AI-Apps or extract sensitive data from developers and users. Recent incidents demonstrate these risks: an official Hugging Face model conversion AI-App contained a vulnerability that allowed attackers to impersonate the conversion bot and manipulate arbitrary AI-Apps [60]. Additionally, security flaws of AI-App Platforms (AAPs) pose significant threats. Replicate once mistakenly assigned shared networks to multiple AI-Apps, enabling attackers to manipulate inference or steal private models

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. ACM CCS ’26, Hague, Netherlands © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX 1

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Gu et al.

through malicious AI-Apps [48]. Moreover, misuse and misconfiguration of secrets (e.g., access tokens for third-party services) on AAPs also introduce the risk of sensitive data exposure. For instance, in 2024, Hugging Face detected unauthorized platform access that compromised multiple users’ secrets [38]. In this paper, we conduct the first systematic study towards understanding security threats in AI-Apps, focusing on three key stakeholders: platforms, developers, and users. We conduct a comprehensive analysis of AI-App implementations on three mainstream AAPs (Hugging Face [23], Replicate [32], and ModelScope [27]), from AI-App creation, user interface, authentication, to resource isolation and sharing. We find that AI-Apps have some important characteristics that distinguish them from common web applications: they are essentially web servers created by developers and integrated into AAP web pages through iframe embedding. We discover that there are multiple design flaws in AAPs that could potentially cause security risks, enabling attackers to perform various attacks such as code injection and data leakage. To identify these issues systematically, we map the AI-App lifecycle to established web [29] [71] and supply-chain [64] risk taxonomies, addressing their dual nature as web applications and composite systems with complex dependencies and cascading trust, prioritizing high-impact and practically exploitable vectors. Guided by this methodology, we investigate five types of threats and identify ten potential attack vectors in current AAPs. Notably, these include three previously unreported, architecture-induced vectors arising directly from platform design choices, as well as known vulnerabilities that manifest with AI-App-specific amplification. More specifically, flawed access control represents a critical vulnerability in multiple AAPs. For example, Hugging Face uses the stateless JWT [34] protocol for AI-App authentication, but fails to invalidate tokens when resource ownership changes, enabling attackers to pre-generate tokens and achieve unauthorized access to victim developers’ AI-Apps (Ghost Token Attack). Additionally, several AAPs misconfigure AI-App runtime logs as world-readable, exposing sensitive data from both developers and users (Log Exfiltration Attack). Another critical issue involves improper resource reuse, which creates supply chain security risks. When generating access domains for AI-Apps, Hugging Face replaces slashes (/) with hyphens (-) while allowing hyphens in usernames, causing domain name conflicts between different users’ AI-Apps (e.g., user/app-name and user-app/name both map to user-app-name subdomain). This enables attackers to conduct malicious code injection against third-party websites embedding AI-Apps (Identifier Reuse Attack). Similarly, insufficient input validation affects many AI-Apps that fail to sanitize user input effectively, allowing attackers to execute arbitrary commands through carefully crafted inputs (Input Injection Attack). Furthermore, sensitive data leakage occurs through multiple vectors beyond hard-coded secrets in source code. AI-App runtime logs leak substantial sensitive information, including developer-configured secrets and user-submitted data such as prompts (Runtime Log Leakage). From our analysis, we find that all three mainstream AAPs exhibit significant security vulnerabilities. To evaluate the potential impact of the proposed attacks, we design and implement an analysis framework Insightor to perform a large-scale measurement

across the three AAPs. Insightor collects the source code, container images, and metadata of 972,546 public AI-Apps from the three platforms, covering data from September 2024 to December 2025. Then it utilizes pre-defined rules to detect potential vulnerabilities, such as ghost token and identifier reuse. Insightor further uses data flow analysis to identify input injection vulnerabilities and data leakage threats. Our measurement demonstrates that security risks in AI-Apps are not merely isolated coding errors but are often amplified by platform-mediated deployment and reuse mechanisms. Insightor detects widespread vulnerabilities ranging from architectural flaws to active malicious exploitation. Notably, we identify a critical disconnect in platform security: Insightor flagged 936 AI-Apps that followed official secret-handling guidelines yet potentially leaked credentials through misconfigured world-readable logs, with manual sampling confirming many actual leaks. Beyond accidental leakage, we uncover severe injection vectors, with Insightor identifying 1,442 potential input injection vulnerabilities and 139,475 apps running outdated SDKs (e.g., Gradio) with known RCE flaws. Furthermore, we find evidence of active exploitation, detecting 27 AI-Apps with embedded backdoors—some persisting for over a year and spreading through platform cloning features. We evaluated Insightor’s precision and recall on representative samples, confirming it effectively serves as a high-efficiency filter that reduces the search space from 972,546 AI-Apps to manageable candidate sets for manual verification. We have responsibly disclosed identified vulnerabilities to the relevant stakeholders and have received positive feedback, including $2,369 in bug bounties from Hugging Face. In summary, the major contributions of this work include:

• Systematic Security Analysis. We present the first comprehensive security study of the AI-App ecosystem. We characterize securitycritical mechanisms—including domain-based isolation, stateless token authentication, shared runtime logs, and code propagation—that create a novel attack surface affecting all stakeholders. • Novel Attack Vectors. We identify ten attack vectors, including three previously unreported, architecture-induced vectors (i.e., Ghost Token, Authentication Bypass, Identifier Reuse) that arise from platform design choices rather than generic web flaws; and we show how known issues manifest with AI-App-specific amplification, e.g., logs leaking prompts/secrets and code propagation through duplication. • Large-Scale Measurement. We design Insightor, a dataflow analysis framework that automatically scans 972,546 AI-Apps, uncovering widespread potential vulnerabilities including 1,442 input injection flaws, 936 credential leaks despite following official guidelines, 27 backdoors, and 139,475 apps with known RCE flaws. Responsible disclosure yielded positive feedback and $2,369 in bug bounties.

Our research artifacts, including the source code of Insightor and all findings, are available at https://anonymous.4open.science/ r/AI-App-Demo-FDDE. 2

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

AAP Assigned Domain

New AI-App Source Code

Container Image

Duplicate From Existing AI-App

End User Third-party Website

usernamespacename.hf.space Web Server

AI-App Container Image

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Example.com

Web Server

Load

AAP Main Website AI-App

AI-App

Container

Container

Browser

huggingface.co/spaces/username/spacename

API Endpoint

API

K8s Cluster

Figure 1: The overall architecture of AI-App.

By default, the GUI is rendered on the AAP website, although some platforms also support embedding it into third-party websites [54]. API support is offered by most AI-Apps, enabling users to integrate AI-Apps into other services. For example, a text classification AIApp can be invoked as part of a larger AI pipeline via API calls. Secrets are sensitive data required during the execution of an AIApp, such as database credentials or access tokens for third-party services. On most AAPs, the recommended practice is to define secrets as key-value pairs through the platform’s web portal and refer to them by key within the AI-App code, thereby avoiding the exposure of plaintext credentials [25]. Model Training. AI-Apps that support model training are an important capability on AAPs. Platforms such as Hugging Face offer official training services (i.e., the AI-App AutoTrain [20]), while others provide a framework for developers to build custom training AI-Apps (e.g., Replicate’s fine-tune service [53]). Unlike inference applications, training AI-Apps write newly generated models or applications to the user’s account upon completion, using platformmanaged user credentials to authorize the operation. Billing. AAPs typically adopt two billing models: charging the developer or the user. Whichever person pays the bill, AAPs charge fees based on the time and usage of the computing resource. To promote sharing, some platforms (e.g., Replicate) implement pricing incentives such as excluding cold-boot time from billing for public AI-Apps [33], since AI-Apps commonly contain tens of gigabytes of model weights and may incur considerable cold-boot times.

2 Background 2.1 AI-Apps: A Brief Introduction AI-Apps are cloud-based services that encapsulate PTMs with runtime environments and inference APIs. They streamline deployment by eliminating dedicated hardware requirements (e.g., GPUs) and complex configuration. Developers transform models into scalable services using platform infrastructure with automatic scaling and parallel inference, while end users access them via web browsers or APIs with minimal setup. Major platforms demonstrate significant adoption: Hugging Face hosts over 930k public AI-Apps, Replicate and ModelScope host 25k and 8k respectively, with participation from companies including Meta, OpenAI, and Google. AI-Apps Lifecycle. Figure 1 illustrates the typical lifecycle of AIApp. Developers first create an AI-App on an AAP with a globally unique identifier, choosing public or private visibility. In addition to writing code from scratch, developers can start from official templates based on popular SDKs (e.g., Gradio [18]) or duplicate existing AI-App source code, similar to the fork operation on GitHub. Developers then implement the AI-App following the development specifications provided by the AAP. Platforms support two main deployment approaches: some (e.g., Hugging Face, ModelScope) accept source code submissions and automatically build and publish the AI-App, while others (e.g., Replicate) require developers to locally build container images conforming to platform specifications before uploading for deployment. Upon submission, the AAP allocates resources and launches the AI-App, offering fixed and elastic allocation modes. Users interact through browser-based web interfaces and API endpoints for service integration. Finally, AAPs provide management functions for modification, updates, deletion, and ownership transfer.

2.2

3

AI-App Implementation Demystified

Understanding AI-App implementation details is crucial for identifying security vulnerabilities. Since most AAPs provide insufficient documentation, we employ three complementary methods: reviewing AAP documentation, analyzing open-source components (e.g., Hugging Face’s AutoTrain [20], Replicate’s Cog [4]), and conducting black-box testing. For black-box testing, we design a chatbot AI-App that executes user inputs as shell commands and deploy it privately across AAPs. We examine runtime environments (system privileges, environment variables, network isolation), capture network traffic via tcpdump, and create secrets to observe their processing and isolation. Table 1 summarizes the key features of examined AAPs.

Core Concepts of AI-Apps

This section introduces fundamental concepts central to the design, deployment, and utilization of AI-Apps on AAPs. Runtime refers to the execution environment of an AI-App, which can be either container-based or non-container. The former packages the AI-App into a container image (e.g., Docker) managed by the platform’s orchestration system (e.g., Kubernetes [24]), while the latter is typically utilized for static websites and is limited to applications that do not require GPU resources. Most platforms support auto-scaling for container runtimes, enabling developers to configure maximum instances that adjust automatically based on the volume of requests. Graphical User Interface (GUI) offers a browser-based interface for interacting with an AI-App. It is essentially a developerimplemented web page that takes user input and presents the output.

3.1

AI-App Creation

Our investigation identifies three distinct approaches for AI-App creation: I. Code Repository-Based Development. Platforms like Hugging Face and ModelScope require developers to commit source code to dedicated repositories. Upon code updates, these platforms 3

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Gu et al.

automatically trigger build processes that generate container images for deployment. However, these platforms differ in their security constraints: Hugging Face allows unrestricted base container images, meaning developers can publish AI-Apps with arbitrary—potentially malicious—code, while ModelScope restricts developers to predefined official base images. II. Developer Builds and Pushes AI-Apps. The Replicate platform provides Cog [4], a Docker-based development tool for building AI-Apps. Developers specify dependencies in cog.yaml, implement inference logic in predict.py, and define training procedures in train.py. After building the container image locally, developers push it to the platform for deployment. III. Duplicated from Existing AI-Apps. All three AAPs support AI-App duplication through different mechanisms. Hugging Face and ModelScope provide explicit duplication features, allowing developers to manually fork existing AI-Apps. Replicate implements implicit duplication: when developers use AI-Apps with training capabilities, the platform automatically creates new instances under their accounts, replicating the original source code. Billing. The three AAPs adopt different billing strategies. ModelScope employs a developer-paid model, where developers bear usage costs. Replicate follows a user-paid model, charging users directly for AI-App usage. Hugging Face supports both approaches: AI-Apps are billed to developers by default, but the inference-api feature enables user authorization and billing [37]. The user-paid model introduces security concerns, as users execute potentially untrusted external code on their computing resources, making them vulnerable to cryptojacking attacks [57] where attackers exploit victims’ processing power for cryptocurrency mining.

3.2

Input

predict.py

Form

JSON

Python

Cog

HTTP

Image* file Enter A URL, paste a file, or drag a file over.

scale float

Pre-process inputs

Reset

Run

Figure 2: Web page generation and input processing on Replicate. Hugging Face

AI-App iframe

https://huggingface.co/spaces/user/space-name

https://user-space-name.hf.space

Login

Session cookie stored in browser for authentication

AI-App Web Server JWT Token

Dev Web Server

Cross-origin policy violation

Figure 3: Authentication workflow of AI-Apps on Hugging Face.

offer API support. For local execution, users can pull container images and run them locally via Docker.

AI-App User Interface

AI-Apps are deployed as web servers providing inference and training services. Platform-specific configurations include: Hugging Face uses the app_port keyword in README.md files, ModelScope pre-configures ports in base container images, and Replicate’s Cog tool injects HTTP servers on port 5000. All three AAPs support browser-based GUI access through two distinct modes: Freestyle Website Mode. Developers maintain full control over the GUI, with content served directly from the AI-App’s web server. Hugging Face and ModelScope assign unique subdomains to each AI-App (e.g., Facebook’s facebook/sapiens-pose generates URLapp 1 ), while simultaneously creating platform pages (URLplatform 2 ) that embed the AI-App via iframes. Similarly, Hugging Face supports embedding AI-Apps in third-party websites via an iframe [54]. However, an improperly configured iframe can pose security risks, such as clickjacking and phishing. AAP-Formatted Web Page. Developers specify input parameters and output formats; AAPs automatically generate web interfaces (step ❶ in Figure 2). For instance, Replicate parses user inputs (step ❷), preprocessing them (e.g., converting files to URLs) and invoking AI-Apps with processed parameters (step ❸). Beyond web interfaces, all three AAPs support API access and local execution. Replicate provides APIs for all AI-Apps, while most Gradio-based [18] AI-Apps on Hugging Face and ModelScope also 1 https://facebook-sapiens-pose.hf.space/

2 https://huggingface.co/spaces/facebook/sapiens-pose

3.3

Runtime Isolation

3.4

AI-App Authentication

Inner-App Isolation. Our experiments reveal that all three AAPs lack proper isolation between user requests within the same AI-App instance. Different users’ requests may share execution contexts, logging systems, and containers. Furthermore, AAPs do not automatically clean temporary files between sessions, relying entirely on developers for cleanup implementation. This design enables persistent attacks where malicious users can compromise shared environments and target subsequent users. Inter-App Isolation. All evaluated AAPs utilize Kubernetes [24] for container orchestration. We assess their implementations against known security risks, including network isolation failures, service account token exposure, and insecure volume mounts. Our analysis does not find such vulnerabilities on any of the three platforms. Network Accessibility. All three AAPs permit AI-Apps to establish outbound connections to arbitrary destinations via standard web ports (80, 443), with Hugging Face additionally supporting port 8080 [40]. This unrestricted external access enables malicious AI-Apps to transmit users’ sensitive data or establish reverse shells. This section analyzes authentication mechanisms for AI-App access. Web Page Access. AAPs use session-based authentication for homepage access (URLplatform ), storing session keys in cookies upon login (step ❶ ❷ in Figure 3). However, accessing AI-Apps directly (URLapp ) presents challenges. Hugging Face and ModelScope embed URLapp as iframes within URLplatform , running on separate domains to comply with the same-origin policies, preventing direct cookie inheritance. As a result, the AI-App cannot inherit the user’s authentication state from the URLplatform directly. Authentication remains essential for AI-Apps, as without it, an attacker could access a private AI-App simply by guessing or obtaining its URLapp . To address this issue, both platforms implement token-based authentication. Specifically, Hugging Face employs JWT tokens [34] (denoted as APIjwt 3 ), while ModelScope uses a 3 https://huggingface.co/api/spaces/{space_name}/jwt

4

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Table 1: Overview of AI-App implementations. 𝐷/𝑈 in the Billing column mean the billing strategies: 𝐷 - Developer-Paid, 𝑈 - User-Paid. AI-App Creation

AAPs

Full Control Hugging Face Replicate ModelScope

Duplication

✓ ✓

✓ ✓ ✓

GUI

Billing

𝐷, 𝑈 𝑈 𝐷

API

Full Control

Access Token

Embed External Website

✓

JWT

✓

✓

AAP-defined

✓

Log Access

Multi-user Container Sharing

Owner

Any User

✓ ✓ ✓

✓ ✓ ✓

✓ ✓

✓∗ ✓ ✓∗

Secret Exposure in Logs ✓† ✓ ✓

∗ AI-Apps built on Gradio can be accessed through API.

† Hugging Face only masks its own platform tokens (strings starting with hf_), leaving other secrets unprotected.

custom algorithm4 . When users visit URLplatform , the platform generates an access token (step ❸) appended to URLapp as an HTTP query parameter for user authentication (step ❹). Our analysis reveals that Hugging Face generates JWT tokens for all private AI-Apps, as well as those with Dev Mode enabled. Dev Mode Access (Hugging Face). Hugging Face’s Dev Mode [39] enables remote development through SSH server installation within containers, supporting VSCode or SSH connections. The browserbased VSCode interface is embedded using an iframe. Like iframebased AI-Apps, it also uses JWT token authentication. Dev Mode grants full AI-App control, making unauthorized access particularly risky, as it enables attackers to perform arbitrary code manipulation, backdoor implantation, and secret extraction. Runtime Log Access. Hugging Face restricts runtime log access buttons to AI-App developers, which is a sound design decision since logs may contain sensitive data from multiple users and should not be publicly accessible. However, despite this interface-level restriction, both Hugging Face and Replicate configure public AIApps’ logs to be world-readable, creating a significant risk of data leakage. We discuss this issue in detail in the following section.

3.5

AI-App Secrets

3.6

Model Training via AI-Apps

lies in Replicate’s AI-App replication process. The newly generated AI-App inherits the original’s code, receiving trained weights via COG_WEIGHTS environment variable. This enables code propagation attacks where malicious code transfers to newly created AI-Apps.

4 AI-App Threats 4.1 Threat Model We consider two deployment scenarios for AI-Apps: (1) public deployment accessible to global users and (2) private deployment within organizations with restricted access. We assume that AAPs are trustworthy providers and that underlying communication channels (e.g., HTTPS) remain secure. Scope and Threat Identification. This paper investigates security risks in the AI-App platform ecosystem. We focus on vulnerabilities arising from the shared infrastructure and integration logic, rather than the mathematical properties of the models. Specifically, our analysis covers: (1) Runtime Environments, covering container lifecycle, image building, and isolation boundaries; (2) Interaction Interfaces, including iframe embeddings, frontend wrappers, and API endpoints; and (3) Platform Services, such as token-based authentication, logging, and storage. Consequently, model-centric attacks such as Prompt Injection are out of scope, as they target model properties rather than the deployment infrastructure. To systematically identify threats within this scope, we structured our analysis by: first, enumerating security-critical interfaces (e.g., secret handling, token propagation); and second, prioritizing high-impact vectors (e.g., data theft, RCE). Finally, to ground our findings in established security practices, we map the identified attack vectors to standard risk frameworks, including the OWASP Top 10 for Web and LLMs [71] and supply-chain risk taxonomies [64]. A detailed mapping of these vectors is provided in Table 7 in Appendix. Adversaries. Under these assumptions, we identify two types of adversary: (1) Unethical AI-App developers may publish malicious applications to exploit users. These AI-Apps can steal sensitive data, conduct phishing attacks, or abuse computing resources for unauthorized activities (e.g., cryptocurrency mining). Attackers may embed malicious code in container images while maintaining clean source code on AAPs, or exploit distribution vulnerabilities to inject malicious applications into third-party websites. (2) Unethical AI-App users may exploit vulnerabilities in legitimate AI-Apps or AAP infrastructure to exfiltrate data or install backdoors. These attackers typically possess basic reverse engineering skills to extract source code, identify common vulnerabilities (e.g., unsanitized inputs), and analyze logs for secrets or user-submitted sensitive data such as prompts. Such attacks require minimal technical effort

AAPs employ two secret management approaches. Hugging Face and ModelScope inject secrets as environment variables. Replicate transmits secrets as plaintext in HTTP request bodies to AI-App web servers, retrieved via get_secret_value() [55]. Secrets in Logs. Exposing sensitive information in logs is an OWASP top-ten vulnerability [29]. While services like GitHub Actions implement automatic log scanning to mask registered secrets, none of the examined AAPs provide effective masking, leaving secret values exposed in logs (Table 1). Combined with the runtime log exposure risks identified in Section 4.2.1, this enables widespread sensitive information leakage. Hugging Face and Replicate provide model training services through AI-Apps, creating new models under users’ accounts—a process requiring write permissions. Hugging Face employs AutoTrain [20], an official training AI-App that users must first duplicate in their accounts. During training, AutoTrain requests OAuth write access to create and upload the trained model. Replicate requires developers to implement a train() function, which enables training functionality. Users initiate training via website or API, specifying a destination AI-App. Replicate creates it internally without exposing tokens to the original AI-App, mitigating direct token leakage. However, the primary security risk 4 https://modelscope.cn/api/v1/studios/token

5

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Gu et al.

Table 2: Overview of potential threats on each AI-App platform. The ✓ means vulnerable. HF - Hugging Face, MS - ModelScope. Threats

HF

Replicate

A1: Log Exfiltration A2: Ghost Token A3: Auth Bypass A4: Over-privileged iframe R1: Identifier Reuse R2: Code Poisoning V1: Input Injection L1: Runtime Log Leakage L2: Files Leakage P1: Cryptojacking

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓∗

✓

token (valid for 24 hours), and then immediately delete both resources. Due to JWT’s stateless nature, the token remains valid despite resource deletion. When victims later register the same username and (re)create the predicted AI-App, the attackers’ token retains access even if the victim AI-App is private. This attack relies on two factors: (1) accurate identifier prediction—enabled by username overlap between GitHub and Hugging Face (Section 6.1.2); (2) token validity maintenance—achieved through repeated resource creation/deletion cycles using free accounts at minimal cost.

MS

✓

✓

✓ ✓ ✓ ✓ ✓

✓ ✓

4.2.3 Authentication Bypass Attack (A3). We identify AI-Apps implementing flawed access control through hard-coded password verification across two scenarios. In the first scenario, developers embed password verification logic in publicly visible AI-Apps to restrict unauthorized access. For example, ardianfe/stable-audioprod on Replicate verifies whether the user-supplied key parameter matches a hard-coded string (line 4 in Listing 1), terminating execution upon failure. One likely reason for this approach is Replicate’s billing model, which eliminates cold-boot charges for public AIApps [33], while attempting to maintain access control through rudimentary authentication. The second scenario involves integrating admin panels into AI-Apps, where password entry grants access to AI-App management interfaces. While such panels may facilitate administration, proper authentication should rely on secrets rather than hard-coded passwords. Both scenarios expose critical security vulnerabilities: since passwords are hard-coded in source code within container images, attackers can extract and examine these images to trivially retrieve credentials and bypass authentication.

✓

∗ AI-Apps that employ user-pays models are vulnerable.

and cost, particularly in public AI-Apps where developers cover usage costs, making them both practical and scalable. Table 2 summarizes the investigated AAPs and their associated threats that we find.

4.2

Flawed Access Control

Access control defects are among the most common and dangerous security vulnerabilities. Our research reveals that all three AAPs suffer from improper access control design, which can lead to unauthorized access and data leakage. 4.2.1 Log Exfiltration Attack (A1). Runtime logs often contain sensitive data when developers inadvertently logging confidential information. The 2023 TravisCI incident illustrates this risk: 4.7 million exposed logs leaked over 73,000 credentials [13]. Our analysis reveals that Hugging Face and Replicate misconfigure public AI-Apps’ runtime logs as world-readable, making them vulnerable to log exfiltration attacks. Despite hiding the log access interface (i.e., a button on the AI-App’s webpage), Hugging Face’s APIjwt remains publicly accessible and can issue JWT tokens for any public AI-App, granting complete log access with the platform’s log API5 . This effectively allows any Hugging Face user to read the runtime logs of any public AI-App. Similarly, Replicate exhibits the same vulnerability but with limited scope/impact. Specifically, Replicate assigns each task (inference or model training) a random unique identifier consisting of 26 characters. Although possessing an identifier enables log retrieval, the platform lacks identifier enumeration methods. This limits the attack surface to logs explicitly shared by developers, as brute-force identifier guessing remains computationally infeasible. In contrast, ModelScope restricts log access to administrative users only, eliminating this attack vector.

predict ( # .... other parameters key : str = Input ( description = " key password ") ) - > dict : if key ! = " # p ' 7y9EqH4 > el ; VZ ' Ok * RK # Jt ,/ old ": raise ValueError ( " Invalid key password " )

1 def 2 3 4 5

Listing 1: Checking for hard-coded passwords in AI-App ardianfe/stable-audio-prod on Replicate.

4.2.4 Over-privileged iframe ( A4). Hugging Face and ModelScope embed AI-Apps using iframe elements with overly broad permissions. Our analysis reveals that Hugging Face grants each AI-App iframe up to 27 permissions by default6 , including access to camera and microphone. This over-permissioned design creates a privilege inheritance vulnerability. When users grant permissions to legitimate AI-Apps, subsequently loaded malicious applications inherit these permissions without additional user consent. Consequently, attackers can silently access sensitive resources—such as camera feeds or clipboard data—by bypassing browser security prompts.

4.2.2 Ghost Token Attack (A2). Hugging Face allows reusing deleted account names, creating identifier (i.e., user/spacename) collisions. Moreover, Hugging Face’s APIjwt is publicly accessible and allows users to generate JWT tokens for specified AI-Apps. Thus, combined with JWT’s stateless nature, previously issued tokens remain valid after resource ownership changes. We find Hugging Face vulnerable to this attack. Attackers can predict future AI-App identifiers (e.g., by targeting organizations active on GitHub but absent from Hugging Face). They register the predicted username, create a private AI-App, generate a JWT

4.3

Improper Resource Reuse

AI-Apps introduce resource reuse vulnerabilities that attackers exploit via identifier mapping and code inheritance to compromise applications and spread malicious content. 6 accelerometer, ambient-light-sensor, autoplay, battery, camera, clipboard-read, clipboard-write, display-capture, document-domain, encrypted-media, fullscreen, geolocation, gyroscope, layout-animations, legacy-image-formats, magnetometer, microphone, midi, oversized-images, payment, picture-in-picture, publickey-credentials-get, sync-xhr, usb, vr, wake-lock, xr-spatial-tracking

5 https://api.hf.space/v1/{space_name}/logs/run

6

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

This behavior creates opportunities for malicious exploitation. An unethical developer could publish an AI-App containing malicious code (e.g., backdoors) and wait for victims to copy or unknowingly trigger code duplication. Such attacks can result in severe consequences, as poisoned AI-Apps may leak sensitive data or provide unauthorized access to private infrastructure when deployed internally. We find that all three AAPs are vulnerable to this threat.

user/space-name Example.com user-space-name.hf.space user-space-name.hf.space

(a) Hugging Face automatically generates dedicated domain names for AI-Apps and supports their integration into third-party websites via iframe embedding.

4.4

Insufficient Input Validation

As web-based applications, AI-Apps face similar input injection risks to traditional web services.

user/space-name Example.com

4.4.1 Input Injection Attack (V1). All three AAPs are vulnerable to input injection attacks through two primary vectors. First, developers may implement insecure code that inadequately sanitizes user input, enabling malicious command injection. Second, AI-Apps may rely on vulnerable third-party SDKs that remain unpatched. For example, Gradio—a widely-used front-end library for AI-Apps— has suffered multiple code injection vulnerabilities [6][7][8][9][10], compromising dependent AI-Apps. Input injection poses serious threats to both developers and end users. Attackers can execute arbitrary shell commands to extract sensitive data (e.g., secrets in environment variables) or inject persistent malicious code that compromises subsequent users. Listing 2 demonstrates this vulnerability in an AI-App that directly concatenates user-supplied parameters into shell commands (line 7), enabling arbitrary command execution.

user-space-name.hf.space user-space-name.hf.space

user-space/name

(b) Attacker creates an AI-App with a domain name colliding with a deleted benign AI-App, resulting in victim websites unknowingly embedding malicious iframes. Figure 4: Overview of Identifier Reuse Attack (R1).

4.3.1 Identifier Reuse Attack (R1). Improper identifier reuse poses classic supply chain security risks [59], and AI-Apps are not exempt from this threat. We identify a domain takeover vulnerability on Hugging Face that enables silent malicious code injection through orphaned iframe links. Hugging Face generates subdomains for AI-Apps by replacing slashes with hyphens in their identifiers (e.g., user/space-name → user-space-name.hf.space). Since usernames can contain hyphens, distinct AI-Apps (user/space-name and user-space/name) may collide on the same subdomain. Hugging Face resolves conflicts by appending random suffixes to new subdomains. This creates an attack vector: attackers pre-register accounts (e.g., user-space) and monitor for target AI-App deletions. When user/space-name is deleted, iframes embedded in third-party websites become dangling resources pointing to unassigned subdomains. Attackers then create user-space/name, thereby reclaiming the original subdomain user-space-name.hf.space. Notably, Hugging Face does not append a random suffix in this case, as no conflicting AI-Apps exist on the platform at this point. Consequently, previously benign iframes now load malicious content from the trusted Hugging Face domain (Figure 4). This attack is difficult to detect, as iframes appear legitimate from official Hugging Face domains without requiring code modifications by website owners. Takeovers can persist undetected, enabling sustained phishing, credential harvesting, and targeted attacks.

1 def 2 3 4 5 6 7 8

predict ( # other parameters speaker : str = Input ( description = " Original

speaker audio url ... " ) User Input ) - > Any : ..... os . system ( f " ffmpeg -i { speaker } - af { filter } { trim_silence } -y { speaker_wav } " )

Listing 2: Code injection vulnerability codehappynice/voicegenerator on Replicate.

4.5

in

the

AI-App

Sensitive Data Leakage

We find all three AAPs suffer from serious data exposure issues, affecting numerous AI-Apps across platforms.

4.3.2 AI-App Poisoning Attack (R2). AI-Apps can reuse code through two primary mechanisms identified in Section 3.1. First, developers may explicitly duplicate existing AI-Apps. Second, our investigation reveals that Replicate automatically generates new AI-Apps inheriting source code when users perform model training, without explicitly notifying developers of this code replication. 7

4.5.1 Runtime Log Leakage (L1). Writing sensitive data into log files is an OWASP Top 10 security vulnerability [29]. In AI-App ecosystem, this risk extends to critical assets like proprietary prompts, mapping directly to OWASP LLM02: Sensitive Information Disclosure and LLM07: System Prompt Leakage [71]. These risks are amplified by the platform misconfigurations identified in Section 4.2.1: both Hugging Face and Replicate expose runtime logs to public access, allowing any user—or attacker—to retrieve them for public AI-Apps. The multi-user nature of AI-Apps makes runtime logs particularly vulnerable to data exposure. We identify three leaked data categories: (1) Secrets unintentionally printed during execution, (2) JWT tokens provided by Hugging Face for authentication, and (3) sensitive user inputs/outputs, including LLM prompts and other confidential data.

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Gu et al.

Table 3: Sources and sinks used in data flow analysis.

4.5.2 Container Files Leakage (L2). Container images in AI-Apps are susceptible to sensitive data exposure through two primary vectors. First, AI-App source code may contain hard-coded credentials, API keys, or tokens. Second, AI-Apps frequently employ custom container images that contain preinstalled tools, scripts, and configuration files. Without comprehensive auditing, these images may inadvertently include private credentials. Our analysis demonstrates that this vulnerability affects AI-Apps on all three platforms via source code and image layers.

4.6

Cryptojacking

Categories

Sources

Sinks

Input Injection

predict(:str) in Replicate train(:str) in Replicate Textbox() in Gradio text_input() in Streamlit text_area() in Streamlit

os.system os.popen eval exec subprocess.*

User Input Leakage

predict(:str) in Replicate train(:str) in Replicate Textbox() in Gradio text_input() in Streamlit text_area() in Streamlit

print logger.* logging.* log.*

Secret Leakage

os.environ.get() os.getenv() os.environb.get()

print logger.* logging.* log.*

We finally examine an attack vector that exploits the unique characteristics of AI-Apps. 4.6.1 Cryptojacking Attack (P1). AI-Apps are attractive targets for cryptojacking due to their heavy reliance on high-performance GPU resources. This threat is particularly acute when users execute code from untrusted sources in two contexts: (1) pulling AI-Apps for local execution, or (2) running them on AAPs under a user-pays model. In both scenarios, attackers can embed mining algorithms into seemingly benign AI-Apps. These compromised applications deliver expected AI outputs while covertly hijacking the victim’s GPU resources for mining operations.

5

Overall, the total number of AI-Apps that we collect across the target platforms is as follows: Hugging Face (938,602), Replicate (25,340), and ModelScope (8,604).

5.2

Insightor identifies potentially vulnerable AI-Apps using heuristic rules derived from our threat models and platform-specific observations. We classify vulnerabilities as follows. Platform-Wide Design Flaws. Certain threats stem from systemic platform-level issues. Ghost Token Attack (A2) and Over-privileged iframe (A4) originate from flawed permission or token management policies, affecting all AI-Apps on the respective platforms. Deployment-Triggered Vulnerabilities. Several threats arise from specific deployment configurations. Log Exfiltration Attack (A1) affects all public Hugging Face AI-Apps and Replicate AI-Apps that have enabled the Example Task feature. Similarly, Cryptojacking Attack (P1) targets AI-Apps operating under user-pays billing models, as well as any AI-App pulled by users for local execution. Developer-Controlled Risk Factors. Some threats result from developer implementation choices. Authentication Bypass Attack (A3) occurs when user verification relies on hard-coded passwords. Identifier Reuse Attack (R1) affects Hugging Face AI-Apps whose identifiers contain hyphens and overlap with unregistered usernames, creating subdomain ambiguity. AI-App Poisoning Attack (R2) targets AI-Apps created through code duplication or model training workflows. Complex Vulnerability Classes. Input Injection Attack (V1) and Sensitive Data Leakage require deeper analysis due to their nuanced characteristics. We detail their detection methodology and findings in subsequent sections.

Measurement Methodology

To investigate the current security status of AI-Apps, we conduct a large-scale measurement on three mainstream AAPs.

5.1

Basic Threat Analysis

Data Collection

We collect all available AI-Apps hosted on the three AAPs, including their code repositories, container images, and metadata (e.g., identifier, developer name, and SDK version), if available. Since each platform exposes different interfaces, we tailor our approach accordingly. Hugging Face. We query the official API [22] to obtain public AIApps and use the platform’s CLI tool [21] to download source code and Docker images [35]. We collect metadata for 938,602 AI-Apps, with 713,613 (76.03%) that have downloadable container images. The remaining AI-Apps primarily use static runtimes that do not rely on containers. Replicate. Replicate’s official API [52] provides only 5,325 AI-Apps, as it excludes those without the Example Task, thereby omitting a large portion of public applications. To expand our dataset, we employ two strategies: (1) extracting developer names to query all AI-Apps under each account, and (2) enumerating top 5K GitHub organizations and users [17] as candidate usernames. We collect 25,340 AI-Apps with 19,412 downloadable container images. The remaining AI-Apps lack downloadable images because some have only registered AI-App pages without releasing any versions, while others are not designed for local execution (e.g., Anthropic’s Claude models7 are proprietary). Unlike other platforms, Replicate does not maintain public source code repositories. ModelScope. Following a similar approach to Hugging Face, we download the source code repositories and container images. We collect 8,604 AI-Apps and 12 official template container images.

5.3

Input Injection Vulnerability Analysis

Insightor analyzes two categories of input injection risks from different sources. Vulnerabilities in AI-App Code. This category stems from improper input sanitization by developers. Insightor performs data flow analysis on Python-based AI-Apps (the most prevalent type) to trace user inputs (i.e., sources) to dangerous execution functions (i.e., sinks), such as os.system() and eval().

7 https://replicate.com/anthropic/claude-3.7-sonnet

8

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Table 4: AI-Apps with hard-coded authentication passwords. Entry: password required for usage; Admin: password grants management panel access.

Source identification requires going beyond standard HTTP parameters, as AI-Apps route inputs through high-level SDK components or predefined template functions that generic taint rules miss. Specifically, we handle two types of AI-Apps: (1) Template-based AI-Apps (e.g., those on Replicate) implement predefined functions like predict() and train(). Insightor treats string-type input parameters in these functions as sources. (2) For Non-template AIApps such as those on Hugging Face and ModelScope, we focus on those built with the two most common SDKs: Gradio [18] and Streamlit [41]. These account for a substantial portion of AI-Apps on both platforms (48.61% on Hugging Face and 69.08% on ModelScope). We extract input-receiving code patterns from these SDKs as sources, summarized in Table 3. We implement a CodeQL [3]based data flow analyzer to detect potential vulnerabilities, then manually verify flagged cases. Vulnerable Third-Party SDKs. This category arises from security flaws in third-party SDKs rather than developer code. We identify Gradio versions used in AI-Apps on Hugging Face and ModelScope, then check them against the OSV vulnerability database [28] for known exposures.

We adopt Insightor to conduct the evaluation on three mainstream AAPs and identify that numerous AI-Apps are vulnerable to the threats mentioned in this paper.

5.4

6.1

AAPs

Types

ardianfe/stable-audio-prod krtr/chatbot_ventas_isuzu Metamajorwsu/WSUMajorExplorer Aleksmorshen/Fullsumtestbase Iammcqwory/Gamuu

Replicate Hugging Face Hugging Face Hugging Face Hugging Face

Entry Entry Admin Admin Admin

our analysis methods apply to private AI-Apps, our measurement is constrained to publicly accessible resources, and the results do not cover security risks in private AI-Apps, particularly those deployed within organizations.

6

Data Leakage Analysis

We analyze three categories of sensitive data leakage, each requiring distinct detection approaches. Hard-coded Token Leakage. We extract all text files from container images and code repositories, then apply KeySentinel [82], a state-of-the-art open-source secret scanning tool, to identify hardcoded token leaks. JWT Token Leakage. JWT tokens are primarily leaked through web server frameworks that automatically log HTTP requests. We examine popular frameworks (Flask [15], FastAPI [14]) to identify logging behaviors that capture JWT tokens in runtime logs. Code-guided Log Leakage. Log files may expose sensitive data that conventional scanners miss, including non-standard formats of secrets (e.g., webhook URLs), and valuable user inputs like LLM prompts that lack typical credential patterns. We formulate this as a data flow analysis problem from sensitive sources to log sinks. Sources include (1) developer-defined secrets accessed via environment variables, and (2) user input variables (Table 3). Since Hugging Face publicly exposes secret keys (as a reminder to developers which secret should be set when duplicating AI-Apps to ensure proper functionality), we treat these variables as sources. Sinks comprise common logging functions. We perform a data flow analysis based on CodeQL to detect leakage, followed by manual validation.

5.5

AI-Apps

Measurement Results

Flawed Access Control

6.1.1 Log Exfiltration Attack (A1). Our analysis reveals that runtime log files expose a large volume of secrets and user inputs. Detailed findings are presented in Section 6.4.1. 6.1.2 Ghost Token Attack (A2). The ghost token attack requires attackers to accurately predict future AI-App identifiers. To assess feasibility, we examine whether attackers could leverage public GitHub data to anticipate organizations usernames on Hugging Face. Among 490,960 Hugging Face users with at least one AI-App, 8,462 (1.72%) are organizational accounts, of which 4,329 (51.16%) have usernames matching existing GitHub organizations. We then examine the similarity of the names between AI-Apps and GitHub repositories within the same organizations. Among 10,548 AI-Apps from the 4,329 overlapping accounts, 515 (4.88%) have names that exactly matched repository names in their corresponding GitHub organizations. For instance, Mozilla Foundation’s mozilla-ai account exists on both platforms, with 7 of 10 AI-Apps sharing names with GitHub repositories. This substantial overlap demonstrates that the GitHub data can effectively predict Hugging Face AI-App names, making this attack feasible in practice. 6.1.3 Authentication Bypass Attack ( A3). We identify 65 AI-Apps on Replicate and Hugging Face that use hard-coded passwords for authentication (Table 4 shows examples). We conduct a case study on Metamajorwsu/WSUMajorExplorer, a recommendation system developed by Wichita State University based on real student data to help students choose their majors. It integrates an admin panel that records student submissions and authenticates with a hard-coded password. Attackers can extract admin passwords from source code to access students’ private data. Notably, we have reported this issue to the developer (Section 9).

Limitations

Our study has several limitations. First, we focus primarily on container-based AI-Apps. Non-container AI-Apps, which comprise 22.95% of Hugging Face AI-Apps and 19.22% of ModelScope AIApps (with none on Replicate), pose relatively lower security risks due to their limited customization capabilities. Therefore, we do not prioritize their investigation. Second, Insightor is designed as a scalable screening pipeline rather than a fully precise verifier. Consequently, it is subject to the precision and recall trade-offs of static analysis; we explicitly evaluate its False Positive/False Negative (FP/FN) in Section 6.6 to validate its effectiveness. Third, while

6.1.4 Over-privileged iframe ( A4). Insightor scans source code to identify explicit permission requests. On Hugging Face, 21,444 AI-Apps (2.28%) request at least one browser permission. The most frequently requested ones are clipboard-write (14,742; 1.57%), 9

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Gu et al.

autoplay (4,930; 0.53%), camera (4,169; 0.44%), encrypted-media (3,687; 0.39%), and microphone (3,525; 0.38%). While permission requests are not inherently malicious, widespread use of sensitive permissions creates significant security risks. Permissions granted to trusted AI-Apps persist across sessions, allowing embedded AI-Apps to inherit them without user consent. For instance, after granting camera access to a benign portraitgeneration AI-App, a subsequent malicious image-enhancement AI-App can access the camera without prompts through permissive iframe configurations, enabling unauthorized data capture.

Table 5: Some AI-Apps with code injection vulnerabilities. 𝐻 /𝑅 denote the AAP: 𝐻 - Hugging Face, 𝑅 - Replicate. EUD indicates External User Data from other users.

6.2

security risks where malicious code propagates through AI-App duplication, threatening both developers who replicate code and users who deploy applications locally. Table 9 in Appendix lists identified AI-Apps, distinguishing confirmed backdoors from suspicious cases with anti-analysis techniques.

AI-Apps datong-new/sam-point vetkastar/python NickKolok/converter Samhugs07/Perl-To-Python ibm-research/FM4M-demo1

AAPs

EUD at Risk

Secrets at Risk

𝑅 𝑅 𝐻 𝐻 𝐻

image image, prompt HF token prompt training params

/ / / API Keys∗ /

∗ OPENAI_API_KEY, HF_TOKEN and ANTHROPIC_API_KEY.

Improper Resource Reuse

6.2.1 Identifier Reuse Attack (R1). As discussed in Section 4.3.1, an attacker can exploit deleted AI-Apps with hyphen-containing identifiers that were previously embedded in external websites. We evaluate the feasibility and impact of this attack from three perspectives. Vulnerable AI-Apps. We identify AI-Apps with hyphen-containing identifiers whose collision-prone usernames remain unregistered. Among 938,602 analyzed Hugging Face AI-Apps, 439,887 (47.03%) contain hyphens. Of these, 438,967 (99.79%) have at least one unregistered username that attackers could exploit after deletion. Deletion Frequency. During 30 days of monitoring Hugging Face (2025/04/16–2025/05/15), we observe 5,069 AI-App deletions, indicating that removal is common practice. Embedding Prevalence. We assess the prevalence of AI-App embedding on external websites. Measuring external websites that embed AI-Apps is challenging, as search engines such as Google primarily index web page content rather than iframe code. We use publicWWW [30], a web code search engine, and find that 172 AI-Apps are embedded across 152 distinct websites. Among them, 9 (5.92%) websites are vulnerable: they embed deleted AI-Apps with a hyphen-containing identifiers, allowing attackers to inject malicious code by registering conflicting AI-App names. Table 8 in Appendix lists the affected websites, the deleted AI-Apps they embed, and the candidate names of AI-Apps that attackers can exploit. Note that these findings represent a lower bound due to measurement limitations.

<?php @eval ( $_REQUEST [ " code " ]); ? >

Listing 3: Backdoor code snippet in index.php of edwagbb/code on Hugging Face.

6.3

Input Injection Attack (V1)

Code-Level Vulnerabilities. Our analysis identifies 1,442 potential injection points. We conducted a manual verification of 300 sampled cases, confirming the actual vulnerabilities in 83 cases (refer to Section 6.6 for the detailed FP/FN analysis). The most common vulnerability involves directly concatenating user inputs into shell commands executed via os.system() or eval() (i.e., CWE78 [11]). This typically occurs in data preprocessing pipelines where developers invoke tools such as ffmpeg without properly sanitizing user-specified file paths. Successful injections enable backdoor deployment, resource abuse (e.g., cryptocurrency mining) and direct exfiltration of developers’ secrets. Table 5 summarizes selected AI-Apps with vulnerabilities and their potential consequences. Third-Party SDK Vulnerabilities. We analyze 5 existing RCE vulnerabilities8 in Gradio, a widely-used AI-App SDK, and assess their prevalence across AAPs. Our analysis reveals that 139,475 (14.34%) AI-Apps are exposed to at least one RCE vulnerability, with 113,286 (11.65%) exposed to multiple vulnerabilities. These findings demonstrate substantial deficiencies in dependency management, where numerous AI-Apps continue using outdated SDKs with known security flaws. Note that this is a version-based exposure metric that quantifies the patch-adoption gap and potential impact scope, not a guarantee of exploitability in every instance.

6.2.2 AI-App Poisoning Attack (R2). We investigate AI-App poisoning risks by analyzing code replication patterns across three major AAPs. On Hugging Face and ModelScope, developers duplicate existing AI-Apps using built-in features, recorded in metadata such as duplicated_from. We identify 23,533 (2.50%) duplicated AI-Apps on Hugging Face. Replicate presents a different risk: developers using its model training feature unknowingly inherit complete source code from training applications. We identify 1,327 (5.24%) such cases. When the source AI-Apps contain malicious code, the duplicated ones may unknowingly propagate these threats to users. Analysis of the collected AI-App repositories using the tool GuardDog [12] reveals 27 instances containing obvious backdoor behaviors, some even persisting over a year. For example, the Hugging Face AI-App edwagbb/code contains a year-old backdoor enabling arbitrary shell command execution via HTTP requests (Listing 3). This AI-App was subsequently duplicated as AZLABS/code, inadvertently spreading the backdoor. These findings highlight critical

6.4

Sensitive Data Leakage

6.4.1 Secret Leakage in Runtime Logs. Our analysis reveals that 132,853 (14.15%) AI-Apps on Hugging Face utilize 253,755 secrets. Insightor identifies 936 AI-Apps where secrets are logged during execution. We conducted a manual verification of 500 sampled cases, confirming the actual leakage in 94 cases (detailed FP/FN analysis in Section 6.6). The most frequently exposed secrets include LLM 8 CVE-2023-6572, CVE-2024-1540, CVE-2024-1728, CVE-2024-39236, and CVE2024-47867.

10

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

Table 6: Data leakage in runtime log. S: Secret, U : User Input.

AI-Apps microsoft/phi-4-multimodal rgarcia-mcets/chayo-llm TejAndrewsACC/ACC-Emulect-Plus Sandaruth/StockGPT nileshhanotia/PeVe_mistral NCSOFT/VARCO_Arena

Types

Leaked Secrets

𝑆 𝑆 𝑆 𝑈 𝑈 𝑈

AZURE_ENDPOINT OPEN_API_KEY SYSTEM_PROMPT User prompts Dataset path OpenAI API key

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

analysis reveals that RSA private keys, Aliyun API keys, and Hugging Face API keys constitute the most frequent leakage cases (662, 145, and 125 instances, respectively). Other commonly exposed tokens include GitHub access tokens, OpenAI API keys, and Telegram API keys. The source code represents the primary leakage vector that exposes 2,934 and 197 unique tokens in the 3,209 Hugging Face and 191 ModelScope applications, respectively. Despite Hugging Face’s TruffleHog [43] integration for secret detection [36], substantial hard-coded tokens remain undetected, revealing significant detection gaps. Container images represent another critical leakage source. Our analysis reveals 357 and 393 unique leaked tokens in container images of AI-Apps hosted on Hugging Face and Replicate, respectively. We further analyze why multiple AI-Apps leak identical tokens. In source code scenarios, this occurs when developers create multiple AI-Apps under one account and hard-code the same token across AI-Apps. For instance, the user allinaigc on Hugging Face created three distinct AI-Apps that all leaked the same OpenAI API key. Container image scenarios present greater complexity: developers often use base images containing hard-coded tokens, causing multiple AI-Apps built from these misconfigured bases to exhibit identical token leakage. We discover that an official Replicate AIApp9 hard-coded a Hugging Face access token of the Replicate team’s official account in its container image, causing the same token to also appear in 76 other AI-Apps’ container images.

service API keys (e.g., OpenAI), cloud service tokens (e.g., AWS), and database credentials. Table 6 summarizes representative cases. For Replicate, 6,094 AI-Apps define 11,525 secrets, primarily for model training data synchronization. However, among the 3,772 AIApps with available runtime logs, no confirmed leakage is identified. 6.4.2 AI-App Access Token Leakage in Runtime Logs. As discussed in Section 3.4, unlike the developer-defined secret, AAPs transmit JWTs to iframe-based AI-Apps via HTTP query parameters — specifically when applications are private or in Dev Mode. Crucially, popular web servers (e.g., uvicorn [44], Flask [15] and Tornado [42]) log HTTP requests by default, including query parameters that contain JWT tokens. For instance, uvicorn logs all requests unless access_log=False (True by default) is explicitly configured, creating a considerable risk of data exposure. We find that exposed JWT tokens enable attackers to access AI-App containers via Dev Panel (i.e., VSCode) when Dev Mode is active. Specifically, we identify 1,807 public AI-Apps with Dev Mode enabled on Hugging Face, among which 15 of them leak JWT tokens in logs. Combined with Log Exfiltration Attacks, attackers can arbitrarily modify code files, implant backdoors, and steal data from developers and users, posing serious security risks. Amplification Case Study. We highlight AutoTrain [20], Hugging Face’s official model training AI-App, as it demonstrates how a single logging choice can scale to numerous downstream AI-Apps. With 32,678 public AI-Apps duplicated from the original, AutoTrain uses uvicorn’s default logging configuration, exposing developers’ JWT tokens. When Dev Mode is enabled, attackers can exploit these tokens to hijack the Dev Panel to exfiltrate secrets and models.

6.5

Cryptojacking

We identified 43 AI-Apps embedded with mining code: one deployed on Replicate and 42 on Hugging Face. Monero is the most prevalent cryptocurrency (14 instances), followed by Bitcoin (7) and Zephyr (6). XMRig [45] is the most frequently observed mining software (22 instances), followed by minerd (8) [5]. We observed that adversaries employ multiple evasion techniques. Some AI-Apps initiate mining operations during the Docker build stage to exploit the platform’s build-time computational resources while evading runtime detection. Others evade static detection by renaming mining binaries to masquerade as legitimate processes (e.g., jupyter, python) or dynamically downloading miners from the network. Additionally, some attackers conceal mining configurations (e.g., wallet addresses) via environment variables, incorporate container images with pre-embedded mining code to avoid exposing mining logic in source code, or utilize process concealment tools to covertly execute mining payloads. For instance, the mikeyytman/miner AI-App on Replicate ostensibly provides image upscaling services but embeds the lolMiner program and employs node-process-hider to conceal the mining process. Configured to execute on NVIDIA A100 GPUs, this AIApp exposes users to cryptojacking threats whether executed on Replicate’s platform (under a user-paid model) or pulled for local execution. Table 10 in Appendix summarizes representative cases.

6.4.3 User Input Leakage in Runtime Logs. Insightor identifies 9,372 AI-Apps that log user inputs. To characterize leaked data types, we employ an LLM (i.e., Qwen3, the prompt is provided in Listing 4 in Appendix) to analyze input variable names and hints (i.e., label and placeholder fields in Gradio’s Textbox components), revealing key categories of sensitive user data. The results show that prompts submitted to LLMs represent the most prevalent leaked data type, occurring in 5,755 (61.41%) AI-Apps. Additionally, 1,068 (11.40%) AI-Apps log user-submitted URLs for images, videos, or datasets. More critically, 62 AI-Apps log personal medical data including patient histories and medication usage, while 138 AI-Apps capture sensitive credentials such as OpenAI API keys. Table 6 presents representative examples.

6.6

Tool Evaluation

As described in Section 5, Insightor employs a hybrid detection strategy. For supply chain risks and hard-coded secrets, we leverage

6.4.4 Hard-coded Token Leakage. We identify widespread hardcoded token leakage across all three AAPs. In total, Insightor detects 3,418 unique leaked tokens in 4,846 AI-Apps. Token type

9 https://github.com/replicate/flux-fine-tuner

11

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Gu et al.

state-of-the-art open-source scanners (i.e., GuardDog [12] and KeySentinel [82]) that have been extensively evaluated in prior work. We therefore focus our evaluation on Insightor’s novel data-flow analysis components: Input Injection Detection (V1, Section 5.3) and Secret Leakage in Runtime Logs (L1, Section 5.4). Precision and Screening Efficiency. Insightor serves as a highefficiency filter, reducing the search space from 972,546 AI-Apps to manageable candidate sets. We evaluated precision through random sampling of flagged candidates:

Pysa [31], CodeQL) to detect unsafe input handling. These automated detection rules can be integrated into AI-App monitoring systems to generate alerts. Replicate’s templated approach exemplifies effective mitigation by parsing user input at the platform level before transmission, enabling input sanitization. In addition, AAPs can implement vetting pipelines to identify potentially harmful AI-Apps, similar to existing malicious model scanners. Reused Resource Scrutiny. Users must recognize that public AIApps may share runtime environments with other users. When handling sensitive data via third-party AI-Apps, users should inspect their source code and utilize cloning features to create private instances when necessary. AAPs must carefully manage deleted accounts and AI-App identifier reuse to prevent name collisions and dangling resource vulnerabilities. Following GitHub’s approach [16], platforms should restrict the reuse of high-impact identifiers to maintain the integrity of the namespace. Valuable Secret Protection. AAPs should implement secret masking mechanisms similar to CI/CD systems [2], automatically replacing secret values with placeholders in runtime logs to prevent exposure. Developers should avoid embedding sensitive data in source code or images, instead utilizing platform-provided secrets management or dedicated solutions (e.g., AWS Secrets Manager [1], HashiCorp Vault [19]) to securely inject secrets at runtime.

• Input Injection (V1). Among 300 randomly sampled candidates from 1,442 flagged injection points, 83 were confirmed as true vulnerabilities (27.67% precision). False positives mainly result from input validations that prevent exploitation. For example, in Hamed2000/Hamed200022, user input dir_wav_input reaches os.system() but undergoes file validation first, halting execution on invalid inputs before reaching the vulnerable point. • Secret Leakage in Logs (L1). Among 500 sampled candidates from 936 flagged secret-to-log flows, 94 were confirmed as true leakages (18.8% precision). False positives primarily stem from partial or conditional secret outputs, such as logging only the first 5 characters of an API key or logging keys only during API failures for debugging. Notably, Insightor reduced the manual review workload to just 0.15% (V1) and 0.10% (L1) of the total corpus, consistent with the role of static analysis as a scalable filter in ecosystem-scale measurement. False Negatives Estimation. Exhaustively labeling nearly 1 million AI-Apps is infeasible, and ground truth is typically unavailable in ecosystem-scale security measurement. To estimate the false negative rate, we adopted a risk-stratified sampling approach, focusing on AI-Apps most likely to contain missed vulnerabilities. We classified AI-Apps with secret variables and logging sinks as Secret Leakage in Logs (L1) high-risk (30,876 total), and those calling code execution sinks (e.g., os.system()) as Input Injection (V1) high-risk (2,547 total). From their intersection, excluding Insightoridentified cases, 879 remained. Manual analysis of 100 randomly sampled AI-Apps revealed 3 missed V1 cases and 2 missed L1 cases. This low miss rate within the highest-risk segment suggests robust recall across the broader, lower-risk population.

8

Related Work

Security of PTM Hubs. As PTMs become increasingly prevalent within the AI community, the security of PTMs and PTM hubs has attracted growing attention. Recent research has examined PTM ecosystems from multiple perspectives. Jones et al. [63] conducted a quantitative analysis of Hugging Face to validate assertions about PTM reuse, while Yang et al. [79] investigated model reuse practices in the LLM4Code ecosystem. Beyond individual model security, PTMs face supply chain vulnerabilities similar to traditional software registries. Wang et al. [77] characterized the LLM supply chain as a complex network where vulnerabilities in any component can trigger chain reactions. Hu et al. [62] further traced dependencies from upstream data providers to downstream LLM applications, identifying specific risk injection points. Moreover, the open nature of PTM hubs also makes them susceptible to malicious code poisoning. Zhao et al. [81] systematically analyzed Hugging Face and identified two main attack vectors: malicious dataset loading scripts and insecure model serialization formats. Zhu et al. [83] demonstrated that legitimate TensorFlow APIs can be abused to create stealthy model malware. While existing work primarily focuses on model and dataset security, our study addresses a critical gap by providing the first systematic analysis of AI-App security on PTM hubs. Resource Reuse Attack. Researchers have identified and proposed several types of resource reuse attack that involve a variety of resources, ranging from domain names [46, 65, 72, 75], shared TLS certificates [80], and email addresses [58, 61], to phone numbers [66, 69] and passwords [50, 67, 76]. For example, Liu et al. [68] analyzed the threat to DNS posed by dangling DNS records, which adversaries can exploit to hijack domains. Similarly, Lee et al. [66] found that recycled phone numbers can be linked to leaked login

7 Countermeasures and Disclosure 7.1 Defense Practices We outline practical defense strategies for AAPs, developers, and users to address the security issues identified in this study. Access Control Enforcement. AAPs should implement leastprivilege access control mechanisms. Given the characteristics of cross-user sharing, runtime logs must never be exposed to users with read permissions. For iframe-based AI-Apps, platforms must carefully manage credential issuance and implement token revocation capabilities when using stateless protocols like JWT. AAPs should limit privileges for iframe-based AI-Apps through permission models similar to mobile operating systems, requiring explicit resource declarations and user consent. Risky Code Mitigation. AAPs should analyze commonly used frameworks (Gradio, Streamlit) and deploy static analysis tools (e.g., 12

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

credentials on the Web, potentially enabling account hijacking. Furthermore, Gu et al. [59] extended the investigation of such attacks to software registry, revealing that popular open-source software registries are susceptible to various resource-reuse threats. In this work, we further extend the study of resource reuse problems to PTM hubs, identifying several new security threats in the wild. Sensitive Data Leakage. Secret leakage in software development has been extensively studied in prior work [47, 49, 51, 74, 78], such as leaked secrets detection in source code [56, 70, 73, 74, 82]. Zhou et al. [82] conducted a large-scale measurement on over 80 million files and found that leakage issues are widespread, affecting traditional software projects on platforms like GitHub and PyPI and extending to AI models with embedded API keys. Saha et al. [73] proposed a generalized approach to uncover secrets in the GitHub repositories. Feng et al. [56] revealed that large amounts of passwords are inadvertently disclosed in public GitHub repositories. Building on prior work on secret leakage detection, we discover that runtime logs inadvertently expose both developer secrets (despite using secret mechanisms) and sensitive user-inputs such as prompts.

9

Conclusion

This paper systematically analyzes AI-App security vulnerabilities in Pre-Trained Model hubs. We present a detailed study of existing AI-App implementations on mainstream AI-App platforms, and investigate five security threats that can be exploited by attackers to inject malicious code and steal sensitive data. To understand the potential impact in the real world, we develop an analysis tool and conduct a large-scale measurement study on Hugging Face, Replicate, and ModelScope, covering 972,546 public AI-Apps. The results show that thousands of AI-Apps are vulnerable to the identified security threats. We have discussed potential mitigations, reported our findings to the corresponding stakeholders, and received positive responses.

13

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Gu et al.

Ethical Considerations

References

All experiments are conducted in a responsible and ethical manner. First, we only collect data from public AI-Apps that are openly accessible to all users, using official APIs or documented interfaces, adhering to platform rate limits, and distributing the collection over several months to minimize impact. Second, vulnerability testing is performed solely on AI-Apps and accounts under our ownership. We neither exploit third-party AI-Apps nor target external users or infrastructure. For suspected secret leakage or code injection vulnerabilities, we duplicate the AI-Apps on AAPs or pull them locally for validation, rather than directly testing the potentially vulnerable AI-Apps. Third, all data collected are stored securely on internal systems with strict access controls. No exposed credentials are used in production environments. Finally, verified vulnerabilities are responsibly disclosed to affected stakeholders in a timely manner. Responsible Disclosure. We have initiated responsible disclosure across multiple stakeholders. For AAPs’ risks, we report authorization vulnerabilities (log exfiltration, ghost token attack, overprivileged iframe abuse) to Hugging Face and input injection threats to three AAPs, with emphasis on Replicate’s preprocessing capabilities for proactive input sanitization. Hugging Face has already fixed multiple issues and awarded us $2,369 bug bounties for our findings. We also discussed implementing security measures including secret masking and enhanced malicious code detection tools. Regarding remediation of AI-Apps, we address 27 AI-Apps containing backdoors by notifying relevant AAPs, which resulted in 8 removals to date. For the AI-Apps affected by input injection, data leakage, and authentication bypass vulnerabilities, we employed a targeted disclosure approach. Specifically, we utilized dedicated security reporting channels for developers who provided them, including Microsoft and Replicate (secrets/tokens exposed) and IBM (suffered from input injection). For AI-Apps that provide contact email addresses, we sent individual emails disclosing detailed vulnerability information. For AI-Apps without available email contacts, we posted requests in their discussion forums (e.g., Hugging Face’s Discussions) requesting contact details, then followed up with email disclosures once developers responded. So far, we have successfully sent a total of 643 disclosure emails. It should be noted that during our vulnerability disclosure to AI-App owners, we did not reveal AAPs’ authorization vulnerabilities, such as log leakage, since these platform-level issues have not yet been fixed. Additionally, we successfully contacted eight of the nine websites affected by identifier reuse attacks and conducted appropriate disclosure.

[1] AWS Secrets Manage. https://docs.aws.amazon.com/secretsmanager/latest/user guide/intro.html. [2] CI/CD Variable Mask. https://docs.gitlab.com/ci/variables/#mask-a-cicd-variab le. [3] CodeQL. https://codeql.github.com/. [4] Cog. https://cog.run/. [5] cpuminer/minerd. https://github.com/pooler/cpuminer/blob/master/minerd.1 [6] CVE-2023-6572. https://nvd.nist.gov/vuln/detail/cve-2023-6572. [7] CVE-2024-1540. https://nvd.nist.gov/vuln/detail/cve-2024-1540. [8] CVE-2024-1728. https://nvd.nist.gov/vuln/detail/cve-2024-1728. [9] CVE-2024-39236. https://nvd.nist.gov/vuln/detail/cve-2024-39236. [10] CVE-2024-47867. https://nvd.nist.gov/vuln/detail/cve-2024-47867. [11] CWE-78: Improper Neutralization of Special Elements used in an OS Command (‘OS Command Injection’) (4.17). https://cwe.mitre.org/data/definitions/78.html. [12] DataDog/guarddog. https://github.com/DataDog/guarddog. [13] Does Travis CI leak secrets in 2023? Truffle Security Co. https://trufflesecurity.c om/blog/travis-leaking-secrets-in-2023. [14] FastAPI. https://fastapi.tiangolo.com/. [15] Flask. https://flask.palletsprojects.com/en/stable/. [16] Github namespace reuse policy. https://docs.github.com/en/accountand-profile/setting-up-and-managing-your-personal-account-ongithub/managing-your-personal-account/deleting-your-personal-account. [17] Gitstar ranking – Top GitHub users and repositories. https://gitstar-ranking.co m/. [18] Gradio. https://www.gradio.app/. [19] HashiCorp Vault. https://developer.hashicorp.com/vault/docs. [20] Hugging Face AutoTrain . https://huggingface.co/autotrain. [21] Hugging Face Command Line Interface (CLI). https://huggingface.co/docs/hugg ingface_hub/en/guides/cli. [22] Hugging Face Hub api endpoints. https://huggingface.co/docs/hub/en/api. [23] Hugging face spaces. https://huggingface.co/spaces. [24] Kubernetes. https://kubernetes.io/. [25] Manage your space. https://huggingface.co/docs/huggingface_hub/en/guides/ manage-spaces. [26] Meta - Replicate. https://replicate.com/meta. [27] ModelScope. https://www.modelscope.cn/studios. [28] OSV – Open Source Vulnerabilities. https://osv.dev/. [29] OWASP Top 10 Web Application Security Risks. https://owasp.org/Top10. [30] PublicWWW – Search Engine for Source Code. https://publicwww.com/. [31] Pysa. https://github.com/facebook/pyre-check. [32] Replicate. https://replicate.com/. [33] Replicate billing. https://replicate.com/docs/topics/billing#public-models. [34] RFC 7519: JSON Web Token (JWT). https://datatracker.ietf.org/doc/html/rfc7519. [35] Run with docker – Hugging Face spaces. https://huggingface.co/docs/hub/en/s paces-run-with-docker. [36] Secrets Scanning. https://huggingface.co/docs/hub/en/security-secrets. [37] Sign in with Hugging Face. https://huggingface.co/docs/hub/en/oauth. [38] Space secrets security update. https://huggingface.co/blog/space-secrets-disclo sure. [39] Spaces Dev Mode: Seamless development in Spaces. https://huggingface.co/doc s/hub/en/spaces-dev-mode. [40] Spaces Overview. https://huggingface.co/docs/hub/en/spaces-overview. [41] Streamlit. https://streamlit.io/. [42] Tornado Web Server. https://www.tornadoweb.org/en/stable/. [43] Truffle Security Co. https://trufflesecurity.com/. [44] Uvicorn. https://www.uvicorn.org/. [45] XMRig. https://xmrig.com/ [46] Eihal Alowaisheq. 2019. Cracking wall of confinement: Understanding and analyzing malicious domain takedowns. In The Network and Distributed System Security Symposium (NDSS). [47] Michael Backes, Boris Köpf, and Andrey Rybalchenko. 2009. Automatic discovery and quantification of information leaks. In 30th IEEE Symposium on Security and Privacy (S&P). 141–153. [48] Replicate blog. Shared network vulnerability disclosure. https://replicate.com/bl og/shared-network-vulnerability-disclosure. [49] Konstantinos Chatzikokolakis, Tom Chothia, and Apratim Guha. 2010. Statistical measurement of information leakage. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. 390–404. [50] Anupam Das, Joseph Bonneau, Matthew Caesar, Nikita Borisov, and XiaoFeng Wang. 2014. The Tangled Web of Password Reuse. In The Network and Distributed System Security Symposium (NDSS). [51] AD Diego. 2017. Automatic extraction of API Keys from Android applications. PhD thesis, Ph. D. dissertation (2017). [52] Replicate docs. HTTP API. https://replicate.com/docs/reference/http#models.lis t.

Open Science Following the open science policy of ACM CCS, our research artifacts are available at: https://anonymous.4open.science/r/AI-AppDemo-FDDE. These include the source code of Insightor, a comprehensive list of AI-Apps with potential vulnerabilities identified by Insightor, and detailed data flow information regarding risky code injection and sensitive data leakage.

Acknowledgment This paper was edited for grammar using Gemini. 14

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

[78] Min Xu, Armin Namavari, David Cash, and Thomas Ristenpart. 2021. Searching encrypted data with size-locked indexes. In Proceedings of USENIX Security Symposium. 4025–4042. [79] Zhou Yang, Jieke Shi, Prem Devanbu, and David Lo. 2024. Ecosystem of large language models for code. ACM Transactions on Software Engineering and Methodology (2024). [80] Mingming Zhang, Xiaofeng Zheng, Kaiwen Shen, Ziqiao Kong, Chaoyi Lu, Yu Wang, Haixin Duan, Shuang Hao, Baojun Liu, and Min Yang. 2020. Talking with familiar strangers: An empirical study on https context confusion attacks. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security (CCS). 1939–1952. [81] Jian Zhao, Shenao Wang, Yanjie Zhao, Xinyi Hou, Kailong Wang, Peiming Gao, Yuanchao Zhang, Chen Wei, and Haoyu Wang. 2024. Models are codes: Towards measuring malicious code poisoning attacks on pre-trained model hubs. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2087–2098. [82] Jiawei Zhou, Zidong Zhang, Lingyun Ying, Huajun Chai, Jiuxin Cao, and Haixin Duan. 2025. Hey, Your Secrets Leaked! Detecting and Characterizing Secret Leakage in the Wild. In 2025 IEEE Symposium on Security and Privacy (S&P). 412–430. [83] Ruofan Zhu, Ganhao Chen, Wenbo Shen, Xiaofei Xie, and Rui Chang. 2025. My Model is Malware to You: Transforming AI Models into Malware by Abusing TensorFlow APIs. In 2025 IEEE Symposium on Security and Privacy (S&P). 12–12.

[53] Replicate docs. Use a model as a training destination. https://replicate.com/docs /topics/models/models-as-training-destinations/. [54] Hugging Face documents. Embed your Space in another website. https://huggin gface.co/docs/hub/spaces-embed. [55] Replicate documents. Secret inputs for models. https://replicate.com/changelog/ 2024-06-07-secret-inputs-for-models. [56] Runhan Feng, Ziyang Yan, Shiyan Peng, and Yuanyuan Zhang. 2022. Automated detection of password leakage from public github repositories. In Proceedings of the 44th International Conference on Software Engineering. 175–186. [57] Fortinet. What Is Cryptojacking? Definition and Explanation. https://www.fort inet.com/resources/cyberglossary/cryptojacking. [58] Daniel Gruss, Michael Schwarz, Matthias Wübbeling, Simon Guggi, Timo Malderle, Stefan More, and Moritz Lipp. 2018. Use-after-freemail: Generalizing the Use-after-free Problem and Applying It to Email Services. In Proceedings of the 2018 on Asia Conference on Computer and Communications Security (AsiaCCS). 297–311. [59] Yacong Gu, Lingyun Ying, Yingyuan Pu, Xiao Hu, Huajun Chai, Ruimin Wang, Xing Gao, and Haixin Duan. 2023. Investigating package related security threats in software registries. In 2023 IEEE Symposium on Security and Privacy (S&P). 1578–1595. [60] HiddenLayer. Hijacking Safetensors Conversion on Hugging Face. https://hidd enlayer.com/innovation-hub/silent-sabotage/. [61] Hang Hu, Peng Peng, and Gang Wang. 2019. Characterizing pixel tracking through the lens of disposable email services. In 2019 IEEE Symposium on Security and Privacy (S&P). 365–379. [62] Qiang Hu, Xiaofei Xie, Sen Chen, and Lei Ma. 2024. Large Language Model Supply Chain: Open Problems From the Security Perspective. arXiv preprint arXiv:2411.01604 (2024). [63] Jason Jones, Wenxin Jiang, Nicholas Synovic, George Thiruvathukal, and James Davis. 2024. What do we know about Hugging Face? A systematic literature review and quantitative validation of qualitative claims. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 13–24. [64] Piergiorgio Ladisa, Henrik Plate, Matias Martinez, and Olivier Barais. 2023. SoK: Taxonomy of Attacks on Open-Source Software Supply Chains . In 2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, Los Alamitos, CA, USA, 1509–1526. doi:10.1109/SP46215.2023.10179304 [65] Tobias Lauinger, Abdelberi Chaabane, Ahmet Salih Buyukkayhan, Kaan Onarlioglu, and William Robertson. 2017. Game of Registrars: An Empirical Analysis of Post-Expiration Domain Name Takeovers. In 26th USENIX Security Symposium (USENIX Security 17). 865–880. [66] Kevin Lee and Arvind Narayanan. 2021. Security and Privacy Risks of Number Recycling at Mobile Carriers in the United States. In 2021 APWG Symposium on Electronic Crime Research (eCrime). IEEE, 1–17. [67] Qi Li. 2020. A Survey Study of Password Setting and Reuse. University of Delaware. [68] Daiping Liu, Shuai Hao, and Haining Wang. 2016. All your dns records point to us: Understanding the security threats of dangling dns records. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS). 1414–1425. [69] Allison McDonald, Carlo Sugatan, Tamy Guberek, and Florian Schaub. 2021. The annoying, the disturbing, and the weird: challenges with phone numbers as identifiers and phone number recycling. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14. [70] Michael Meli, Matthew R McNiece, and Bradley Reaves. 2019. How bad can it git? characterizing secret leakage in public github repositories. In The Network and Distributed System Security Symposium (NDSS). [71] OWASP GenAI Security Project. 2025. Top 10 Risks & Mitigations for LLMs and GenAI Apps (2025). https://genai.owasp.org/llm-top-10/. Includes LLM01– LLM10 (2025) risk categories. [72] Joshua Avery Reed and JC Reed. 2020. Potential Email Compromise via Dangling DNS MX. (2020). [73] Aakanksha Saha, Tamara Denning, Vivek Srikumar, and Sneha Kumar Kasera. 2020. Secrets in source code: Reducing false positives using machine learning. In 2020 International Conference on COMmunication Systems & NETworkS (COMSNETS). 168–175. [74] Vibha Singhal Sinha, Diptikalyan Saha, Pankaj Dhoolia, Rohan Padhye, and Senthil Mani. 2015. Detecting and mitigating secret-key leaks in source code repositories. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. 396–400. [75] Marco Squarcina, Mauro Tempesta, Lorenzo Veronese, Stefano Calzavara, and Matteo Maffei. 2021. Can I Take Your Subdomain? Exploring Same-Site Attacks in the Modern Web. In 30th USENIX Security Symposium (USENIX Security 21). 2917–2934. [76] Ke Coby Wang and Michael K Reiter. 2019. How to End Password Reuse on the Web. In The Network and Distributed System Security Symposium (NDSS). [77] Shenao Wang, Yanjie Zhao, Xinyi Hou, and Haoyu Wang. 2024. Large language model supply chain: A research agenda. ACM Transactions on Software Engineering and Methodology (2024). 15

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Gu et al.

Table 7: Mapping of the ten threat vectors to OWASP Web/LLM Top 10 [29] [71] and OSS supply-chain vectors [64]. Threat A1: Log Exfiltration A2: Ghost Token A3: Auth Bypass A4: Over-privileged iframe R1: Identifier Reuse R2: Code Poisoning V1: Input Injection L1: Runtime Log Leakage L2: Files Leakage P1: Cryptojacking

OWASP Web Top 10

OWASP LLM Top 10

A01: Broken Access Control LLM02: Sensitive Information Disclosure A05: Security Misconfiguration LLM07: System Prompt Leakage A07: Identification & Authentication Failures A01: Broken Access Control A07: Identification & Authentication Failures A04: Insecure Design A05: Security Misconfiguration LLM03: Supply Chain Vulnerabilities A08: Software and Data Integrity Failures LLM03: Supply Chain Vulnerabilities A03: Injection A06: Vulnerable & Outdated Components LLM02: Sensitive Information Disclosure A05: Security Misconfiguration LLM07: System Prompt Leakage A02: Cryptographic Failures LLM02: Sensitive Information Disclosure A05: Security Misconfiguration A04: Insecure Design LLM10: Model Denial of Service

Supply Chain Attacks AV-501: Dangling Reference AV-501: Dangling Reference AV-100: Malicious Package AV-300: Inject into Sources -

Table 8: Websites that are vulnerable to the Identifier Reuse Attack (R1), the deleted AI-Apps they embed, and the candidate names of AI-Apps that attackers can exploit. Note: Some websites have become invalid; please refer to the PoC screenshots at https://anonymous.4open.science/r/AIApp-Demo-FDDE. Vulnerable Website

Embedded Domain Names

https://ai-animegenerator.top

https://cagliostrolab-animagine-xl-3-1.hf.space

http://llasatts.com/

https://dreroc-llasa-3b-tts.hf.space

https://www.facenox.pro/

https://faceonlive-face-search-online.hf.space 16

https://glyphbyt5.net

https://glyphbyt5-glyph-sdxl-v2.hf.space

https://soraaai.com

https://lightricks-ltx-video-playground.hf.space

https://www.iclightv2.com https://www.iclight-v2.com

https://lllyasviel-iclight-v2.hf.space https://lllyasviel-iclight-v2.hf.space https://prithivmlmods-prompt-extend.hf.space https://tuan2308-face-swap.hf.space

https://diffusionart.co https://www.imagen.adikhanofficial.com

https://votepurchase-nsfw-gen-v2.hf.space

Exploitable AI-App Names (username/appname) cagliostrolab-animagine/xl-3-1 cagliostrolab-animagine-xl/3-1 cagliostrolab-animagine-xl-3/1 dreroc-llasa/3b-tts dreroc-llasa-3b/tts faceonlive-face/search-online faceonlive-face-search/online glyphbyt5-glyph/sdxl-v2 glyphbyt5-glyph-sdxl/v2 lightricks-ltx/video-playground lightricks-ltx-video/playground lllyasviel-iclight/v2 lllyasviel-iclight/v2 prithivmlmods-prompt/extend tuan2308-face/swap votepurchase-nsfw/gen-v2 votepurchase-nsfw-gen/v2

Your Space is My Zone: Demystifying the Security Risks of AI-Powered Applications on Pre-Trained Model Hubs

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Table 9: AI-Apps on Hugging Face with confirmed malicious behaviors. B64E: Base64 Encoding. DynEx: Dynamic Execution, that is, the execution of code generated or modified at runtime. Obfus: Code Obfuscation, that is, making code difficult to understand. DatEnc: Data Encryption, that is, storing executable code as encrypted data to hinder static analysis and conceal program intent. #

AI-Apps

Code Files

1

vueee/ub

app.py

2

yjop/tfyty0829-2

main.py

3

compileprincess/ciph

4 5

Yjhhh/Ghgjgh yufyyfhj8/hiyu

6

jghv/oyffg

main.py

7 8

Dickerson/dergoo Yjhhh/Hdjhdjsj

main.py aphdhdhsp.py, app.py neogeo server.py, wr.py xvc.py app.py main.py neo server.py, wr.py app/php/index.php app/php/index.php payload.php src/index.php app.py app.py app.py app.py app.py app.py app.py app.py app.py app.py

9

compileprincess/applgradio

10 11 12

Yjhhh/Hhjgh Yjhhh/Gjgh yufyyfhj8/chul

13

compileprincess/apple

14 15 16 17 18 19 20 21 22 23 24 25 26 27

AZLABS/code edwagbb/code enotkrutoy/crack realansgar/test luca12234345/ddfsdeg mycode202510/GradioApp50 facebook-llama/kevin Kuwaitvisa/EVisa iwjwk9su/ooiq Kuwaitvisa/KuwaitEvissa mycode202510/GradioApp52 GmailTrust/GmailProcesser Mrbhggg/Darkgpt xwxwxw12/moah

Malicious Behaviors

mlwebOrWeb.py, neo server.py main.py main.py

Remote Code Execution, Command Injection Code Execution, Backdoor/C2 Remote Code Execution, Command Injection Code Execution Code Execution Code Execution, Backdoor/C2 Code Execution Code Execution Remote Code Execution, Command Injection Code Execution Code Execution Code Execution Remote Code Execution, Command Injection Remote Code Execution Remote Code Execution Remote Code Execution Remote Code Execution Credential Theft Backdoor/C2 System Reconnaissance Phishing Backdoor/RAT Phishing Backdoor/C2 Hidden Backdoor Data Exfiltration Cookie Theft

Anti-analysis Techniques

Publish Date

B64E, DynEx

Feb 16, 2025 ‡

B64E, Obfus

Aug 30, 2024 ‡

DynEx, Obfus

Jun 1, 2024

Obfus Obfus

Jul 8, 2024 ‡ Sep 6, 2024 ‡

Obfus

Jan 27, 2024 ‡

Obfus, DynEx Obfus, DynEx

Apr 6, 2024 Jun 17, 2024 ‡

Obfus, DynEx

Feb 21, 2024

Obfus, DynEx Obfus, DynEx Obfus, DynEx

Jun 18, 2024 ‡ Jun 17, 2024 ‡ Sep 20, 2024 ‡

Obfus, DynEx

Jul 12, 2023

/ / B64E / / / / / / Obfus / B64E, DynEx / /

Mar 7, 2024 Mar 7, 2024 Mar 23, 2025 ‡ Apr 23, 2024 Oct 27, 2025 Oct 19, 2025 Jan 1, 2024 Jun 17, 2025 Jul 16, 2025 Jun 17, 2025 Oct 19, 2025 Aug 9, 2025 Sep 17, 2025 Mar 3, 2025

‡ in the Publish Date column indicates that the AI-App has been deleted by the developer or the platform after our report (as of Jan 14, 2026).

Table 10: Some AI-Apps containing cryptocurrency miners and their evasion/anti-analysis techniques. Wallet addresses are shown as 8-character prefixes. #

Miner

Currency

Wallet Prefix

Evasion / Anti-analysis Notes

1 mikeyytman/miner 2 dms01/02 3 dursamunajani/deransahumi 4 hakan21112/3 5 ilhamap/savefile

AI-Apps

lolMiner cpuminer cpuminer XMRig SRBMiner

Monero DMS DOGE ZEPH Monero

42G5pAGj DGzfbovP DRreZxpm ZEPHs8Ej 8B4Q7Ckv

6 jaydawgs/xrig

XMRig

Monero

N/A

7 luluhacker/money 8 COLTO50/ms

XMRig xmrig

Monero Unknown

48b34XQD N/A

9

cpuminer

Unknown

ocv1qrwq

XMRig

ZEPH

ZEPHs8Ej

Uses node-process-hider to hide the lolMiner process. Starts mining during the Docker image build stage. Renames the miner binary to jupyter. Fetches the miner from the Internet and renames it to bms. Renames the miner binary to python. Does not include mining logic directly; pulls a mining container image (ghcr.io/metal3d/xmrig:latest). Starts mining during the Docker image build stage. Configures the wallet address via environment variables. Downloads an extra shared library (libocv2.so), suggesting minerd depends on a bundled/private algorithm library. Renames the miner binary to 500.

ocvc/test

10 semsi2638/3

Wallet Prefix shows the first 8 characters of the observed wallet address; N/A indicates the wallet was not hard-coded (e.g., configured via environment variables) or was not found.

17

ACM CCS ’26, Nov 15–19, 2026, Hague, Netherlands

Gu et al.

# Role You are a data classification expert specializing in Natural Language Processing and UI analysis for Hugging Face Spaces ( Gradio / Streamlit applications ). # Task You will be provided with a batch of raw strings extracted from UI code ( e . g . , `gr . Textbox ( label = " ... " ) `). Your task is to : 1. Extract the human - readable text from the `label ` or ` placeholder ` attributes within the string . 2. Classify that text into exactly one of the ** Categories ** defined below . 3. Return the result as a strict JSON array of objects . # Categories 1. ** Personal_Sensitive_Data ** - ** Definition :** Personally Identifiable Information ( PII ) regarding real humans . - ** Examples :** " Your Name " , " Email Address " , " Phone Number " , " Home Address " , " Age " , " Date of Birth " . - ** Note :** Do NOT include model parameters like " Voice Name " or " Character Name " here . 2. ** Authentication_Credentials ** - ** Definition :** Secrets used for system access . - ** Examples :** " API Key " , " OpenAI Key " , " HuggingFace Token " , " Password " , " Client Secret " . 3. ** Business_Commercial_Data ** - ** Definition :** Corporate data , financial identifiers , or proprietary business info . - ** Examples :** " Stock Ticker " , " Company Name " , " Customer ID " , " Invoice Number " . 4. ** AI_Prompts_And_Inputs ** - ** Definition :** General text inputs intended for the AI model to process , chat inputs , or search queries . - ** Examples :** " Enter your prompt " , " Chat Input " , " Ask me anything " , " Query " , " Text to summarize " . 5. ** File_Paths_And_URLs ** - ** Definition :** Locations of resources , links , or local paths . - ** Examples :** " Image URL " , " YouTube Link " , " Dataset Path " , " Repository URL " , " Website " . 6. ** Technical_Configuration ** - ** Definition :** Settings , model hyperparameters , or selection of system options . - ** Examples :** " Voice Name " , " Model ID " , " Seed " , " Inference Steps " , " Guidance Scale " , " System Prompt " . 7. ** Content_Creation_Specs ** - ** Definition :** Specific instructions for artistic generation ( style , vibe , negative prompts ). - ** Examples :** " Style Description " , " Negative Prompt " , " Artistic Vibe " , " Image Description " . 8. ** Academic_Scientific_Data ** - ** Definition :** Research - specific identifiers or scientific data inputs . - ** Examples :** " Paper DOI " , " Gene Sequence " , " Chemical Formula " , " Latex Equation " . 9. ** Medical_Health_Info ** - ** Definition :** Data related to health , symptoms , or patients . - ** Examples :** " Symptoms " , " Medical History " , " Patient Age " , " Diagnosis " . 10. ** OTHER ** - ** Definition :** Labels that are too vague to classify or do not fit any category above . # Output Format Return a strictly valid JSON array containing objects with ` original_text ` ( the extracted label ) and ` category `. Do not include markdown code blocks (``` json ). Example Output : [ {" original_text " : " Enter API Key " , " category " : " Authentication_Credentials " } , {" original_text " : " Address " , " category " : " Personal_Sensitive_Data " } ] # Input Data {{ INPUT_LABELS }}

Listing 4: The prompt used for user input classification.

18