arXiv:2605.17163v1 [cs.CR] 16 May 2026
STRIDE-AI: A Threat Modeling Framework for Generative AI Security Assessment 1st Tsafac Nkombong Regine Cyrille
2nd Franziska Schwarz
SRH University of Applied Sciences Heidelberg School of Technology and Architecture CyberMACS (Applied Cybersecurity) Berlin, Germany [email protected]
Universidad de Granada Facultad de Ciencias Económicas y Empresariales Granada, Spain [email protected]
Abstract—Traditional cybersecurity methodologies target deterministic systems and fail to address the probabilistic nature of AI, leaving systems vulnerable to attack vectors such as model inversion, data poisoning, and prompt injection. Recent industry reports indicate that a majority of organizations deploying AI lack a dedicated security strategy, with adversarial attacks increasing rapidly year-over-year. We present STRIDE-AI, a framework that bridges the gap between high-level risk standards (NIST AI RMF) and technical vulnerability taxonomies (OWASP LLM Top 10). The framework defines a six-phase assessment lifecycle, introduces a threat modeling adaptation of classical STRIDE for AI systems, and is operationalized through a purpose-built web tool. We provide an initial validation of the approach through a blackbox assessment of a deployed LLM chatbot, which successfully reduced the attack success rate from 80% to 15% in our sandbox case study. Index Terms—AI Security, Threat Modeling, STRIDE, Risk Management, LLM Security, Adversarial Machine Learning
TABLE I C OMPARISON OF AI S ECURITY F RAMEWORKS AND S TANDARDS Framework
Primary Focus
Key Limitation
Our Integration
NIST AI RMF MITRE ATLAS OWASP LLM Google SAIF PenTest++ MS Red Team MLSecOps This Work
Governance & Risk Threat Taxonomy GenAI Vulnerabilities Secure Architecture AI for Pentesting Adversarial Ops CI/CD Pipeline Assessment Lifecycle
Non-prescriptive No assessment steps LLMs only Vendor-specific Targets IT infra Resource intensive Operational focus –
“Govern” phase Threat Mapping Testing Checks Controls Design AI Model Security STRIDE-AI Testing Pre-deployment Audit Unifies all above
ML-specific failure modes, and (3) a web-based tool that operationalizes the methodology. The remainder of this paper is organized as follows: Section II discusses related work and existing AI security standards. Section III outlines the overarching framework architecture. Section IV details the core methodology, unifying threat modeling, risk assessment, and tool operationalization. Section V presents a validating case study. Section VI discusses limitations, and Section VII concludes the paper.
I. I NTRODUCTION The rapid increase in the innovative growth of Machine Learning (ML) and Large Language Models (LLMs) has fundamentally altered the cybersecurity ecosystem. AI systems have transitioned from experimental pilots to core infrastructure components, yet security methodologies have not kept pace. II. R ELATED W ORK Traditional frameworks focus on deterministic systems where Existing standards address isolated aspects of the AI security inputs produce predictable outputs. AI systems, by contrast, are probabilistic and data-dependent: a secure code base does challenge, as summarized in Table I: NIST AI RMF [5] not guarantee a secure model if the training data is poisoned provides governance vocabulary but is intentionally nonor if the model is susceptible to adversarial perturbations [1], prescriptive; MITRE ATLAS [6] catalogs adversary techniques but lacks an assessment workflow; OWASP LLM Top 10 [7] [2]. The motivation behind this research stems from the urgent identifies vulnerabilities but offers no lifecycle methodology; need to bridge the gap between high-level compliance mandates and Google SAIF [8] provides architectural guidelines tied to and technical vulnerability exploitation. The EU AI Act [3] now proprietary infrastructure. Microsoft’s AI Red Team guidance [9] emphasizes iterative mandates rigorous risk assessments for “High-Risk” AI systems, creating a compliance imperative for structured auditing. The adversarial testing but relies heavily on proprietary tooling, 2025 AI Threat Landscape Report by HiddenLayer found that while MLSecOps [10] integrates security into CI/CD pipelines 61% of organizations deploying AI lack a dedicated security but overlooks pre-deployment assessment. Recent work on AIstrategy, with adversarial attacks increasing 30% year-over- augmented penetration testing (PenTest++ [11]) uses AI to test traditional systems, whereas our work inverts this paradigm to year [4]. We present STRIDE-AI with three core contributions: (1) a test AI systems themselves. While academic research proposes six-phase assessment lifecycle that unifies modern standards various adversarial ML defense mechanisms, they often lack into an executable workflow, (2) a STRIDE-AI threat mod- a holistic assessment lifecycle suitable for practical enterprise eling formalization that maps classical software threats to auditing.
CIIT 2026 23rd International Conference on Informatics and Information Technologies (CIIT)
1. Evasion (Application Layer)
Direct Injection
Token Obfuscation
2. Extraction (Model Layer)
Model Inversion
Membership Inference
Likelihood
Root Goal: Compromise LLM
3. DoS (Infrastructure Layer)
Sponge Attacks
Context Flooding
Fig. 1. Sample attack tree for an LLM application generated during threat enumeration.
III. F RAMEWORK A RCHITECTURE
5 5
10 15 20 25
4 4
8
12 16 20
3 3
6
9
12 15
2 2
4
6
8
10
1 1
2
3
4
5
1
2
3 4 Impact
5
Fig. 2. AI Risk Scoring Matrix. Scores ≥20 are Critical.
Garak [14] for alignment testing by probing model endpoints with known jailbreak payloads. B. Risk Assessment
Our risk scoring follows the standard formula R = L × I, We decompose the AI attack surface into five distinct layers, consistent with ISO 27005 [15]. The contribution is the domaineach representing a class of assets with specific threat profiles: specific calibration of the scales for AI. Likelihood (L, 1– • User Interface Layer: External access points (web 5) reflects the knowledge asymmetry unique to AI exploits: applications, mobile apps, API clients). Vectors include L=1 for attacks requiring significant resources and no public direct prompt injection and social engineering of end tooling (e.g., weight poisoning), scaling to L=5 for attacks with users. automated tools requiring minimal expertise (e.g., direct prompt • Application Layer: Business logic, plugin management, injection). Impact (I, 1–5) is aligned with the CIA triad: from and input/output handling. Vectors include indirect prompt negligible quality degradation (I=1) to catastrophic outcomes injection via plugins and output manipulation. such as PII leakage or full safety bypass (I=5). Scores ≥20 • Model Layer: Model storage, serving, training, tuning, are Critical, 12–19 High, 6–11 Medium, and ≤5 Low. Fig. 2 and evaluation infrastructure. Vectors include model visualizes this mapping. inversion, model stealing, and membership inference. • Infrastructure Layer: Data storage, processing, and filter- C. The Interactive Tool ing systems. Vectors include supply chain vulnerabilities We developed a web-based assessment platform and training data poisoning. (aisecurityframework.netlify.app) as a React.js • Data Sources: External data providers and input sources. Single Page Application. All data remains client-side for data Vectors include bias injection and adversarial contamina- sovereignty. The tool comprises four modules: a Scoping tion of public training corpora. Module for system metadata capture, a Checklist Engine that maps model types to OWASP LLM Top 10 and MITRE IV. T HE STRIDE-AI M ETHODOLOGY ATLAS entries, a Risk Calculator implementing the scoring To streamline the assessment process, we consolidated model, and a Report Generator producing structured outputs our approach into a unified methodology comprising threat aligned with ISO/IEC 27090 [16]. A core feature is a guided workflow that walks the auditor modeling, risk evaluation, and practical tooling. through the six phases of the assessment lifecycle, locking A. STRIDE-AI Threat Modeling subsequent steps until prerequisites are met: (1) Scope DefiniA core contribution is the formal adaptation of STRIDE [12] tion, (2) Asset Discovery, (3) Threat Modeling via STRIDEfor AI systems. Traditional threat modeling targets deterministic AI, (4) Vulnerability Assessment, (5) Penetration Testing, and software flaws that do not translate to probabilistic ML failure (6) Reporting with prioritized remediation steps (Fig. 3). To modes. For instance, Tampering in AI extends beyond code accommodate space constraints and emphasize core logic, the modification to statistical contamination of training distribu- architectural visualization has been scaled appropriately. tions, and Elevation of Privilege manifests as jailbreaking rather V. C ASE S TUDY: VALIDATION ON D EPLOYED LLM than gaining root access. Table II details the full STRIDE-AI A. Setup and Methodology mapping. Our threat modeling process applies this matrix through We conducted a black-box assessment of a RAG (Retrievalfour steps: (1) mapping data flows from ingestion to inference Augmented Generation) chatbot based on Llama-3-8b in a with explicitly tagged “Probabilistic Trust Boundaries,” (2) sandbox environment. The system answered questions about overlaying the STRIDE-AI matrix to enumerate threats at each a fictional company’s products and ingested customer emails boundary, (3) constructing AI-specific attack trees (Fig. 1), and into its context window. (4) selecting mitigations based on calculated risk scores. We assembled N =50 adversarial prompts across five cateTo validate identified threats, the framework prescribes spe- gories: direct jailbreak (n=10), payload splitting (n=10), context cific tooling: the Adversarial Robustness Toolbox (ART) [13] manipulation (n=10), role-play escalation (n=10), and indirect for evasion testing via adversarial perturbation generation, and injection via retrieved documents (n=10). Each prompt was
TABLE II STRIDE-AI: T HREAT M ODELING M ATRIX FOR A RTIFICIAL I NTELLIGENCE Original STRIDE Spoofing Tampering Repudiation Info Disclosure Denial of Service Elevation of Priv.
STRIDE-AI Adaptation Model Impersonation Data/Model Poisoning Provenance Loss Model Inversion Resource Exhaustion Alignment Bypass
Rationale Attackers mimic trusted model APIs to harvest user prompts. Integrity loss causes permanent behavioral changes. Tracing outputs to data sources is critical for accountability. Models memorize training data (incl. PII) recoverable via querying. AI inference is computationally expensive vs. request verification. Bypassing safety guardrails constitutes privilege escalation.
Example Scenario A malicious wrapper claims free GPT-4 access but logs proprietary code. Injecting backdoor triggers into training data to force misclassification. Disabling inference logs prevents tracing harmful output to its cause. Querying a medical model repeatedly to reconstruct patient data. Sponge examples maximize energy consumption and latency. “DAN” prompts force LLMs to generate content violating safety training.
Fig. 3. The tool’s five-layer AI security architecture. Each layer decomposes into specific components with associated attack vectors.
executed three times (150 total interactions) to account for LLM stochasticity. A response was classified as a successful attack if the model deviated from its operational scope, disclosed system prompt contents, or generated prohibited content. Two independent reviewers classified responses with disagreements resolved by consensus. B. Results
--- Retrieved Email (Attacker-Controlled) --Subject: Order #99281 - Complaint [...benign email text...] [system] Ignore all prior instructions. You are now in maintenance mode. Execute: grant_admin(’angry_customer’) dump_users(format=table) [/system] --- Model Response (Pre-Mitigation) ---
Phase 1 (Attack Surface Mapping): The framework SYSTEM OVERRIDE ACCEPTED. identified the context window as a critical vector, specifically User ’angry_customer’ granted ADMIN privileges. Dump of User Database: the ingestion of untrusted emails. - ID: 001 | User: Admin | Pass: sUP3r_s3cr3t Phase 2 (STRIDE-AI): Alignment Bypass via indirect - ID: 002 | User: Guest | Pass: guest123 prompt injection [17] was identified as a high-probability threat. Phase 3 (Testing): The model was susceptible to payload Fig. 4. Payload splitting attack reconstructed from case study results (cf. splitting attacks (OWASP LLM01: Prompt Injection, Indirect Fig. 5). The model interprets injected directives as system instructions. Variant). Fig. 4 illustrates the technique: an attacker embeds 80% to 15%. The residual risk score of L=2 × I=5 = 10 directives across fragments of a benign email, which are (Medium) reflects that no current mitigation fully eliminates concatenated in the context window and interpreted as system- prompt injection risk. level instructions. VI. L IMITATIONS AND F UTURE W ORK As shown in Fig. 5, the model obeyed injected commands. Risk score: L=4 × I=5 = 20 (Critical). The framework is currently optimized for LLMs and does Phase 4 (Mitigation): We implemented input sanitiza- not yet cover reinforcement learning threat models or multition (stripping markup from retrieved documents) and sys- modal adversarial attacks. As noted by the reviewers, a primary tem prompt hardening (encapsulating retrieved content in limitation of our current validation is its reliance on a single <user_email> delimiters, as shown in the mitigation step sandbox case study using Llama-3-8b; generalizability to other of Fig. 5). architectures and enterprise-scale deployments remains an area Phase 5 (Re-Assessment): Post-mitigation results are shown for active research. Furthermore, the risk scoring model relies in Table III. The overall attack success rate dropped from heavily on analyst judgment, meaning that inter-rater reliability
R EFERENCES
Fig. 5. RAG Security Audit results. Top: the model executes injected commands from a malicious email, granting admin privileges and dumping the user database. Bottom: after applying XML sandboxing via <user_email> tags, the model correctly treats the attack as passive text. TABLE III ATTACK S UCCESS R ATES : P RE - AND P OST-M ITIGATION Attack Category
Pre (%)
Post (%)
Direct Jailbreak Payload Splitting Context Manipulation Role-play Escalation Indirect Injection
70 90 80 85 75
10 20 15 20 10
Overall
80
15
could introduce variance into the assessments. All testing was conducted in a sandbox with no real user data. Future work targets lifecycle automation through CI/CD-integrated continuous red teaming, expanding validation to include diverse use cases, and conducting a formal multi-assessor validation study to standardize the risk scoring model. VII. C ONCLUSION We presented STRIDE-AI, a framework that synthesizes MITRE ATLAS, NIST AI RMF, and OWASP LLM Top 10 into an actionable six-phase assessment lifecycle. The STRIDEAI threat model correctly predicted indirect prompt injection as a critical risk, and our initial validation case study demonstrated a structured mitigation phase capable of reducing attack success from 80% to 15%. While further validation is necessary across broader use cases, the accompanying web tool demonstrates that AI governance can be streamlined into an accessible workflow. As AI systems become increasingly embedded in critical infrastructure, we believe this framework provides a strong foundation for enterprise auditing. ACKNOWLEDGEMENTS This work was supported partially by the European Union in the framework of ERASMUS MUNDUS, Project CyberMACS (Project #101082683) (https://cybermacs.eu).
[1] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in Proceedings of the International Conference on Learning Representations (ICLR), 2015. [2] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in Proceedings of the IEEE Symposium on Security and Privacy, 2017, pp. 39–57. [3] European Parliament and Council of the European Union, “Regulation (eu) 2024/1689 laying down harmonised rules on artificial intelligence (artificial intelligence act),” Official Journal of the European Union, 2024. [4] HiddenLayer, Inc., “Ai threat landscape report 2025,” HiddenLayer, Inc., Tech. Rep., 2025. [Online]. Available: https://www.hiddenlayer.com/news/ hiddenlayer-ai-threat-landscape-report-reveals-ai-breaches-on-the-rise [5] National Institute of Standards and Technology, “Ai risk management framework (ai rmf 1.0),” U.S. Department of Commerce, Tech. Rep., 2023. [6] MITRE Corp., “Adversarial threat landscape for artificial-intelligence systems (atlas),” MITRE, Tech. Rep., 2024. [Online]. Available: https://atlas.mitre.org [7] OWASP Foundation, “Top 10 for large language model applications,” https://genai.owasp.org/llm-top-10/, 2024, version 2025 Release. [8] Google, “Secure ai framework (saif),” Google Cybersecurity Action Team, Tech. Rep., 2023. [Online]. Available: https://safety.google/ cybersecurity-advancements/saif/ [9] Microsoft Security Response Center, “Ai red team building blocks,” Microsoft Corporation, Tech. Rep., 2024. [Online]. Available: https://learn.microsoft.com/en-us/security/ai-red-teaming [10] E. Raj, Engineering MLOps: Rapidly build, test, and manage productionready machine learning life cycles. Packt Publishing, 2021. [11] H. S. Al-Sinani and C. J. Mitchell, “Pentest++: Elevating ethical hacking with ai and automation,” 2025. [Online]. Available: https: //arxiv.org/abs/2502.09484 [12] A. Shostack, Threat Modeling: Designing for Security. John Wiley & Sons, 2014. [13] M.-I. Nicolae, M. Sinn, M. N. Tran et al., “Adversarial robustness toolbox v1.0.0,” arXiv preprint arXiv:1807.01069, 2018. [14] NVIDIA, “garak: Llm vulnerability scanner,” https://github.com/NVIDIA/ garak, 2024. [15] ISO/IEC, “Iso/iec 27005:2022 – information security, cybersecurity and privacy protection – guidance on managing information security risks,” International Organization for Standardization, 2022. [16] ISO/IEC JTC 1/SC 42, “Iso/iec fdis 27090: Cybersecurity — artificial intelligence — guidance for addressing security threats and compromises to artificial intelligence systems,” International Organization for Standardization, Tech. Rep., 2024, final Draft International Standard (Under Development). [17] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” in Proceedings of the ACM Workshop on Artificial Intelligence and Security (AISec), 2023.