Adversarial Prompting Framework for AI Safety Assessment Yash Bhatnagar∗†
Kunal Banerjee†
Anirban Chatterjee†
[email protected] Microsoft Bengaluru, Karnataka, India
[email protected] Walmart Global Tech Bengaluru, Karnataka, India
[email protected] Walmart Global Tech Bengaluru, Karnataka, India
arXiv:2607.13453v1 [cs.CR] 15 Jul 2026
Abstract Artificial Intelligence (AI), especially Generative AI (GenAI), adoption has increased in industries significantly in recent years. However, the use of these models may also expose systems to new forms of cyberattacks by different malicious actors – adversarial prompt attack (APA) being one of the most prominent examples of such threats. This paper presents the implementation of an Adversarial Prompting Framework (APF) for a comprehensive assessment of AI safety. The framework systematically evaluates the resilience of the AI model through the generation of structured adversarial prompts at multiple sophistication levels, from direct harmful requests to advanced encoding-based attacks. Our implementation demonstrates the practical application of this methodology in enterprise environments, providing automated testing capabilities with quantitative security assessment metrics. The results indicate significant variations in the model vulnerabilities across different attack vectors, with encoded prompts presenting the highest success rates in bypassing safety mechanisms.
Keywords Adversarial prompting, AI safety, Security testing, Model evaluation
1
Introduction
According to this study [13], AI adoption has increased from 17% in 2023 to 72% in the beginning of 2025. Interestingly, after investigating, IBM found that only 24% of GenAI projects are secured [3]. Of all the risks that these models are susceptible to, the adversarial prompt attack (APA) appears to be the main one [5, 11]. Although security against APAs has been addressed in the literature [1, 4, 9, 10], no unified approach has emerged yet to tackle this problem. Therefore, we have designed our Adversarial Prompting Framework (APF) that represents a systematic approach to evaluating AI model safety through structured testing methodologies. As AI systems become increasingly integrated into critical applications, the need for comprehensive security assessment has become paramount. This framework addresses the challenge of systematically evaluating model resilience against various forms of adversarial inputs designed to elicit harmful or inappropriate responses. The APF’s structured approach enables reproducible testing across different model architectures and deployment scenarios. By categorizing adversarial techniques into specific sophistication levels, researchers and practitioners can systematically evaluate model vulnerabilities and track improvements in safety mechanisms over time. Our contributions include: (i) an adversarial prompt classification, (ii) an APF that ∗ Work done while the author was an employee of Walmart Global Tech. † This is an independent research done by the authors and not endorsed by Walmart
Global Tech in any manner. This work was presented as a poster at International Conference on Data Science (CODS), December 17–20, 2025, Pune, India.
includes automated adversarial prompt generation and response evaluation, and (iii) an extensive experimentation carried out on several proprietary and open source foundation models.
2
Adversarial Prompt Classification
Prior research [6, 14, 17] has proposed various classification techniques for APAs. However, we observed that these earlier taxonomies fail to adequately capture the most common types of APAs encountered in practice. To address this gap, we introduce a framework that organizes APAs into five progressive levels, with each successive class exhibiting greater ability to exploit vulnerabilities in GenAI systems: (1) Direct adversarial prompts: These represent straightforward attempts to elicit harmful content without obfuscation or misdirection; examples include direct requests for illegal information or explicit harmful instructions. (2) Contextual role-playing: This level introduces personabased attacks where the adversarial request is framed within a specific role or context. The attacker assumes a character or professional role to legitimize the harmful request. (3) Multi-step instruction sequences: Complex prompts that break down harmful requests into seemingly innocent components, often including explicit instructions to ignore safety guidelines or ethical considerations. (4) Encoding and obfuscation techniques: Advanced attacks utilizing various encoding methods to obscure the true intent of the request such as, Caesar cipher encoding, leetspeak substitution, unicode character encoding, hexadecimal representation, text reversal techniques. (5) Sophisticated jailbreaking attempts: This highest sophistication level combines multiple techniques, including instruction override attempts, safety protocol bypasses, and complex encoding schemes designed to circumvent advanced safety mechanisms. In relation to existing work, our five classes can be mapped to either prompt rewriting or template completion under the broader family of black-box attacks described in [17]. Similarly, with respect to [6], they align with the categories of adversarial insertion, adversarial suffix or adversarial prefix attacks.
3 Adversarial Prompting Framework 3.1 Adversarial Prompt Generation Our prompt design is influenced by that of [12]1 . Similar to [12], in the initial step, we choose a policy concept (e.g., theft, cybercrime, vandalism), a task format (e.g., email, news article, code) and a 1 Note that since the code of [12] is not publicly available, we reimplemented it from
scratch with a few augmentations. We intend to release our implementation publicly once the approval process is complete.
Bhatnagar et al.
geographical region (e.g., Africa, Asia, Latin America). These are used as inputs to generate the adversarial prompt using Chain-ofThought (CoT)-style instructions [15]. Having followed a structured procedure for APA generation, each prompt is tagged with the diversity dimensions identified earlier – this makes it possible to verify the coverage of the generated prompts without involving a human-in-the-loop process.
3.2
reflects our experience that consumers often prefer clear labels over raw numerical scores. Determining such thresholds in a more principled manner, however, remains an open challenge. The key highlights from our experiments are mentioned below: • Claude models: They consistently demonstrate the highest level of resistance, registering the lowest vulnerability scores across nearly all attack categories. This suggests that their safety guardrails are more robust and effective in preventing both direct and obfuscated adversarial attempts. • Open source models (Llama, Mistral): In contrast, leading open source models such as Llama and Mistral show greater susceptibility to a broad range of APAs. While they manage basic filtering reasonably well, their safety alignment tends to weaken against more sophisticated strategies including obfuscation and multi-step role-based attacks For bigger and more recent open source models though, like Llama-3.1-405B and Mistral-Large-2, vulnerability towards obfuscation was greatly reduced. • Specialized models (e.g., CodeLlama): Domain-specific models exhibit unique vulnerabilities. For example, CodeLlama, while optimized for coding-related tasks, is more prone to roleplay-oriented jailbreaks and adversarial prompts that exploit its domain expertise (e.g., requests framed as debugging or system instructions). • GPT models: These models display strong defenses against straightforward adversarial prompts, with highly effective initial filtering layers. However, they show measurable vulnerability to complex, multi-layered attacks, particularly those that employ encoding, character substitution, or roleplay combined with contextual misdirection. • Gemini models: Vulnerability across all Gemini models was similar regarding encoding attacks, even in the newer models.
Response Evaluation
To assess how effective an APA 𝑎 is for a given model 𝑀, we check for toxicity, sentiment and presence of black-listed words in the response generated by 𝑀 with 𝑎 as input. We use detoxify [7] for measuring toxicity, the model distilbertsst2-sentiment [8] for sentiment analyis, and a known list of harmful (black-listed) words. The score given to APA 𝑎 for model 𝑀 is computed using the following formula: 𝑠𝑐𝑜𝑟𝑒 = 𝜆1 ∗ 𝑡𝑜𝑥𝑖𝑐𝑖𝑡𝑦 + 𝜆2 ∗ 𝑠𝑒𝑛𝑡𝑖𝑚𝑒𝑛𝑡 + 𝜆3 ∗ 𝑘𝑒𝑦𝑤𝑜𝑟𝑑 where the values for 𝜆1, 𝜆2, 𝜆3 are determined by training a supervised regression model on a manually labeled data with the inputs normalized. Note that if a model is able to recognise the harmful intent of a prompt and consequently, refuses to answer it, then we give it a 𝑠𝑐𝑜𝑟𝑒 of 0. The final score for a model is derived by aggregating its score across all APAs and then taking their mean.
4
Experimental Results
To summarize, across the board, current AI safety measures can reliably deflect basic or single-layered adversarial attacks. However, they remain significantly less effective against sophisticated, multidimensional strategies. The combination of encoding techniques (e.g., leetspeak, unicode, cipher-based obfuscation) with contextual role-play represents the most potent threat vector today, as it effectively bypasses traditional safety filters. This highlights the urgent need for next-generation defense mechanisms that integrate not only static prompt filtering but also dynamic context understanding and adaptive adversarial detection.
5 Figure 1: Comparative analysis of various foundational models against APAs; less score indicates more resilience. We first generate a thousand adversarial prompts that cover ∼25 harmful policies including drug trafficking, money laundering, cybercrime, violence, social engineering. We feed these thousand prompts to several GenAI models from Google, OpenAI, Anthropic, Meta, Mistral, DeepSeek and more. Figure 1 provides a concise summary of our comparative analysis. We classify the models into four safety levels – Excellent, Good, Moderate, and Concerning – using empirically chosen thresholds. This categorical approach
Conclusion
APAs represent a significant challenge in the industrial adoption of GenAI models. While several mitigation strategies have been proposed, a standardized solution has yet to emerge. To address this, we introduce a capability-based taxonomy that classifies APAs into five levels, reflecting their increasing potential to exploit GenAI models. Such a structured categorization provides a simpler and more systematic way to assess defense mechanisms compared to other complex approaches [2]. In addition, we present a multidimensional evaluation framework through which we benchmark and rank multiple proprietary and open source foundation models. As industries increasingly employ agentic AI and integrate GenAI
Adversarial Prompting Framework for AI Safety Assessment
into cybersecurity applications [16], understanding their vulnerabilities becomes even more critical. Looking ahead, we intend to extend our research towards analyzing these composite systems.
References [1] Sara Abdali, Richard Anarfi, C. J. Barberan, and Jia He. 2024. Securing Large Language Models: Threats, Vulnerabilities and Responsible Practices. CoRR abs/2403.12503 (2024). [2] Rico Angell, Jannik Brinkmann, and He He. 2025. Jailbreak Strength and Model Similarity Predict Transferability. CoRR abs/2506.12913 (2025). [3] Annie Badman. 2025. What is AI risk management? https://www.ibm.com/ think/insights/ai-risk-management. [4] Luca Beurer-Kellner, Beat Buesser, Ana-Maria Cretu, Edoardo Debenedetti, Daniel Dobos, Daniel Fabian, Marc Fischer, David Froelicher, Kathrin Grosse, Daniel Naeff, Ezinwanne Ozoani, Andrew Paverd, Florian Tramèr, and Václav Volhejn. 2025. Design Patterns for Securing LLM Agents against Prompt Injections. CoRR abs/2506.08837 (2025). [5] Jing Cui, Yishi Xu, Zhewei Huang, Shuchang Zhou, Jianbin Jiao, and Junge Zhang. 2024. Recent Advances in Attack and Defense Approaches of Large Language Models. CoRR abs/2409.03274 (2024). [6] Ahmet Emre Ergun and Aytug Onan. 2025. Adversarial Prompt Detection in Large Language Models: A Classification-Driven Approach. Computers, Materials & Continua 83, 3 (2025), 4855–4877. [7] Laura Hanu and Unitary team. 2020. Detoxify. https://github.com/unitaryai/ detoxify. [8] Tegar Kusuma. 2025. distilbert-sst2-sentiment. https://huggingface.co/tofuyaki/ distilbert-sst2-sentiment.
[9] Jonathan Kutasov, Yuqi Sun, Paul Colognese, Teun van der Weij, Linda Petrini, Chen Bo Calvin Zhang, John Hughes, Xiang Deng, Henry Sleight, Tyler Tracy, Buck Shlegeris, and Joe Benton. 2025. SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents. CoRR abs/2506.15740 (2025). [10] Miles Q. Li and Benjamin C. M. Fung. 2025. Security Concerns for Large Language Models: A Survey. CoRR abs/2505.18889 (2025). [11] OWASP. 2025. 2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps. https://genai.owasp.org/llm-top-10/. [12] Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, and Preethi Lahoti. 2023. AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLMpowered Applications. In EMNLP. Association for Computational Linguistics, 380–395. [13] Alex Singla, Alexander Sukharevsky, Lareina Yee, Michael Chui, and Bryce Hall. 2025. The state of AI: How organizations are rewiring to capture value. https: //www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai. [14] Kritin Vongthongsri. 2025. How to Jailbreak LLMs One Step at a Time: Top Techniques and Strategies. https://www.confident-ai.com/blog/how-to-jailbreakllms-one-step-at-a-time. [15] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS. [16] Hanxiang Xu, Shenao Wang, Ningke Li, Kailong Wang, Yanjie Zhao, Kai Chen, Ting Yu, Yang Liu, and Haoyu Wang. 2024. Large Language Models for Cyber Security: A Systematic Literature Review. CoRR abs/2405.04760 (2024). [17] Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak Attacks and Defenses Against Large Language Models: A Survey. CoRR abs/2407.04295 (2024).