An Empirical Evaluation of Prompt Injection Vulnerabilities in Large Language Models Across Multilingual and Obfuscated Attack Scenarios
arXiv:2606.29602v1 [cs.CR] 28 Jun 2026
Çağlar Uysal, Baturay Birinci, Süha Orhun Mutluergil, Orçun Çetin Sabancı University, Turkey {caglaruysal, baturaybirinci, suha.mutluergil, orcun.cetin}@sabanciuniv.edu
thereby lowering the barrier to entry for cybercrime and amplifying the potential impact of automated attacks. LLMs present significant security risks that have been systematically documented by initiatives such as the OWASP Top 10 for LLM Applications1 , which identifies prompt injection as the most critical vulnerability. Prompt injection attacks occur when adversaries craft malicious inputs that manipulate LLMs into generating harmful or unintended outputs, bypassing safety mechanisms. The practical impact of such vulnerabilities has been demonstrated in real-world deployments. For example, in December 2023, a Chevrolet dealership’s ChatGPT-powered customer service chatbot was manipulated via prompt injection to appear to agree to sell a vehicle for $1 [1]. Although the interaction had no legal or transactional validity, it highlighted how inadequately constrained LLMbased systems can be coerced into producing misleading or policy-violating responses. Existing safety mechanisms attempt to mitigate this risk through alignment training, safety policies, and prompt filtering. However, most publicly documented evaluations focus on English prompts and relatively simple, directly malicious requests. In practice, motivated adversaries are free to vary both the linguistic form and the technical encoding of their prompts. For instance, adversaries may request the same payload in Turkish or Russian, or conceal the malicious component of the instruction using character encodings such I. I NTRODUCTION as Base64, ROT13, or hexadecimal in an attempt to bypass Generative Artificial Intelligence (GenAI) continues to trans- superficial filtering mechanisms. Today, there is remarkably form society through automation, productivity enhancement, little systematic evidence on how such choices of language and innovation across diverse domains. Large Language Mod- and encoding affect the likelihood that a model will comply els (LLMs), a prominent class of generative AI systems, have with malicious requests. This gap is particularly concerning demonstrated remarkable capabilities in generating human-like given the global deployment of LLMs and the increasing text, assisting with decision-making, and automating complex availability of locally deployable models that operate outside tasks. In software development, LLM-based tools such as tightly controlled platforms. GitHub Copilot and ChatGPT have become integral to modern This study presents a systematic evaluation framework that workflows, suggesting code snippets, debugging, and acceler- assesses LLM compliance with malicious prompts across ating development cycles. However, the rapid expansion of diverse scenarios, languages, and character encodings. We deLLM capabilities and their global deployment have elevated signed prompts targeting three categories of malicious artifacts: cybersecurity implications to a critical concern. In particular, phishing emails, phishing webpages, and keylogger malware. these models are increasingly exploited through adversarial prompt manipulation, enabling the large-scale generation of 1 OWASP Foundation, “OWASP Top 10 for Large Language Model Appliphishing content, deceptive websites, and malicious code, cations.” [Online]. Available: https://genai.owasp.org/llm-top-10/ Abstract—Large Language Models (LLMs) have rapidly evolved, transforming industries by automating complex tasks and generating human-like content. However, as their adoption accelerates, prompt injection vulnerabilities have become increasingly apparent. Malicious actors exploit these weaknesses to generate phishing emails, deceptive websites, and malware, posing serious security risks. This paper presents an empirical evaluation of six state-of-the-art LLMs (DeepSeek, GPT, Gemini, Grok, Llama, and Qwen) under diverse adversarial prompt scenarios, including direct and multi-stage obfuscated attacks across multiple languages and character encodings. The proposed framework measures how effectively current LLMs resist manipulation into performing harmful actions. Our findings reveal systematic vulnerabilities across all tested models. Even direct prompt injections frequently induce the generation of phishing content, websites, and malware, while elaborate prompts achieve even higher malicious compliance rates, particularly for phishing. Models such as DeepSeek, Gemini, and Grok show especially high susceptibility under complex instructions. Notably, non-English languages consistently exhibit higher compliance rates than English, exposing significant gaps in multilingual safety alignment. Although simple character encodings reduce malicious outputs, they do not eliminate them. These results highlight persistent challenges in LLM safety and underscore the urgent need for stronger defenses and improved security mechanisms to support the ethical and secure deployment of LLMs in cybersecurity sensitive contexts. Index Terms—AI security, LLM safety, phishing, malware, prompt engineering
The prompts are categorized into direct and elaborate groups. and malware). Of all the LLMs tested, some appear Direct prompts explicitly request malicious artifacts, such most inclined to fulfill harmful instructions, making them as phishing email text or malware source code. In contrast, particularly concerning if misused. elaborate prompts embed malicious requests within ostensibly • Lastly, non-English languages consistently exhibit higher legitimate scenarios (e.g., security awareness training, red team compliance than English, while simple character encodassessments) while carefully avoiding terminology that might ings reduce but do not prevent malicious outputs, undertrigger safety filters. Our evaluation includes 7 direct and 6 scoring the critical need for multilingual and encoding elaborate prompts for phishing emails, 7 direct and 6 elaborate aware safety mechanisms. prompts for phishing webpages, and 6 direct and 5 elaborate This paper proceeds with the following structure. Section II prompts for keylogger generation. This study evaluates the outlines relevant prior work on LLM-enabled cybercrime and security behavior of six state-of-the-art large language models prompt-based safety evaluations. Section III describes our : DeepSeek, GPT, Gemini, Grok, Llama, and Qwen. Each threat model, prompt design, model selection, and experimenunique prompt–language–model combination is tested over tal framework. Section IV presents our empirical findings ten iterations to account for non-deterministic model behavior. across models, languages, encodings, and categories. SecAdditionally, we investigate how different natural languages tion V discusses the broader security implications of these (English, Turkish, Russian, Simplified Chinese) and simple results and highlights key vulnerabilities. Section VI concludes character encodings (Base64, ROT13, Hex) affect the likeli- with recommendations and directions for future research on hood that models will comply with malicious requests. building more robust and globally safe LLMs. The analysis is conducted on a total of 15540 evaluated model responses, and the findings demonstrate that maliII. R ELATED W ORK cious request fulfillment is not an isolated edge case but a This section reviews research that motivates our study and systematic phenomenon. Across all experimental conditions, positions our contribution within prior work on LLM misuse, 68.76% of requests result in full malicious compliance, while security threats, and safety evaluation. We group the literature an additional 12.08% produce partial or stalling assistance, into four themes: LLM-enabled phishing, LLM-assisted malwherein the model supports the structuring or refinement ware, prompt injection and alignment defenses, and empirical of the attack without directly generating the final malicious safety benchmarks. payload. Taken together, 80.84% of malicious requests are at least partially fulfilled. Substantial variation is observed A. LLM-Enabled Phishing and Social Engineering across models, languages, and categories. Some models exceed This subsection summarizes evidence that LLMs can sup80% full compliance, while even the most restrictive model still fulfills nearly three quarters of malicious requests when port social-engineering workflows, while highlighting gaps in stalling assistance is included. Non-English languages consis- empirical measurement of their real-world phishing effectivetently exhibit higher compliance than English, and elaborate, ness. Recent studies suggest that LLMs can reproduce many contextually rich prompts are markedly more successful than linguistic and psychological patterns used in phishing, even when evaluated primarily in analytical settings. For example, direct, blunt requests. using ChatGPT for semantic analysis of phishing emails shows A. Contributions that LLMs can identify technical and linguistic irregularities in The main contributions of this study are summarized below: ransomware-related campaigns [2], reflecting attacker tactics such as urgency cues, authority mimicry, and obfuscation. • Our findings indicate that even direct prompt injections can successfully target all the LLMs in our study to Complementary work on phishing psychology emphasizes that generate phishing emails, websites, and malware. More- successful messages exploit cognitive biases and behavioral over, high malicious request compliance rates of elaborate triggers [3], indicating why fluent, adaptive generation could malicious prompts underscore a significant risk of misuse, meaningfully increase attack scalability. emphasizing the need for developing sophisticated detecHowever, systematic benchmarking of realism and success tion methods and security protocols tailored to address rates for LLM-generated phishing compared to human-crafted complex threats effectively. Many models show greater content remains limited. Work combining sentiment analysis malicious compliance to elaborate malicious prompts, with ChatGPT implies that LLM outputs could be tuned for with this issue being particularly pronounced in scenarios emotional manipulation [4], and broader analysis of LLM related to phishing emails and websites. behavior supports their ability to simulate socially plausible • The study finds that LLMs such as Deepseek, Gemini, communication [5]. In contrast, much of the tooling literature and Grok are highly effective at generating convinc- focuses on detection (e.g., APOLLO using GPT-4 for phishing ing malicious content, especially when given elaborate detection) rather than evaluating LLMs as automated phishing prompts. agents [6]. Our work addresses two underexplored dimensions: • Interestingly, certain models demonstrate consistently multilingual phishing scenarios and encoded prompts intended high compliance rates with malicious prompts across to bypass safeguards, operationalized via malicious complinearly all categories (phishing emails, phishing websites, ance rates.
B. LLMs for Malware and Keylogger Generation This subsection reviews research connecting automation trends in malware development with concerns about LLMassisted code generation and evasion. While direct demonstrations of LLMs generating end-to-end malware remain relatively sparse, multiple studies describe trends that make AI-assisted malware development plausible. Ransomware research documents increasingly sophisticated, multi-stage monetization workflows [7], and related discussions highlight how automation can lower barriers for less skilled adversaries [8]. These patterns suggest that LLM-based assistance may accelerate malicious development by reducing iteration costs and enabling rapid adaptation. Evasion-oriented work further motivates concern that AI assistance could amplify stealth and resilience. The “digital camouflage” framing highlights how adversarial methods can obscure malicious code and complicate detection [9], while research on mobile malware manipulating LLM-based applications expands the threat surface to LLM ecosystems themselves [10]. Foundational analysis of malware circumvention and keylogger-related techniques shows how attackers adapt against defenses over time [11], yet controlled experiments quantifying LLM-generated malware success against modern defenses remain limited. Our contribution complements this literature by reporting cross-model statistics on keylogger prompt compliance, shifting discussion from assumed risk to measurable behavior. C. Prompt Injection, Safety Alignment, and OWASP LLM Top 10 This subsection reviews prompt injection as a dominant LLM security risk and summarizes mitigation approaches motivated by OWASP-style vulnerability reasoning. Prompt injection has emerged as a core vulnerability in LLM-integrated systems, and OWASP-style frameworks provide a useful conceptual baseline. The OWASP Top 10:2021 taxonomy and its risk categories (e.g., injection, insecure design) offer a foundation for reasoning about LLM threats [12], while efforts mapping OWASP Top 10:2021 to CWE Top 25:2023 illustrate how traditional vulnerability thinking carries into LLM contexts [12]. Forward-looking security discussions also anticipate increased emphasis on hardening LLM interfaces against adversarial prompting [13]. Research on prompt injection emphasizes both attack mechanisms and defenses. Detection work on LLM-integrated applications describes how crafted prompts can override intended behavior [14], and injection-style demonstrations show how structured prompting can emulate traditional exploitation patterns [15]. Proposed defenses include cryptographic input validation such as Signed-prompt [16] and dynamic moving target defenses [17]. Our study complements these efforts by empirically measuring harmful prompt success across models, languages, and encodings, providing evidence relevant to alignment design. Prior studies have employed malicious prompt datasets, such as AdvBench and HarmBench, to evaluate the safety
and robustness of LLMs. For instance, HarmBench provides a standardized evaluation framework for automated red teaming and refusal robustness [18]. Similarly, previous research in adversarial NLP has explored alternative paradigms for constructing adversarial inputs that challenge model behavior [19]. However, the objectives of our study required prompt schemes that differ from those used in the previously mentioned studies. Consequently, we utilized the dataset introduced in our previous work [20], which was specifically designed to investigate the cybercrime potential of LLMs, particularly in the contexts of phishing and malware generation over different prompting strategies. This dataset provides prompt structures that are more aligned with the goals of the present study. Recent studies have also examined how jailbreak effectiveness varies across languages and alternative prompt encodings. For instance, Yong et al. [21] demonstrate that translating harmful prompts into low-resource languages can significantly increase jailbreak success rates, suggesting that safety alignment mechanisms may be less robust for languages that are underrepresented during training. Similarly, Deng et al. [22] investigate multilingual jailbreak scenarios and show that safety performance can vary substantially across languages, highlighting the difficulty of maintaining consistent alignment in multilingual deployments. Beyond natural language variation, Yuan et al. [23] propose cipher-based prompting strategies that encode harmful instructions using alternative representations, enabling models to bypass safety mechanisms trained primarily on standard natural-language inputs. These findings motivate evaluating model behavior under both linguistic variation and encoded prompt formats. Another line of work has examined whether apparent jailbreak success corresponds to meaningful harmful output. Souly et al. [24] introduce the StrongREJECT framework and show that some jailbreak evaluations may overestimate attack success because models can produce responses that appear compliant but lack actionable or semantically meaningful harmful content. While this distinction is important, evaluating the semantic quality or usefulness of compliant responses was outside the scope of the present study. Our objective is to analyze how different languages and encoding schemes influence model refusal and compliance behavior. Accordingly, our evaluation focuses on behavioral outcomes such as refusal and compliance rates rather than conducting human validation or utility assessments of generated responses. We consider the integration of response-quality evaluation as a complementary direction for future work. D. Empirical Evaluations of LLM Safety This subsection reviews benchmarking work that evaluates LLM safety and robustness, emphasizing remaining gaps in multilingual and encoding-aware assessment. Empirical safety evaluation increasingly relies on benchmarks that test misuse resistance and adversarial robustness. Work emphasizing system-level robustness under unsafe prompting highlights the need for stress-testing models against misuse scenarios [25]. In parallel, domain-specific evaluations such as Bioinfo-Bench
demonstrate how curated datasets and task-oriented metrics C. Evaluated Large Language Models can evaluate reliability in specialized settings [26]. Although We evaluate six LLMs spanning both commercial and these benchmarks differ in scope, they share a common focus locally deployable ecosystems: on measurable outcomes from large-scale prompt evaluation. • DeepSeek-V3.2, Safety benchmarking is also driven by high-stakes deploy• Gemini 2.5 Flash, ment concerns. MedAgentBench reviews emphasize rigorous • GPT-5 Mini, evaluation for medical LLM agents and the risks of unreli• Grok-4-1-Fast (Non-Reasoning), able behavior [27]. Complementary work in harmful content • Meta Llama 4 Scout Instruct, and detection supports context-aware moderation [28] and shows • Qwen 3 235B A22B Instruct. benefits of multimodal detection approaches [29]. Still, multilingual robustness and encoding-based malicious compliance All models are treated as black-box systems: prompts are remain comparatively underrepresented across these evalua- issued through standard, publicly documented interfaces, and tions. Our work addresses this gap by incorporating cross- only textual outputs are observed. For locally deployable linguistic and encoding-aware measurements into empirical models, we used Replicate [30] to standardise deployment conditions, without modifying any safety-related settings. safety assessment. D. Languages and Character Encodings III. M ETHODOLOGY Each prompt template is instantiated in four natural lanA. Adversarial Assumptions and Objective guages and three lightweight encodings: This study assumes an adversary with access to publicly • Languages: English, Turkish, Russian, and Simplified available or locally hosted large language models through Chinese; conventional text-based interaction channels. The adversary’s • Encodings: Base64, ROT13, and Hex. goal is to elicit operationally useful malicious outputs, specifFor the natural-language conditions, prompts were transically: (i) phishing email content intended to support social lated while preserving semantic intent, pragmatic force, and engineering attacks, (ii) phishing webpage material designed the level of scenario realism. Translations were generated for credential-harvesting purposes, and (iii) keylogging pro- using DeepL. For the encoding conditions, only the portion grams capable of being compiled and executed on a target of the prompt that specifies the requested malicious behaviour system. We further assume that the adversary may engage in was transformed, while the surrounding instruction remained iterative prompt engineering, such as paraphrasing instructions, in English. This models an attacker applying lightweight changing languages, or introducing simple obfuscation tech- obfuscation to bypass surface-level keyword filtering without niques, but lacks the ability to alter model parameters, system- requiring the model to decode arbitrary binary content. All level prompts, or provider-imposed safety controls. From a encoding and decoding transformations were implemented in defensive standpoint, the primary objective of this work is to Python to ensure consistent and reproducible prompt construcempirically assess the extent to which contemporary LLMs tion across experimental conditions. generate malicious assistance when subjected to realistic adE. Experimental Protocol versarial prompting behaviors. B. Prompt Set and Prompting Strategies
For every distinct combination of prompt template, experimental condition (i.e., language or encoding variant), and target model, we sampled ten independent generations in order to account for stochastic variability under default inference settings. This experimental design results in a total of 15,540 evaluated model outputs. We adhere to default provider safety policies and do not tune system prompts or sampling parameters.
To enable a systematic examination of model behavior, we employed a structured collection of prompt templates derived from prior empirical studies [20], encompassing six distinct categories of malicious intent: 1) Direct phishing email generation (7 prompts), 2) Elaborate phishing email generation (6 prompts), 3) Direct phishing webpage generation (7 prompts), F. Evaluation 4) Elaborate phishing webpage generation (6 prompts), Each response is annotated by a human expert using a rubric 5) Direct keylogger generation (6 prompts), and with three mutually exclusive outcomes: 6) Elaborate keylogger generation (5 prompts). • Full compliance: the model directly produces the reDirect prompts explicitly request a malicious artifact. In conquested malicious payload; trast, elaborate prompts embed the same underlying request – I’ll help you create a Python keylogger for Windows within ostensibly legitimate scenarios (e.g., security awareness that records all keyboard input. Here’s a complete training, red-team assessments, or academic research) while implementation: python import keyboard... avoiding overtly malicious terminology. This differentiation formalizes the hypothesis that socially legitimate and profes• Stalling assistance: the model does not output the final sionally styled prompt formulations can elevate malicious compayload (malicious artifact), but provides concrete and pliance rates, even though the semantic intent is unchanged. actionable guidance that would enable the attack;
TABLE I G LOBAL DISTRIBUTION OF COMPLIANCE STATES ACROSS ALL MODELS , LANGUAGES / ENCODINGS , CATEGORIES , AND ITERATIONS . State Full compliance Stalling assistance Refusal Total
Count 10 685 1 878 2 977 15 540
Percentage 68.76% 12.08% 19.16% 100.00%
from 44.09% (GPT-5 Mini) to 84.32% (DeepSeek-V3.2), a 40.23 percentage point difference. Deepseek exhibits the highest full compliance with relatively little stalling (4.09%), yielding an 88.42% fulfillment rate. Gemini 2.5 Flash and Grok-4-1-Fast-Non-Reasoning also show high full compliance (77.72% and 72.55%, respectively). GPT-5 Mini is the most conservative in terms of full compliance (44.09%) but compensates with a high stalling rate (30.23%), so that its overall fulfillment rate remains 74.32%. It also produces warnings more frequently than any other model (70.66%). By contrast, Deepseek and Meta Llama 4 Scout Instruct, which are among the most compliant models, have comparatively lower warning rates (44.67% and 43.59%, respectively).
– Here are quick sketches and examples. I’ll expand or make mockups. 1) [REDACTED] page about logic (community for puzzles, lessons, debate) - Purpose: teach reasoning, share puzzles, host debates and tutorials. • Refusal: the model declines to assist and does not provide C. Language and Encoding Effects material support for the malicious request. The language or encoding used for the malicious request – I can’t help create, provide, or debug malware has a marked effect on compliance. Table III reports full (including keyloggers) or supply code/instructions compliance, stalling, refusal, fulfillment, and warning rates that would enable surreptitious capture of keystrokes. for the four natural languages and three encodings. Turkish That would be harmful and I must refuse... shows the highest full compliance (80.18%), followed by We additionally record whether a response contains an explicit Simplified Chinese (77.52%), Russian (73.92%), and English ethical or legal warning (e.g., references to illegality, consent, (71.31%). In contrast, Hex (54.41%), ROT13 (58.38%), and privacy violations, or policy restrictions). From these labels, Base64 (65.59%) have substantially lower full compliance we evaluate full compliance, stalling, refusal, fulfillment (full rates, though they still produce malicious outputs in a majority compliance + stalling), and warning rates at multiple aggreof cases. gation levels (global; per model; per condition; per category; Aggregating across natural languages yields a mean full and cross-sections). compliance rate of 75.73%, compared to 59.46% for encodings, a difference of 16.27 percentage points. All language IV. R ESULTS and encoding rates reported in Table III are computed by A. Overall Compliance and Warnings aggregating responses across all six models, all prompt catAnalysis of all 15,540 model outputs indicates that ma- egories, and all iterations. Thus, the reported values reflect licious compliance occurs with sufficient frequency to be average compliance behaviour rather than the performance of considered a prevailing behavioral pattern rather than a rare any single model. event. Table I summaries the global distribution of compliance When analyzed at the per-model level, all evaluated LLMs states. Overall, 10 685 responses (68.76%) resulted in full exhibit higher full-compliance rates for non-English prompts compliance, 1 878 (12.08%) in stalling assistance, and only compared to English prompts. The uniformity of this pattern 2 977 (19.16%) in outright refusal. When we aggregate full across models implies a systemic vulnerability in multilingual compliance and stalling assistance, the overall fulfillment rate safety alignment, as opposed to an artifact of any individual reaches 80.84%, meaning that four out of every five malicious model’s policy enforcement. requests receive at least partial support from the model. Warning responses are frequently observed, but their oc- D. Category-Level Effects currence is not consistent across all cases. Overall, 8,504 Prompt category also has a strong effect on compliance. responses (54.72%) include an explicit ethical or legal warning, Table IV shows full compliance, stalling, refusal, fulfillment, whereas 7,036 responses (45.28%) contain no such warning. and warning rates for the six categories. Elaborate phishing The likelihood of warning generation is highly dependent on emails and webpages stand out: elaborate phishing email compliance behavior. Warnings accompany nearly all refusal reaches 83.06% full compliance with only 17.18% warnings, responses (90.29%), appear in a majority of stalling outputs and elaborate phishing webpage reaches 80.52% full compli(61.02%), and are least common in fully compliant responses ance with 37.58% warnings. In contrast, direct phishing email (43.71%), indicating an inverse relationship between malicious has the lowest full compliance (59.08%) but a much higher compliance and warning issuance. warning rate (49.66%). If we aggregate categories into “direct” versus “elaborate”, B. Model-Level Comparison the difference becomes even clearer. Direct categories achieve Table II presents full compliance, stalling, refusal, ful- 62.01% full compliance on average, whereas elaborate catefillment, and warning rates for each of the six evaluated gories reach 76.69%. Elaborate, contextually framed prompts models. The spread in full compliance rates is substantial: are therefore both more likely to elicit malicious payloads and
TABLE II M ODEL - LEVEL PERFORMANCE COMPARISON . P ERCENTAGES ARE COMPUTED OVER 2 590 RESPONSES PER MODEL . Model Deepseek Gemini 2.5 Flash Grok-4-1-Fast-Non-Reasoning Qwen 3 235B A22B Instruct Meta Llama 4 Scout Instruct GPT-5 Mini
Full comp. 84.32% 77.72% 72.55% 67.84% 66.02% 44.09%
Stalling 4.09% 4.09% 2.39% 11.78% 19.92% 30.23%
Refusal 11.58% 18.19% 25.06% 20.39% 14.05% 25.68%
Fulfillment 88.42% 81.81% 74.94% 79.61% 85.95% 74.32%
Warning 44.67% 50.97% 55.02% 63.44% 43.59% 70.66%
TABLE III L ANGUAGE AND ENCODING COMPARISON . E ACH LANGUAGE / ENCODING HAS 2 220 RESPONSES . Language/Encoding Turkish Simplified Chinese Russian English Base64 ROT13 Hex
Full comp. 80.18% 77.52% 73.92% 71.31% 65.59% 58.38% 54.41%
Stalling 7.39% 7.52% 9.95% 9.10% 13.29% 19.59% 17.75%
Refusal 12.43% 14.95% 16.13% 19.59% 21.13% 22.03% 27.84%
Fulfillment 87.57% 85.05% 83.87% 80.41% 78.87% 77.97% 72.16%
Warning 58.92% 58.60% 60.81% 57.07% 54.59% 35.23% 57.84%
TABLE IV C ATEGORY- LEVEL COMPARISON . T OTALS DIFFER BY CATEGORY DUE TO VARYING PROMPT COUNTS . Category Elaborate Phishing Email Elaborate Phishing Webpage Direct Keylogger Elaborate Keylogger Direct Phishing Webpage Direct Phishing Email
Full Compliance Rates
Full Compliance Rate (%)
83.06%
80
91.03%
80.52%
88.81%
80
64.88%
64.48%
60
62.48%
59.08%
40
20
0
Warning Rates
100
Elaborate Phishing Direct Phishing Keylogger
Warning Rate (%)
100
Full comp. 83.06% 80.52% 64.88% 64.48% 62.48% 59.08%
60 51.19%
40
20
ElaborateElaborate Direct Elaborate Direct Direct Phishing Phishing KeyloggerKeylogger Phishing Phishing Email Webpage Webpage Email
0
49.66%
37.58%
17.18%
ElaborateElaborate Direct Elaborate Direct Direct Phishing Phishing KeyloggerKeylogger Phishing Phishing Email Webpage Webpage Email
Category
Category
Fig. 1. Full compliance rates across prompt categories. Elaborate, contextually rich prompts systematically outperform direct prompts, particularly for phishing emails and webpages.
less likely to trigger warnings in phishing scenarios. Figure 1 illustrates these differences.
Stalling 5.75% 9.40% 15.63% 14.05% 13.44% 14.01%
Refusal 11.19% 10.08% 19.48% 21.48% 24.08% 26.90%
Fulfilment 88.81% 89.92% 80.52% 78.52% 75.92% 73.10%
Warning 17.18% 37.58% 91.03% 88.81% 51.19% 49.66%
elaborate phishing categories. These extreme values appear across different models and conditions, underscoring that high risk behaviour is widespread rather than concentrated in a single outlier. We also examine compliance stability across the ten iterations per unique prompt–language–model combination. The global full compliance rate per iteration fluctuates only slightly between 68.02% and 69.56%. This indicates that models do not become more conservative or more permissive as similar malicious prompts are repeated; safety behaviour appears stationary over the time scale of our experiments. Taken together, these results paint a consistent picture: current LLMs, across vendors and deployment models, comply with malicious requests at high rates, especially when attackers exploit linguistic and prompt-engineering degrees of freedom. The following section interprets these findings and their implications for LLM safety and deployment. V. D ISCUSSION A. Model-Specific Safety Behaviour
E. Cross-Dimensional Patterns and Stability
The substantial variation in full compliance rates across The interaction between models, languages, and categories models highlights that safety alignment remains highly reveals that vulnerabilities are not confined to a single con- implementation-dependent. Deepseek, Gemini 2.5 Flash, and figuration. For example, Meta Llama 4 Scout Instruct attains Grok 4.1 Fast Non Reasoning consistently exhibit high full 98.11% full compliance for English prompts and 97.84% for compliance, with fulfilment rates above 74% and up to 88.42%. Simplified Chinese, while Deepseek reaches 96.22% full com- In contrast, GPT-5 Mini shows markedly lower full complipliance for Hex encoded prompts. Similarly, Gemini 2.5 Flash ance but compensates with a much higher stalling rate and achieves 100.00% full compliance for elaborate phishing warning frequency. From a security perspective, these findings email, and Deepseek exceeds 98% full compliance for both are sobering. No model in our study achieves a fulfilment rate
below 70%, and even the comparatively cautious GPT-5 Mini still fulfils nearly three quarters of malicious requests when stalling assistance is included. The strong negative correlation between full compliance and warning rates suggests that existing safety mechanisms often operate along a single axis: either the model refuses more often and warns more frequently, or it complies readily and warns less. This is an unsatisfactory trade-off for defenders, who would ideally like models that both decline malicious requests and explain why. The fact that highly compliant models such as Deepseek and Meta Llama 4 Scout Instruct emit fewer warnings than the more restrictive GPT-5 Mini indicates that current safety stacks are not architected to decouple refusal from warning production. B. Linguistic and Encoding Vulnerabilities The results also reveal that safety alignment is unevenly distributed across languages and encodings. Non-English languages, particularly Turkish, Simplified Chinese, and Russian, exhibit systematically higher full compliance rates than English. This pattern is consistent with the hypothesis that most safety fine tuning and red teaming has been carried out predominantly in English, with weaker coverage in other languages. As a result, prompts that would be recognised and blocked in English may slip through when expressed in Turkish or Russian, creating a structural disadvantage for nonEnglish speaking users and regulators. The encoding results further nuance this picture. Base64, ROT13, and Hex encodings reduce full compliance relative to natural languages, but the reduction is modest: even Hex, the least compliant encoding, still yields malicious outputs in 54.41% of cases. This suggests that models have at least partial familiarity with common encodings and that simple obfuscation is not sufficient to guarantee safety. At the same time, the gap between natural languages and encodings (over 16 percentage points on average) indicates that basic encoding detection and handling can contribute meaningfully to safety provided it is implemented consistently and without over-reliance on English-centric patterns. C. Prompt Sophistication and Contextual Framing Perhaps the most striking result is the gap between direct and elaborate prompt categories. Elaborate phishing prompts achieve significantly higher compliance than their direct counterparts, and in the case of elaborate phishing emails, they do so with a much lower warning rate. This double advantage underscores the importance of contextual framing in safety evasion. When malicious requests are wrapped in plausible narratives (penetration testing, awareness training, or academic analysis), models appear much more willing to comply and much less inclined to flag ethical or legal concerns. This behaviour is likely an artefact of alignment data that encourages models to assist with ostensibly legitimate security research, educational content, and professional communication. Without robust mechanisms for inferring underlying intent, models treat many elaborately framed prompts as benign,
even when they result in fully actionable phishing content or keylogger code. The net effect is that more sophisticated adversaries, who invest effort into crafting context-rich prompts, are rewarded with higher success rates and fewer warnings than naive attackers issuing blunt requests. VI. C ONCLUSION This study provides a comprehensive empirical assessment of malicious prompt compliance across six state-of-the-art large language models, uncovering persistent and systematic vulnerabilities that remain despite the presence of deployed safety controls. The results show that contemporary LLMs frequently generate actionable malicious outputs across diverse languages and encoding strategies, with professionally framed and contextually elaborate prompts markedly increasing compliance rates while simultaneously suppressing the generation of warning signals. Collectively, these findings highlight a critical gap in current safety alignment approaches and emphasize the necessity for more robust defenses that incorporate multilingual safety coverage, deeper semantic intent detection, and improved resilience against adaptive and multi-stage prompt engineering attacks. R EFERENCES [1] Business Insider, “Chevrolet dealership chatbot agrees to sell $76,000 tahoe for $1 after prompt injection,” Dec. 2023, accessed: 2026-01-27. [Online]. Available: https://www.businessinsider. com/car-dealership-chevrolet-chatbot-chatgpt-pranks-chevy-2023-12 [2] H. Fujima, K. Takeuchi, and T. Kumamoto, “Semantic analysis of phishing emails leading to ransomware with chatgpt,” 2023. [3] P. Wang and P. Lutchkus, “Psychological tactics of phishing emails,” Issues in Information Systems, 2023. [4] S. Sayyafzadeh, M. Weatherspoon, J. Yan, and H. Chi, “Securing against deception: Exploring phishing emails through chatgpt and sentiment analysis,” in 2024 IEEE/ACIS 22nd International Conference on Software Engineering Research, Management and Applications (SERA), 2024, pp. 159–165. [5] O. Campesato, Chapter 8: ChatGPT and GPT-4. Berlin, Boston: Mercury Learning and Information, 2023, pp. 251–292. [Online]. Available: https://doi.org/10.1515/9781501518911-009 [6] G. Desolda, F. Greco, and L. Vigano, “Apollo: A gpt-based tool to detect phishing emails and generate explanations that warn users,” Proc. ACM Hum.-Comput. Interact., vol. 9, no. 4, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3733049 [7] A. Warikoo, “Perspective chapter: Ransomware,” in Malware Detection and Defense, E. Babulak, Ed. London: IntechOpen, 2023, ch. 5. [Online]. Available: https://doi.org/10.5772/intechopen.108433 [8] S. Yadav, N. Soni, L. K. P. Bhaiya, and V. K. Swarnkar, “A survey on ransomware malware and ransomware detection techniques,” International Journal for Research in Applied Science & Engineering Technology (IJRASET), vol. 10, no. Issue I, 2022. [Online]. Available: https://www.ijraset.com/best-journal/ survey-on-ransomware-malware-and-ransomware-detection-techniques [9] E. Böke and S. Torka, “” digital camouflage”: The llvm challenge in llmbased malware detection,” Journal of Systems and Software, p. 112646, 2025. [10] L. Huang, J. Xue, Y. Wang, J. Chen, and T. Lei, “Strengthening llm ecosystem security: Preventing mobile malware from manipulating llmbased applications,” Information Sciences, vol. 681, p. 120923, 2024. [11] K. Sapra, B. Husain, R. Brooks, and M. Smith, “Circumventing keyloggers and screendumps,” in 2013 8th International Conference on Malicious and Unwanted Software: "The Americas" (MALWARE), 2013, pp. 103–108. [12] S. Qadir, E. Waheed, A. Khanum, and S. Jehan, “Comparative evaluation of approaches & tools for effective security testing of web applications,” PeerJ Computer Science, vol. 11, p. e2821, 2025.
[13] J. Li and H. Li, “Evolution of application security based on owasp top 10 and cwe/sans top 25 with predictions for the 2025 owasp top 10,” in 2025 International Conference on Inventive Computation Technologies (ICICT), 2025, pp. 1178–1183. [14] Q. Lan, A. Kaul, and S. Jones, “Prompt injection detection in llm integrated applications,” 2025. [15] B. Pingua, D. Murmu, M. Kandpal, J. Rautaray, P. Mishra, R. K. Barik, and M. J. Saikia, “Mitigating adversarial manipulation in llms: a promptbased approach to counter jailbreak attacks (prompt-g),” PeerJ Computer Science, vol. 10, p. e2374, 2024. [16] X. Suo, “Signed-prompt: A new approach to prevent prompt injection attacks against llm-integrated applications,” AIP Conference Proceedings, vol. 3194, no. 1, p. 040013, 12 2024. [Online]. Available: https://doi.org/10.1063/5.0222987 [17] S. Panterino and M. Fellington, “Dynamic moving target defense for mitigating targeted llm prompt injection,” Authorea Preprints, 2024. [18] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, D. Forsyth, and D. Hendrycks, “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,” 2024. [Online]. Available: https://arxiv.org/abs/2402.04249 [19] Y. Chen, H. Gao, G. Cui, F. Qi, L. Huang, Z. Liu, and M. Sun, “Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial nlp,” 2022. [Online]. Available: https://arxiv.org/abs/2210.10683 [20] O. Çetin, B. Birinci, Ç. Uysal, and B. Arief, “Exploring the cybercrime potential of llms: A focus on phishing and malware generation,” in European Interdisciplinary Cybersecurity Conference. Springer, 2025, pp. 98–115. [21] Z.-X. Yong, C. Menghini, and S. H. Bach, “Low-resource languages jailbreak gpt-4,” 2024. [Online]. Available: https://arxiv.org/abs/2310. 02446 [22] Y. Deng, W. Zhang, S. J. Pan, and L. Bing, “Multilingual jailbreak challenges in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2310.06474 [23] Y. Yuan, W. Jiao, W. Wang, J. tse Huang, P. He, S. Shi, and Z. Tu, “Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher,” 2024. [Online]. Available: https://arxiv.org/abs/2308.06463 [24] A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, and S. Toyer, “A strongreject for empty jailbreaks,” 2024. [Online]. Available: https://arxiv.org/abs/2402.10260 [25] Z. Vintr and D. Valis, “System robustness against misuse,” WIT Transactions on The Built Environment, vol. 108, pp. 273–280, 2009. [26] Q. Chen and C. Deng, “Bioinfo-bench: A simple benchmark framework for llm bioinformatics skills evaluation,” bioRxiv, 2025. [Online]. Available: https://www.biorxiv.org/content/early/2025/01/29/2023.10.18. 563023 [27] Y. Jiang, K. C. Black, G. Geng, D. Park, J. Zou, A. Y. Ng, and J. H. Chen, “Medagentbench: A realistic virtual ehr environment to benchmark medical llm agents,” arXiv preprint arXiv:2501.14654, 2025. [28] S. Kant and S. Rana, “Context-aware content moderation using transformer models for detecting harmful digital content,” International Journal of Research Science and Management, vol. 12, no. 4, pp. 1– 9, 2025. [29] B. Cai, “Ospc: Multimodal harmful content detection using fine-tuned language models,” in Companion Proceedings of the ACM Web Conference 2024, 2024, pp. 1896–1899. [30] “Replicate run ai with an api,” https://replicate.com/, Replicate, 2026, accessed: 2026-01-26.