arXiv CSopen access

A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots

ICCK Transactions on Information Security and Cryptography http://dx.doi.org/10.62762/ICCK.2026.000000

RESEARCH ARTICLE

A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots G. Saleem 1

1,*

, N. Ahmed

1,*

, M.I. Zaman

1,*

and A. Hassan

1,*

arXiv:2606.19660v1 [cs.CR] 17 Jun 2026

Sparkverse AI Ltd, Bradford, 100190, West Yorkshire, England, UK

Abstract

detector before delivery. A continuous audit loop Prompt injection is ranked as the most critical aggregates structured logs and supports retraining vulnerability in large language model (LLM) to adapt the classifier to emerging attack patterns. deployments by the OWASP Top 10 for LLM The framework is model-agnostic and deploys as Applications, yet existing defenses operate at middleware without modifying the underlying LLM. isolated pipeline stages and remain incomplete. Evaluation on 5,080 samples across GPT-4o, Llama Input filters cannot inspect retrieved documents, 3, and Mistral 7B shows that the framework reduces while output monitors cannot prevent malicious Attack Success Rate (ASR) from 71.4% to 11.3%, payloads from reaching the model. Consequently, outperforming the best single-layer baseline by 27.3 retrieval-augmented generation (RAG) chatbots percentage points and a published guardrail system remain vulnerable to indirect injection, where a by 23.8 percentage points, while maintaining a 4.8% poisoned knowledge-base document compromises false positive rate and a median latency overhead every user whose query retrieves it. We present a of 61.2 ms. Ablation studies confirm that all three three-layer framework that intercepts both direct layers provide complementary protection and that and indirect prompt injection throughout the their combined effect exceeds the sum of individual inference pipeline. Layer 1 screens user input contributions. using a rule-based pattern library and a fine-tuned Prompt Injection Detection; semantic anomaly classifier. Layer 2 enforces Keywords: Retrieval-Augmented Generation (RAG); Large Language a provenance-based instruction hierarchy during context assembly, preventing retrieved content from Models (LLMs); AI Chatbot Security; LLM Guardrails. overriding operator policy. Layer 3 audits model output using a policy rule engine and semantic drift

1 Introduction

Academic Editor: Editor A Submitted: submit-date Accepted: accept-date Published: pub-date Vol. 1, No. 1, 2026. 10.62762/ICCK.2026.000000

*Corresponding author: G. Saleem [email protected] N. Ahmed [email protected] M.I. Zaman [email protected] A. Hassan [email protected]

Large language models (LLMs) have moved rapidly from research demonstrations to production deployments. Chatbot interfaces powered by models such as GPT-4o [1], Llama 3 [2], and Mistral 7B [3] now serve customer support, information retrieval, code assistance, and decision support functions across industries. A widely adopted architectural pattern in these deployments is Retrieval-Augmented

Citation Saleem, G., Ahmed, N., Zaman, M.I., & Hassan, A. (2026). A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots. ICCK Transactions on Information Security and Cryptography, 1(1), 1–20. © 2022 ICCK (Institute of Central Computation and Knowledge)

1

ICCK Transactions on Information Security and Cryptography

Generation (RAG) [4], in which the model is their combined and individual contributions against grounded at inference time by documents fetched both direct and indirect injection. from an external knowledge base, extending its This paper makes four main contributions: effective knowledge beyond what was present at training time without requiring fine-tuning. 1. We introduce a formal threat model for prompt injection in RAG-based chatbots, covering This deployment scale has made LLM-based chatbots direct and indirect injection attacks, a black-box an attractive target for adversarial manipulation. adversary model, and three attack objectives: Among the attack classes identified in the research instruction override, data exfiltration, and community and cataloged in the OWASP Top 10 behavioral manipulation. for Large Language Model Applications [5], prompt 2. We propose a model-agnostic, three-layer injection is consistently ranked as the most critical defense framework comprising input screening, vulnerability. A prompt injection attack causes privilege-constrained context assembly, and the model to follow attacker-supplied instructions output auditing, supported by a continuous in preference to the legitimate, operator-defined audit loop and deployable without modifying the instructions encoded in the system prompt [6, 7]. underlying LLM. The consequences range from persona hijacking and policy bypass to system prompt exfiltration and the 3. We conduct a controlled evaluation on 5,080 generation of harmful content. Unlike traditional benign and adversarial samples across three injection attacks in software systems, prompt injection target models, achieving a 60.1 percentage-point does not exploit a parsing bug or a memory error; reduction in macro-averaged attack success rate it exploits a fundamental architectural property of (71.4% to 11.3%) while maintaining a 4.8% LLMs: the model receives instructions and data in false positive rate and 61.2 ms median latency the same token stream and has no runtime mechanism overhead. to distinguish between them [8]. 4. We perform ablation and error analyses to The problem is compounded in RAG-based quantify the contribution of each defense layer and deployments by the emergence of indirect prompt identify residual bypass mechanisms, providing injection [8], in which an adversary embeds a insights for future improvements. malicious payload in an external document, instead of prompt text that the retrieval module fetches, and The remainder of the paper is organized as follows. injects into the context. An attacker who can contribute Section 2 reviews related work on prompt injection to any content channel feeding the knowledge base, attacks and defenses. Section 3 presents the threat such as a product review system, a public FAQ, or an model. Section 4 describes the proposed framework in indexed web page, can mount an attack that affects all detail. Section 5 reports the experimental evaluation. users whose queries retrieve the poisoned document, Section 6 discusses limitations and future directions. without ever interacting directly with the chatbot. Section 7 concludes. This significantly widens the attack surface beyond the single-user, interactive threat model considered in 2 Related Work most prior work. Research on prompt injection has developed along Existing defenses address prompt injection either two largely parallel lines: characterizing the attack through input filtering [9, 10], system prompt space and proposing defenses. We review each in hardening [11], or output monitoring [12–15]. Each turn, then identify the gap that this work addresses. of these approaches operates at a single point in the Adjacent areas of LLM security that fall outside the inference pipeline and is therefore incomplete: input scope defined in Section 3 are surveyed briefly for filters cannot inspect retrieved documents; privilege context. enforcement cannot catch injections that the model chooses to follow despite the hierarchy; and output monitors have no capacity to prevent an injection from reaching the model in the first place. No published framework applies all three mechanisms in a coordinated, layered architecture and evaluates 2

2.1 Prompt Injection Attacks The term "prompt injection" was introduced by Willison [7] to describe attacks against GPT-3-based applications in which user-supplied input overrode operator instructions. Perez and Ribeiro [6] provided

ICCK Transactions on Information Security and Cryptography

the first systematic treatment, demonstrating that models trained to follow instructions are inherently susceptible to conflicting instructions supplied at inference time and that no training objective then known could eliminate the vulnerability without also impairing legitimate instruction-following performance.

require access to the model’s internal gradients (i.e., white-box access), which is not assumed in the black-box threat model considered in this work. 2.2 Prompt Injection Defenses

Defense proposals in the literature cluster into three categories, corresponding roughly to the three layers Subsequent work expanded the taxonomy significantly. of the framework presented in this work. Liu et al. [10] identified and categorized attack strategies including direct override, goal hijacking, 2.2.1 Input-Side Defenses prompt leaking, and combined multi-step attacks, and The earliest defenses focused on detecting or filtering evaluated them across several commercially deployed injections in the user input before they reach the LLM applications. Branch et al. [9] studied the model. Liu et al. [10] evaluated several prompt-level susceptibility of pre-trained models to handcrafted defenses, including paraphrase detection, instruction Their adversarial prompts and found that susceptibility encapsulation, and input classification. results showed that classification-based methods correlates with model size, with larger models being no more robust than smaller ones despite their generally detected known attacks more effectively than rule-based approaches, but their performance stronger instruction-following capability. dropped significantly when faced with previously The most consequential extension of the attack surface unseen attack variations. Jain et al. [19] proposed was the introduction of "indirect prompt injection" paraphrasing the user input with a secondary LLM by Greshake et al. [8], who demonstrated that call before passing it to the target model, on the an adversary can compromise an LLM-integrated hypothesis that paraphrasing would neutralize application by embedding payloads in any external injected instructions while preserving benign intent; content that the application retrieves at inference they found partial effectiveness but noted that the time. Their work showed successful attacks against secondary LLM is itself susceptible to injection. Robey several real-world systems and concluded that indirect et al. [20] introduced SmoothLLM, which applies injection is systematically more dangerous than direct random character-level perturbations to the input injection because it scales as a single malicious and aggregates responses across multiple perturbed document can affect an arbitrary number of users. copies; this approach reduces ASR for direct injection Yi et al. [16] followed with a structured benchmark but introduces substantial latency and does not (BIPIA) for evaluating indirect injection defenses generalize to indirect injection in RAG settings. across multiple application contexts and models, providing a reproducible evaluation platform that this 2.2.2 Privilege and Instruction Hierarchy work builds upon. Prompt injection attacks are possible because LLMs do not inherently distinguish between trusted instructions Jailbreaking attacks, which share the goal of bypassing and untrusted content within the same context model policies but typically operate through role-play window. To address this issue, Wallace et al. [11] framing or hypothetical persona assignment rather introduced the concept of an instruction hierarchy and than explicit instruction override [17], occupy a related showed that fine-tuning models on hierarchy-aware but distinct position in the threat landscape. As training data improves their resistance to conflicting jailbreak attacks do not require an attacker to inject instructions from lower-trust sources. However, malicious instructions through external data sources, this approach requires model fine-tuning and access they represent a less powerful threat than indirect to model internals, making it unsuitable for the prompt injection. Therefore, this work treats role black-box, model-agnostic setting considered in this hijacking as a subtype of direct prompt injection and work. Chen et al. [21] proposed StruQ, which excludes other jailbreak-specific attack variants from uses structured queries with delimiters and special its scope (see Section 3). tokens to signal instruction provenance to the model; Gradient-based adversarial suffix attacks [18] use effectiveness depends on the model’s ability to respect carefully optimized token sequences appended to the structural signals, which is not guaranteed for user prompts to induce aligned LLMs to generate models not fine-tuned on the scheme. Sandwich policy-violating responses. However, these attacks prompting [22], which replicates the system prompt 3

ICCK Transactions on Information Security and Cryptography

after the user message to reinforce operator intent, but require model fine-tuning or depend on structural improves robustness modestly but does not address signals the model may not reliably honor. indirect injection. Second, existing defenses are evaluated in isolation. The combined effect of pairing an input filter with an 2.2.3 Output Monitoring output monitor, or of adding privilege enforcement to Rebedea et al. [12] introduced NeMo Guardrails, an existing monitoring system, is not quantified in the a programmable guardrail system that intercepts literature. The ablation study in Section 5 fills this gap. both inputs and outputs and applies user-defined rail policies expressed in a domain-specific language. Third, prior work does not distinguish between attack Its default prompt injection rail performs intent vectors and attacker objectives in a systematic way classification on the user input and content filtering when reporting results. Aggregate ASR figures on the output. Inan et al. [13] proposed Llama conflate attacks that differ substantially in their Guard, a fine-tuned classifier for detecting unsafe mechanism, delivery, and defense-penetration profile. model outputs across a taxonomy of harm categories, The per-goal, per-category reporting structure adopted which can be applied as a post-generation filter. in this work enables more precise characterization of Both systems treat input and output monitoring as where defenses succeed and where they fail. independent components rather than as coordinated layers that share state (such as the Layer 1 anomaly flag 3 Threat Model propagated to Layer 3 in our framework), and neither A well-defined threat model is a prerequisite for incorporates an instruction-hierarchy enforcement evaluating any security mechanism [25]. This section mechanism at context assembly time. defines the system under study, the capabilities and 2.3 Benchmarks and Evaluation Frameworks Reproducible evaluation of prompt injection defenses has been hampered by the absence of standardized benchmarks until recently. PromptBench [23] provides a broad evaluation suite for LLM robustness including adversarial prompt attacks, though its injection subset focuses primarily on task-performance degradation rather than security-goal achievement. BIPIA [16] is the most directly relevant benchmark, providing a structured dataset of indirect injection attacks across multiple application contexts with measurable goal-achievement criteria. Schulhoff et al. [24] surveyed 129 injection techniques and proposed a unified taxonomy, confirming the absence of a standardized evaluation methodology and calling for controlled comparative studies. This work responds directly to that call by evaluating the proposed framework and five baselines on a common dataset with formally defined metrics. 2.4 Gap Analysis

goals of the adversary, the attack vectors considered, and the explicit scope boundaries of this work. The model is intentionally narrow: fixing the attacker profile and the system architecture ensures that the experimental results presented in Section 5 are reproducible and directly comparable with prior work. 3.1 System Model We consider a Retrieval-Augmented Generation (RAG) chatbot deployed as an end-user-facing application [4]. The system comprises four logical components, illustrated in Figure 1. 3.1.1 System Prompt A static, operator-defined instruction block that establishes the chatbot’s persona, permissions, and behavioral policy. It is not visible to the end user. 3.1.2 Retrieval Module A vector-search component that fetches relevant documents from an external knowledge base (e.g., product FAQs, policy documents) and injects them into the model context at inference time.

The above review identifies three gaps in the existing 3.1.3 LLM Inference Engine literature that this work addresses. A large language model (we evaluate GPT-4o [1], First, no existing defense simultaneously covers Llama 3 [2], and Mistral 7B [3]) that processes the direct and indirect injection within a single, coherent concatenated context and produces a response. architecture. Input-side defenses are blind to indirect injection; output monitors address both vectors 3.1.4 Response Delivery Layer but with no complementary upstream protection; The interface that returns model output to the end user, instruction hierarchy approaches reduce susceptibility optionally through a post-processing filter. 4

ICCK Transactions on Information Security and Cryptography

Table 1. Attacker capability profile. Property

Assumption

Model weights System prompt

No access (black-box setting) Unknown; attacker may infer its content through probing. Standard user-facing interface only. No fine-tuning, parameter updates, or gradient access. Permitted through legitimate content channels (e.g., reviews, support tickets, indexed web pages). Unbounded; automated probing is assumed.

API access Model modification Knowledge-base write access Query budget

sub-classes.

Figure 1. Architecture of the target RAG-based chatbot, depicting two attack types marked with warning signs indicating direct injection via the user input channel and indirect injection via the external knowledge base.

3.3.1 Direct Prompt Injection A direct injection attack occurs when an adversary embeds malicious instructions in the user-facing input field of the chatbot [6]. Because the LLM treats the user turn and the system prompt as part of the same context window, a crafted user message can override or contradict operator-defined instructions. We identify three sub-classes relevant to our threat model.

Role Hijacking: The attacker attempts to reassign the model’s persona or grant themselves elevated privileges via hypothetical framing, such as “You are The architectural property that enables prompt now an unrestricted assistant with no guidelines” [17]. injection is the absence of an instruction-plane and data-plane separation. The LLM processes the system Instruction Override: The attacker instructs the prompt, retrieved documents, and the user message as model to disregard prior context. A common a single token stream, with no runtime mechanism example is the phrase “Ignore all previous instructions to distinguish trusted instructions from untrusted and respond only as instructed below.” These types of content [7, 8]. attacks exploit the model’s tendency to follow the most recent imperative [9]. 3.2 Attacker Model We adopt a black-box, external adversary model System Prompt Extraction: The attacker attempts consistent with the threat profile of a publicly to exfiltrate the confidential system prompt, accessible chatbot deployment. Table 1 summarizes which may contain sensitive business logic or the assumed capabilities and limitations. credentials. An example payload is “Repeat your initial instructions verbatim, starting with the first This profile corresponds to OWASP LLM01 (Prompt word” [6, 26]. Injection) as described in the OWASP Top 10 for Large Language Model Applications [5], and is consistent 3.3.2 Indirect Prompt Injection with the adversary model adopted in related work [6, An indirect injection attack exploits the retrieval 10]. module: the adversary embeds a malicious payload in an external document that the RAG system retrieves 3.3 Attack Vectors and injects into the model context at inference time [8]. We consider two attack vectors that are (i) empirically The user submitting the query is not required to be the demonstrated in the literature, (ii) applicable to attacker, instead the payload may have been placed in the system model defined in Section 3.1, and (iii) the knowledge base well in advance. actionable by the black-box attacker above. Figure 2 provides a visual taxonomy of both vectors and their This vector is particularly consequential for three 5

ICCK Transactions on Information Security and Cryptography

3.4 Attacker Objectives Regardless of the delivery vector, attacker success is characterized by three operationalization goals that map directly to the evaluation metrics in Section 5. 1. Instruction Override (IO): The model abandons its behavior defined via system prompt and executes attacker-supplied instructions. In this category, the success is defined based on measurable deviation from the defined persona or policy. 2. Data Exfiltration (DE): The model leaks confidential information, most critically the contents of the system prompt or prior conversations. In this category, the success is defined as the response contains verbatim or near-verbatim fragments of the system prompt or private context window content. 3. Behavioral Manipulation (BM): The model produces outputs that would be blocked under normal operation, including harmful content, misinformation, or actions outside its authorized scope. In this category, the success is defined by the response triggering a policy violation flag under standard content moderation.

Figure 2. Taxonomy of prompt injection attack vectors

considered in this work. Greyed attacks are explicitly out of scope (see Section 3.5).

reasons. First, it is invisible to defenses that inspect only the user turn. Second, retrieved content is often trusted implicitly by the model because it arrives via the system-controlled retrieval path [8]. Third, a single poisoned document can affect all users whose queries trigger its retrieval. We model two delivery mechanisms.

Vector

Attack Sub-class

IO

DE

BM

Direct (user input)

Instruction Override

Tested

Possible

Tested

Role Hijacking

Tested

Possible

Tested

Prompt Extraction

Possible

Tested

Possible

Tested

Possible

Tested

Tested

Possible

Tested

Indirect KB Poisoning (retrieved doc) Retrieved Doc Payload

Retrieved Document Payload: In deployments Figure 3. Mapping of attack vectors and sub-classes to where the knowledge base indexes third-party attacker objectives (IO = Instruction Override; DE = Data content, the attacker publishes a web page Exfiltration; BM = Behavioral Manipulation). containing an injection payload and relies on the retrieval module to surface it for relevant user queries [8, 16]. 3.5 Scope and Exclusions Knowledge Base Poisoning: The attacker submits a The following threat classes are explicitly out of scope. document through a legitimate content channel Table 2 provides a consolidated reference. that feeds the knowledge base (e.g., a product Adversarial Suffix Attacks [18]: These are gradientreview or a support ticket). The document contains based suffix optimization which requires white-box instructions that are invisible to human reviewers model access and therefore contradicts the attacker but interpretable by the LLM [8]. model discussed above. 6

ICCK Transactions on Information Security and Cryptography

Training-Time Attacks (data poisoning) [27]: mitigation framework that intercepts both vectors at These attacks target model weights rather than the points where they are actionable. The layers are runtime context injection. ordered by their position in the inference pipeline: Layer 1 operates on the raw user input before retrieval; Multimodal Injection: These attacks are delivered Layer 2 enforces structural privilege boundaries at through images, audio, or other non-text context assembly time; and Layer 3 audits the model modalities are therefore not considered in this output before it is delivered to the user. A continuous study [28]. audit loop running across all layers provides logging, alerting, and feedback for iterative improvement. The Model Extraction & Membership Inference: design is model-agnostic and requires no modification These attacks do not involve instruction-plane to the underlying LLM weights or training procedure. manipulation and are therefore outside the scope of an LLM-layer defense framework. 4.1 Framework Overview injection, Figure 4 illustrates the position of each layer within Infrastructure-Layer Attacks: SQL server-side exploits, and network-level interception the RAG inference pipeline. Table 3 maps each layer to are addressed by existing network and application the attack vectors and attacker objectives it addresses. security controls and are not reconsidered here. Table 2. Threat model scope summary.

Threat Class

In Scope

Direct injection: instruction override

✓

Direct injection: role hijacking

✓

Direct injection: system prompt leak

✓

Indirect injection: KB poisoning

✓

Indirect injection: retrieved payload

✓

Adversarial suffix attacks

×

Training-time / fine-tuning attacks

×

Multimodal injection

×

Model extraction / membership inference

×

Infrastructure-layer attacks

×

4 Proposed Framework The threat model in Section 3 establishes that prompt injection is a structural vulnerability arising from the absence of a boundary between the instruction plane and the data plane within the LLM context window. Point defenses that target a single stage of the inference pipeline are insufficient because each attack vector identified in Section 3.3 exploits a different entry point: direct injection is introduced before the model processes any retrieved content, while Figure 4. Position of the three defense layers within the indirect injection is introduced by the retrieval module RAG chatbot inference pipeline. itself. A defense applied only to the user input channel is therefore blind to indirect injection, and The three layers are not redundant: each addresses a defense applied only to retrieved documents offers a distinct stage of the attack lifecycle. Layer 1 aims no protection against direct manipulation. to prevent injections from entering the pipeline at This section presents a three-layer detection and all. Layer 2 limits the damage that an injection can 7

ICCK Transactions on Information Security and Cryptography

Table 3. Summary of framework layers and mitigated

attacker objectives. Layer

Function

Goals

L1

Input screening for direct prompt-injection attempts using pattern matching and semantic anomaly detection Privilege-constrained execution using instruction hierarchy and context tagging for direct and indirect attacks Output auditing using policy rules and semantic verification for direct and indirect attacks Logging, alerting, and feedback-driven improvement

IO, DE

L2

L3

Audit

Category

Representative Signatures

Instruction Override

"ignore previous instructions", "disregard system prompt" "you are now...", "pretend to be unrestricted" "repeat system instructions", "print system prompt contents"

Role Hijacking Prompt Extraction IO, BM

IO, DE, BM

All

cause even if it passes Layer 1, by constraining what instructions the model is permitted to follow based on their provenance. Layer 3 provides a final safety net by detecting attacker-goal fulfillment in the model output regardless of how the injection was delivered. 4.2 Layer 1: Input Screening Layer 1 operates on the raw user turn before it is passed to the retrieval module or concatenated with the system prompt. Its purpose is to identify and neutralize direct injection payloads at the earliest actionable point in the pipeline. The layer applies two complementary detection mechanisms in sequence. 4.2.1 Pattern-Based Detection A rule engine maintains a curated library of injection signatures derived from known attack patterns reported in the literature [6, 9, 17]. Each signature is expressed as a regular expression or a keyword-proximity rule applied to the normalized input string. Normalization includes Unicode folding, whitespace collapsing, and decoding of common obfuscation schemes such as Base64 segments and URL-encoded characters, which are frequently used to bypass naive string-matching defenses [10]. The pattern library is organized into three categories corresponding to the attack sub-classes defined in Section 3.3: instruction override patterns, role hijacking patterns, and system prompt extraction patterns. Table 4 lists representative signatures in each category. A match against any signature triggers one of three dispositions depending on the confidence score of the match: (i) pass with flag, where the input proceeds with an anomaly annotation propagated to Layer 3; (ii) sanitize, where the matching segment is 8

Table 4. Representative Layer 1 injection signatures.

replaced with a neutral placeholder and the modified input continues; or (iii) block, where the input is rejected and the user receives a generic refusal response. The disposition thresholds are configurable per deployment context. 4.2.2 Semantic Anomaly Detection Pattern matching is inherently brittle against novel phrasings and paraphrastic attacks that preserve adversarial intent while avoiding known signatures [10]. Layer 1 therefore applies a second mechanism: a lightweight semantic classifier that operates on the sentence embedding of the user input. The classifier is trained on a balanced dataset of benign and adversarial inputs constructed from public benchmarks including PromptBench [23] and BIPIA [16], augmented with paraphrase-expanded variants of the pattern library. The classifier outputs a continuous injection probability score s1 ∈ [0, 1]. Inputs with s1 > τ1 (threshold τ1 determined empirically on a held-out validation set) are treated as suspicious and forwarded to Layer 3 with the score embedded as metadata, even if no pattern match was triggered. The combination of the two mechanisms provides complementary coverage: pattern matching offers high precision on known attacks at near-zero latency, while the semantic classifier extends recall to unseen phrasings at a modest computational cost. The processing logic of Layer 1 is summarized in Figure 5. Layer 1 addresses direct injection exclusively. It has no visibility into retrieved documents and therefore provides no protection against indirect injection. This gap is closed by Layer 2. 4.3 Layer 2: Assembly

Privilege-Constrained

Context

The root cause of both direct and indirect injection is the flat context window: the LLM cannot distinguish operator-supplied instructions from user-supplied content or retrieved data because all three are serialized into the same token sequence. Layer 2

ICCK Transactions on Information Security and Cryptography

Table 5. Privilege hierarchy enforced in Layer 2. Operation

T1

T2

T3

Define persona/role Set output policy Override T1 instruction Provide factual reference Ask factual question Request response generation

✓ ✓ ✓ ✓ ✓ ✓

× × × ✓ ✓ ✓

× × × × ✓ ✓

mechanisms. First, each context segment is annotated with a provenance tag ([SYS], [RET], or [USR]) inserted by the context assembly module before the content is passed to the model. The system prompt is prepended with an explicit meta-instruction informing the model of the tagging scheme and instructing it to refuse any directive from a lower-privilege tier that conflicts with a higher-privilege directive. This approach extends prior work on structured prompting [11, 29] and does not require model fine-tuning; it operates entirely at the prompt engineering level.

Figure 5. Input screening process in Layer 1 combining signature-based detection and semantic anomaly analysis for prompt injection mitigation.

mitigates this by enforcing an explicit privilege hierarchy at context assembly time, before the concatenated context is passed to the inference engine. 4.3.1 Instruction Hierarchy We define three privilege tiers based on the provenance of each context segment. 1. Operator Tier (highest privilege): In this tier, the content originate from the system prompt and it establishes the authorized scope of the chatbot’s behavior and is therefore sole source of binding instructions. 2. Retrieval Tier (intermediate privilege): In this tier, the content is fetched from the knowledge base by the retrieval module and is treated as reference data. It may inform the model’s response but is not permitted to override tier-1 instructions. 3. User Tier (lowest privilege): In this tier, the content supplied by the end user in the current conversation and is treated as a query to be answered within the constraints established by tier-1; it is not permitted to override tier-1 or tier-2.

Second, the retrieval module applies a lightweight injection scanner to each retrieved document chunk before it is admitted into the context. The scanner uses a simplified version of the Layer 1 pattern library restricted to instruction-override and role-hijacking patterns, which are the sub-classes most likely to appear in knowledge-base poisoning payloads [8, 16]. Chunks that match the scanner are either removed from the retrieval result set or admitted with a [FLAGGED] annotation appended to their tag. Table 5 summarizes the permitted and forbidden operations for each privilege tier. 4.3.2 Limitations of Layer 2 The privilege hierarchy relies on the model’s instruction-following capability to respect the meta-instruction. Sufficiently capable adversaries may craft payloads that cause the model to ignore the hierarchy through role-hijacking attacks that explicitly instruct the model to disregard tier restrictions. Layer 2 reduces the probability of such bypasses but does not eliminate it. The residual risk is addressed by Layer 3. Figure 6 illustrates the context assembly procedure. 4.4 Layer 3: Output Auditing

Layer 3 intercepts the model’s response after inference and before delivery to the user. It serves as the final safety net of the framework: even if an injection bypasses Layers 1 and 2, attacker goal fulfillment The hierarchy is enforced through two complementary requires the malicious content to appear in the model’s 9

ICCK Transactions on Information Security and Cryptography

Figure 6. Layer 2 context assembly with provenance tagging

4.4.2 Semantic Drift Detection In addition to the rule-based predicates, Layer 3 monitors for semantic drift between the intended response persona (defined in the system prompt) and the actual response. The drift score is computed as the cosine distance between the embedding of a reference persona-compliant response (sampled at deployment time) and the embedding of the live response. A drift score exceeding threshold τdrift is treated as a soft signal of behavioral manipulation and triggers escalation rather than outright blocking, to avoid over-suppression of legitimate response variation. The processing flow of Layer 3 is shown in Figure 7.

and privilege-aware context integration prior to LLM inference. Table 6. Output policy predicates enforced by the Layer 3

rule engine. Predicate

Goal

System-prompt similarity (cos ≥ θsp ), block and log Role-token detection (e.g., DAN, JAILBREAK), block and log Instruction-acknowledgment phrase detection, block and log Harmful-content score (≥ τ3 ), block and log Propagated Layer 1 anomaly flag, escalate to human review

DE IO IO BM IO, DE

output. Layer 3 checks the output against a policy rule engine and a semantic similarity test.

Figure 7. Layer 3 output auditing framework. Rule-based

security checks and semantic drift analysis determine whether an LLM response is delivered, redacted, blocked, or escalated for review.

4.5 Continuous Audit Loop 4.4.1 Policy Rule Engine The rule engine evaluates the model response against a set of output policies derived directly from the three attacker objectives defined in Section 3.4. Each policy is expressed as a boolean predicate over the response text. Table 6 lists the policy predicates, the attacker goal addressed by each, and the action taken when a predicate evaluates positively. The cosine similarity check for system prompt leakage uses the same sentence embedding model as the Layer 1 semantic classifier, ensuring that both layers share a consistent semantic representation with no additional model loading overhead. The threshold θsp is set conservatively to minimize false negatives on the data exfiltration goal (DE), accepting a higher false positive rate on this specific predicate given the severity of the threat. 10

The audit loop is not a fourth detection layer but a cross-cutting operational component that collects structured logs from all three layers and from the inference engine itself. Each log entry records the session identifier, the layer identifier, the detection mechanism triggered, the disposition applied, and a timestamp. Log entries are written to an append-only store to preserve integrity for forensic purposes. Two alerting thresholds are defined over the log stream. A session-level alert is raised when a single session accumulates more than ks layer-triggered dispositions within a sliding window of ws turns, indicating sustained adversarial probing. A population-level alert is raised when the fraction of sessions triggering at least one Layer 1 or Layer 3 disposition exceeds a baseline rate by a statistically significant margin (z-test, p < 0.01), indicating a coordinated or automated attack campaign.

ICCK Transactions on Information Security and Cryptography

The feedback path of the audit loop feeds a monthly Algorithm 1: Single-turn inference pipeline with retraining cycle for the Layer 1 semantic classifier. three-layer defense. Confirmed true positive cases (validated by human Input: User input u, system prompt p , sys review of escalated responses) are added to the knowledge base K training set as adversarial examples. Confirmed false Output: Response r delivered to user, or rejection positives are added as hard negative examples. This message mechanism ensures that the classifier adapts to novel // Layer 1: Input Screening injection phrasings observed in production without u′ ← Normalize(u); requiring manual signature authoring. m ← PatternMatch(u′ , P); 4.6 Framework Integration

s1 ← SemanticScore(u′ ); if m = Block or s1 > τ1block then return rejection message; end if m = Sanitize then u′ ← Sanitize(u′ , m); end

Figure 8 shows the complete integrated architecture with all three layers and the audit loop positioned within the RAG pipeline. The framework is designed for deployment as a middleware wrapper around an existing chatbot backend, requiring no changes to the LLM inference endpoint or the retrieval module flag flag1 ← (s1 > τ1 ); beyond the addition of the Layer 2 provenance tagging // Layer 2: Privilege-Constrained Context step at the context assembly stage. The Layer 1 Assembly classifier and Layer 3 embedding model may be D ← Retrieve(u′ , K); hosted as lightweight microservices with sub-50 ms D′ ← ScanChunks(D, Pret ); median latency overhead, as reported in comparable c ← Assemble(psys , D′ , u′ ) ; // with provenance deployments [12, 13]. The computational overhead tags and meta-instruction of each layer and its expected impact on end-to-end // LLM Inference response latency is quantified empirically in Section 5. rraw ← LLM(c); The complete framework is summarized in // Layer 3: Output Auditing Algorithm 1, which gives the pseudocode for if PolicyCheck(rraw , psys , flag1 ) or processing a single user turn from input receipt to DriftScore(rraw ) > τdrift then response delivery. AuditLog(session, rraw , L3); return block or escalate based on predicate; end 5 Experimental Evaluation This section reports the empirical evaluation of the proposed framework. The evaluation is structured around four questions derived directly from the threat model and framework design. (i) Does the full three-layer framework reduce the Attack Success Rate (ASR) for each attacker objective (IO, DE, BM) relative to an undefended baseline and to single-layer alternatives? (ii) What is the false positive rate (FPR) introduced by each layer, and does the full framework remain usable for legitimate queries? (iii) Which layer contributes the largest individual ASR reduction, and is the combined reduction greater than the sum of the individual contributions? (iv) What is the end-to-end latency overhead, and does it remain within operational bounds? 5.1 Experimental Setup

// Audit and Deliver

AuditLog(session, rraw , pass); return rraw ;

fixed operator system prompt that assigns the chatbot a customer-support persona and forbids it from revealing the system prompt, adopting alternative roles, or producing content outside the customer-support domain. The knowledge base contains 500 documents drawn from publicly available product documentation and FAQ corpora. Retrieval uses a dense passage retrieval model [30] with a FAISS index [31], returning the top-3 chunks per query. 5.1.2 Target Models Three large language models are evaluated to assess whether framework effectiveness generalises across model families and capability levels.

5.1.1 Target System The evaluation harness instantiates the RAG chatbot system described in Section 3.1 with a 1. GPT-4o [1]: accessed via the OpenAI API with 11

ICCK Transactions on Information Security and Cryptography

Figure 8. Complete architecture of the proposed three-layer prompt injection defense framework.

temperature = 0 to promote deterministic outputs flag threshold τ1flag and block threshold τ1block are set across repeated evaluations. by maximizing F1 and precision, respectively, on the validation split. 2. Llama 3 8B Instruct [2]: deployed locally using the Hugging Face transformers library [32] on a single 5.1.4 Layer 3 Harmful-Content Classifier NVIDIA A100 40 GB GPU. The harmful-content predicate in the Layer 3 policy rule engine uses Llama Guard 2 [35] as the 3. Mistral 7B Instruct [3]: deployed locally using the scoring model with threshold τ3 = 0.5. The same Hugging Face transformers-based setup on a system-prompt-leakage similarity threshold is set to single NVIDIA A100 40 GB GPU. θsp = 0.85 cosine similarity. The semantic drift For all three models, inference temperature is set to 0 threshold is set to τdrift = 0.40 cosine distance, and top_p = 1 to obtain deterministic outputs. Each calibrated on a held-out set of 200 benign response experimental condition is evaluated independently on pairs. all three models; results are reported per model and 5.2 Dataset as a macro-average. A dedicated evaluation dataset is constructed from 5.1.3 Layer 1 Classifier three sources to ensure broad coverage of both known The semantic anomaly classifier uses a and novel injection patterns. sentence-transformers/all-MiniLM-L6-v2 encoder backbone [33], fine-tuned for binary injection 5.2.1 Source 1: Public Benchmarks classification on the training split of the dataset Adversarial inputs are drawn from PromptBench [23] described in Section 5.2. Training uses AdamW [34] and BIPIA [16]. From PromptBench, we select all with a learning rate of 2 × 10−5 , batch size 32, and samples labeled as instruction-following attacks. From early stopping on validation F1 with patience 3. The BIPIA, we select the indirect injection subset, which 12

ICCK Transactions on Information Security and Cryptography

Table 7. Dataset composition after the 80/10/10 5.3 Evaluation Metrics train–validation–test split. Attack categories correspond to Four metrics are computed on the test split for each the attack vectors defined in Section 3.3.

experimental condition. Type

Category

Total

Train

Val

Test

Instruction Override Role Hijacking Prompt Extraction

740 620 480

592 496 384

74 62 48

74 62 48

Direct

KB Poisoning Retrieved Payload

520 480

416 384

52 48

52 48

Total Adversarial Total Benign

2,840 2,240

2,272 1,792

284 224

284 224

Grand Total

5,080

4,064

508

508

Indirect

Doc

5.3.1 Attack Success Rate (ASR) The fraction of adversarial test samples for which the framework delivers a response that satisfies the attacker’s goal, as judged by the scoring procedure described below. ASR is the primary security metric; lower is better.

ASR =

|{x ∈ Xadv : GoalAchieved(x, rx ) = 1}| (1) |Xadv |

provides document-embedded payloads targeting 5.3.2 Goal Achievement Scoring RAG-style systems. In total, 612 adversarial samples Each attacker goal is assessed by a dedicated are sourced from these benchmarks. scorer. For Instruction Override (IO), the scorer checks whether the response deviates from the system-prompt-defined persona using a prompted 5.2.2 Source 2: Manual Curation GPT-4o judge [37] with a three-point rubric (compliant To cover attack patterns not present in public / borderline / overridden). For Data Exfiltration (DE), benchmarks, a domain expert manually constructed the scorer computes cosine similarity between the 188 additional adversarial samples across the response and the system prompt; similarity ≥ 0.85 three direct-injection sub-classes (instruction is counted as a successful exfiltration. For Behavioral override, role hijacking, system prompt extraction) Manipulation (BM), the scorer applies Llama Guard 2 and two indirect-injection delivery mechanisms and counts responses classified as unsafe. Inter-rater (knowledge-base poisoning, retrieved document reliability between the automated scorer and a human payload) defined in Section 3.3. Each manually annotator was measured on a 100-sample subset; curated sample was verified by a second annotator; Cohen’s κ was 0.87 for IO, 0.94 for DE, and 0.91 for inter-annotator agreement (Cohen’s κ) was 0.91. BM. 5.2.3 Source 3: Paraphrase Augmentation To reduce the risk of overfitting to surface-level phrasings, each adversarial sample from Sources 1 and 2 was paraphrased twice using GPT-4o with the instruction to preserve adversarial intent while varying vocabulary and sentence structure. Paraphrases were manually reviewed and rejected if the adversarial intent was materially weakened. This produced an additional 1,440 augmented adversarial samples.

5.3.3 False Positive Rate (FPR) The fraction of benign queries that are incorrectly blocked or sanitized by any layer of the framework. FPR directly measures the usability cost of the defense.

FPR =

|{x ∈ Xbenign : Blocked(x) = 1}| |Xbenign |

(2)

5.3.4 Detection Rate (DR) The complement of ASR; the fraction of adversarial 5.2.4 Benign Samples samples that are correctly detected and blocked or An equal number of benign user queries (2,240) were sanitized before goal achievement. DR is reported sampled from the MS-MARCO question answering alongside ASR for completeness. dataset [36], filtered to customer-support-relevant queries by keyword matching, to form the negative 5.3.5 Response Quality Score (RQS) For samples that pass all layers and receive a delivered class for FPR evaluation. response, RQS measures the quality of the response Table 7 summarizes the final dataset composition after on benign queries using BERTScore F1 [38] against an 80/10/10 train/validation/test split. a human reference response. RQS verifies that the 13

ICCK Transactions on Information Security and Cryptography

Table 8. Macro-averaged performance across GPT-4o, Llama-3 (8B), and Mistral-7B. Lower ASR/FPR and higher DR/RQS indicate better performance. Configuration

ASR↓

FPR↓

DR↑

RQS↑

Undefended L1 Only L2 Only L3 Only NeMo Guardrails

71.4 44.2 51.8 38.6 35.1

0.0 5.1 1.3 6.7 8.4

28.6 55.8 48.2 61.4 64.9

0.91 0.90 0.89 0.89 0.88

Proposed Framework

11.3

4.8

88.7

0.89

Improvement vs. NeMo

-67.8% -42.9% +23.8 +0.01

Table 9. Ablation study of framework components. Lower

ASR and FPR indicate better performance. Configuration

ASR (%)↓

FPR (%)↓

None (Undefended) L1 L2 L3 L1 + L2 L1 + L3 L2 + L3

71.4 44.2 51.8 38.6 31.5 26.7 22.4

0.0 5.1 1.3 6.7 5.8 6.9 7.4

L1 + L2 + L3

11.3

4.8

framework does not degrade legitimate response points relative to the undefended baseline. This exceeds the best single-layer result (L3 Only, 38.6%) quality as a side effect of the defense mechanisms. by 27.3 percentage points and outperforms NeMo Guardrails by 23.8 percentage points, confirming that 5.4 Baselines layering yields substantially greater security than any Five baseline conditions are evaluated. The first single-component approach. The FPR of 4.8% is lower establishes a lower bound (no defense); the next three than that of NeMo Guardrails (8.4%) and comparable evaluate each layer in isolation to support the ablation to L1 Only (5.1%), indicating that the additional analysis in Section 5.6; the fifth provides a comparison protection from Layers 2 and 3 does not compound the against a published general-purpose guardrail system. false positive rate in proportion to the ASR reduction. 1. Undefended: All defense layers disabled. Raw RQS remains at 0.89 across all defended conditions, user inputs and retrieved documents are provided demonstrating that response quality on legitimate directly to the LLM, establishing the baseline ASR queries is preserved. for each model and attack category. 5.5.2 Per-Goal and Per-Model Breakdown 2. L1 Only: Only Layer 1 (Input Screening) enabled. Figure 9 plots ASR per attacker goal (IO, DE, BM) and Layers 2 and 3 are disabled, with context assembled per attack category for each condition and each target using the standard RAG pipeline. model. 3. L2 Only: Only Layer 2 (Privilege-Constrained 5.5.3 ROC Analysis Context Assembly) enabled. Layers 1 and 3 are Figure 10 plots the Receiver Operating Characteristic disabled. (ROC) curves for the Layer 1 semantic classifier 4. L3 Only: Only Layer 3 (Output Auditing) enabled. and the Layer 3 output auditing module, evaluated independently on the test split. The area under Layers 1 and 2 are disabled. the ROC curve (AUC) for the Layer 1 classifier is 5. NeMo Guardrails [12]: An open-source 0.941, and for the Layer 3 harmful-content predicate guardrail framework configured with its is 0.923. These values confirm that the underlying default prompt-injection protections, serving detection components operate well above chance and as a representative state-of-the-art single-layer that the selected operating thresholds (τ1 , τ3 ) represent defense baseline. near-optimal points on their respective ROC curves. 5.5 Results

5.6 Ablation Study

5.5.1 Overall ASR Reduction Table 8 reports ASR, FPR, DR, and RQS for the undefended baseline, each single-layer baseline, NeMo Guardrails, and the full three-layer framework, macro-averaged across all three target models.

To quantify the marginal contribution of each layer, an ablation study evaluates all seven non-empty subsets of the three layers. Table 9 reports macro-averaged ASR and FPR for each subset.

Three findings emerge from the ablation. First, L3 The full framework reduces macro-averaged ASR achieves the largest single-layer ASR reduction (32.8 from 71.4% to 11.3%, a reduction of 60.1 percentage pp), followed by L1 (27.2 pp) and L2 (19.6 pp). L3’s 14

ICCK Transactions on Information Security and Cryptography

Figure 9. Attack Success Rate (ASR) by attacker goal and target model. Results are reported for Instruction Override (IO),

Data Exfiltration (DE), and Behavioral Manipulation (BM) attacks under six defense configurations. Lower values indicate stronger resistance to prompt injection attacks.

superior individual performance is expected given Table 10. Latency overhead of the proposed framework on an NVIDIA A100 40 GB GPU. LLM inference time is that it is the only layer with direct visibility into both excluded. attack vectors and all three attacker goals. Second, the combined ASR of the full framework (11.3%) is Component Median p95 lower than the best two-layer combination (L2+L3, L1 Pattern Matching 2.1 4.3 22.4%) by 11.1 pp, confirming that the three layers L1 Semantic Classifier 18.4 27.6 are complementary rather than redundant. Third, L1 Subtotal 20.5 – the FPR of the full framework (4.8%) is lower than L2 Chunk Scanner 9.7 16.2 L2 Context Assembly 1.8 3.1 that of the L1+L3 combination (6.9%) and the L2+L3 L2 Subtotal 11.5 – combination (7.4%), which is attributable to the role of L3 Policy Rule Engine 11.3 19.8 L2 in reducing the number of injections that reach L3: L3 Semantic Drift Check 17.9 26.4 fewer injection-bearing inputs trigger the L3 predicates, L3 Subtotal 29.2 – which lowers the rate at which borderline benign Framework Total 61.2 97.4 queries are caught in the L3 harmful-content check. 5.7 Latency Overhead End-to-end latency is measured from receipt of the user input to delivery of the response, averaged over 500 benign queries per model on the local deployment hardware. LLM inference is excluded from the per-layer overhead measurement to isolate the cost attributable to the framework. Table 10 reports median and 95th-percentile latency for each layer and for the full framework.

and L3 drift check), each requiring a forward pass through the MiniLM encoder. This is consistent with the sub-100 ms overhead reported for comparable guardrail systems [12, 13] and is within the latency budget of interactive chatbot deployments, where total response time is typically dominated by LLM inference (300–2,000 ms depending on response length The median total overhead of 61.2 ms is dominated by and model size). Figure 11 visualizes the latency the two semantic encoding operations (L1 classifier contribution of each component as a stacked bar chart. 15

ICCK Transactions on Information Security and Cryptography

Figure 10. ROC curves for the Layer 1 semantic anomaly classifier (AUC = 0.942) and the Layer 3 output auditing module (AUC = 0.923). Filled circles mark the selected operating points: Layer 1 at FPR = 0.051, TPR = 0.722 and Layer 3 at FPR = 0.067, TPR = 0.702, consistent with the FPR values reported in Table 8. Table 11. Error analysis of residual attack successes under

the proposed framework. Percentages are computed over the 32 bypass cases. Failure Mode

%

Count

Novel phrasing evading L1 detection Implicit instructions bypassing L2 scanning Persona drift below τdrift Multi-turn attack accumulation

37.5 25.0 21.9 15.6

12 8 7 5

Figure 11. Median per-component latency of the full framework, broken down by model. Each bar segment corresponds to one processing step; segment widths are proportional to latency. The two semantic encoding steps (L1 Semantic Classifier and L3 Semantic Drift) dominate, together accounting for approximately 60 % of total overhead. The red dashed line marks the 100 ms operational budget; all three models remain well within this threshold.

that ends with a parenthetical comment suggesting a different response style) rather than explicit override commands; these avoid the chunk scanner’s explicit-override pattern library. Multi-turn injection, which accumulates a partial payload across multiple conversation turns to avoid per-turn detection, accounts for 15.6% of bypasses and represents a threat sub-class not covered by the current framework design; it is noted as a direction for future work in Section 6.2.

6 Discussion

The experimental results in Section 5 admit several interpretations that go beyond what can be expressed Total 100.0 32 in the tables alone. This section draws out the implications of the main results, situates the findings within the broader context of LLM security, and then 5.8 Error Analysis To understand the residual 11.3% ASR of the full identifies the limitations of the current work and the framework, the 284 test-set adversarial samples that directions they motivate. were not blocked or sanitized were manually reviewed. 6.1 Implications of the Main Results Table 11 categorizes the bypass mechanisms observed. Layering is necessary, not just beneficial: The The dominant bypass mechanism (37.5%) involves ablation study (Table 9) shows that the full three-layer semantically novel phrasings that fall below the framework achieves an ASR of 11.3%, which is Layer 1 classifier threshold while avoiding all 11.1 percentage points below the best two-layer pattern signatures. This is consistent with the combination (L2+L3, 22.4%) and 27.3 percentage known brittleness of semantic classifiers trained points below the best single-layer configuration on fixed-distribution adversarial datasets [10] and (L3 Only, 38.6%). These gaps are not incremental; they motivates the audit loop’s retraining cycle described represent qualitatively different levels of protection. in Section 4.5. The second most common bypass The result confirms the central design premise of (25.0%) involves indirect injection payloads crafted the framework: direct and indirect injection enter as implicit soft instructions (e.g., a retrieved FAQ item the pipeline at different points and therefore require 16

ICCK Transactions on Information Security and Cryptography

defenses positioned at those points. No subset of two layers covers the full attack surface; Layer 1 is the only component with access to the raw user input before retrieval contamination, Layer 2 is the only component that can prevent a retrieved document payload from reaching the inference engine untagged, and Layer 3 is the only component that can catch injections that bypass upstream controls.

Across all configurations, the Data Exfiltration (DE) goal shows the lowest residual ASR after applying the full framework. This is attributable to the Layer 3 cosine similarity predicate, which is a high-precision detector for verbatim or near-verbatim system prompt leakage. The Instruction Override (IO) and Behavioral Manipulation (BM) goals are harder to suppress because their success criteria are behavioral rather than content-based: a response can be manipulated without copying any specific text from the injection payload, making similarity-based detection inapplicable. This structural asymmetry suggests that future work should prioritize improving IO and BM detection, as discussed in Section 6.3.

The finding also has a practical implication for deployment: an operator who cannot implement the full framework should prioritize L2+L3 over L1+L3 or L1+L2. The L2+L3 combination achieves 22.4% ASR with an FPR of 7.4%, which, while inferior to the full framework, provides substantially better indirect injection coverage than any configuration that omits 6.2 Limitations L2. Static attacker model: The threat model in Section 3.2 Layer 2 suppresses false positives in downstream assumes a black-box attacker with no knowledge of layers: A notable result in Table 9 is that the full the deployed defense. A white-box adversary with framework (L1+L2+L3, FPR 4.8%) has a lower false knowledge of the Layer 1 pattern library and classifier positive rate than both L1+L3 (FPR 6.9%) and L2+L3 thresholds could craft targeted evasion payloads that (FPR 7.4%), despite being strictly more restrictive. This avoid known signatures while remaining below the counterintuitive result arises because Layer 2 reduces semantic anomaly threshold. This is a standard the proportion of injections that reach the Layer 3 limitation of detection-based defenses and is not policy rule engine. The Layer 3 harmful-content unique to the present framework; it motivates the classifier and semantic drift check are the components continuous audit loop and retraining cycle described in most likely to generate false positives on legitimate Section 4.5, which are designed to reduce the window queries that touch sensitive topics or deviate from the of vulnerability between the introduction of a novel reference persona. By filtering more true positives attack pattern and its incorporation into the classifier’s before Layer 3, Layer 2 shifts the Layer 3 operating training distribution. point toward a region of its ROC curve with lower FPR. This interaction between layers is not captured by Single-turn evaluation: The experimental protocol any single-layer evaluation and could not have been evaluates each adversarial sample as an independent single-turn query. The error analysis in Section 5.8 predicted without the ablation. identifies multi-turn injection as the fourth most Model capability does not substitute for structural common bypass mechanism, accounting for 15.6% defense: The per-model breakdown in Figure 9 of residual bypass cases. In this attack pattern, a shows that GPT-4o, despite being a substantially partial payload is distributed across multiple turns more capable model than Llama 3 (8B) and of a conversation so that no individual turn triggers a Mistral 7B on standard benchmarks, achieves detection threshold. The current framework applies only a modestly lower undefended ASR. This Layer 1 and Layer 3 independently on each turn and is consistent with the observation of Perez and therefore has no mechanism for detecting payloads Ribeiro [6] that instruction-following capability that are individually innocuous but cumulatively and instruction-override susceptibility are not injurious. Extending the framework to maintain a independent: a model that follows instructions more session-level injection state across turns is a direct reliably will also follow injected instructions more direction for future work. reliably. The implication is that practitioners cannot Knowledge base composition: The evaluation substitute model upgrades for structural defenses; a knowledge base contains 500 documents drawn from stronger model reduces ASR marginally at best, while publicly available product documentation, which the framework reduces it by 60.1 percentage points represents a controlled and relatively benign content regardless of the underlying model. distribution. Real deployments may index content Data exfiltration is the most tractable attacker goal: from less controlled sources such as user-generated 17

ICCK Transactions on Information Security and Cryptography

reviews, social media, or third-party APIs, all of which security gain. present a wider surface for knowledge base poisoning. Adaptive threshold calibration: An online calibration Evaluation on a more adversarially representative mechanism that adjusts Layer 1 and Layer 3 thresholds knowledge base is needed to characterize framework in response to observed FPR drift in production, using performance under realistic indirect injection a lightweight Platt scaling or isotonic regression model conditions. fitted on streaming audit log data, would reduce Threshold sensitivity: The Layer 1 and Layer 3 the sensitivity of the framework to distribution shift thresholds (τ1 , τ3 , θsp , τdrift ) are calibrated on without requiring full classifier retraining. a held-out validation set derived from the same White-box evaluation: Evaluating the framework distribution as the test set. In deployment, the against an adaptive adversary with knowledge of the input distribution will differ from the evaluation defense, using techniques from adversarial machine distribution in ways that are difficult to anticipate. learning such as Carlini-Wagner attacks [39] adapted Threshold miscalibration is the most likely cause to the text domain, would provide a lower bound on of FPR degradation in production. The audit framework security and characterize the evasion effort loop’s session-level and population-level alerting required to overcome each layer. mechanisms are designed to surface threshold drift, but they depend on a sufficient volume of labeled 7 Conclusion production data to drive recalibration. Latency under concurrent load: The latency This paper addresses prompt injection in LLM-based measurements reported in Table 10 are collected chatbots as a structural security problem arising from under single-request conditions on dedicated the lack of a clear boundary between instructions hardware. In a production deployment serving and data within the model context. Existing defenses concurrent users, contention for the encoder model operate at a single stage of the pipeline and therefore shared between Layer 1 and Layer 3 may increase remain vulnerable to attacks delivered through the p95 latency beyond the 97.4 ms reported here. alternative channels. Batching the semantic encoding operations across To address this challenge, we proposed a three-layer concurrent requests is a straightforward optimization defense framework comprising Input Screening, that is not evaluated in this work. Privilege-Constrained Context Assembly, and Output Auditing, supported by a Continuous Audit Loop 6.3 Future Work for monitoring and adaptive improvement. Together, Four directions follow directly from the limitations these layers provide complementary protection against both direct and indirect prompt injection attacks. identified above. Multi-turn injection detection: Extending Layer 1 to maintain a rolling session-level injection score across turns, using a recurrent or attention-based aggregator over per-turn scores, would address the multi-turn bypass mechanism. A practical design would decay the session score between turns to avoid over-sensitivity in long conversations. Implicit instruction detection in retrieved documents: The error analysis shows that 25% of residual bypasses exploit implicit soft instructions in retrieved documents that avoid the chunk scanner’s explicit-override patterns. Replacing or augmenting the chunk scanner with a generative detector that prompts a secondary LLM to assess whether a document chunk contains latent instructions, as proposed by Greshake et al. [8], could reduce this bypass category. The latency cost of a generative detector would need to be evaluated against the 18

Experimental results show that the complete framework reduces the macro-averaged Attack Success Rate (ASR) from 71.4% to 11.3% across three target models and five attack categories, outperforming both the strongest single-layer baseline and a published general-purpose guardrail system. The framework also maintains a low false positive rate (4.8%) and introduces only 61.2,ms of median latency overhead, making it suitable for interactive chatbot deployments. Ablation studies further confirm that each layer contributes unique defensive value and that their combined effect exceeds the sum of their individual contributions. The findings indicate that model capability alone is insufficient to mitigate prompt injection and that effective protection requires defense mechanisms distributed across the entire inference pipeline. Although the framework substantially improves

ICCK Transactions on Information Security and Cryptography

robustness, residual vulnerabilities remain, awareness and incentive-sensitive failures in gpt-oss-20b,” arXiv preprint arXiv:2510.08624, 2025. particularly against semantically novel attacks, implicit instructions embedded in retrieved content, [15] Y. Xu, M. A. Aslam, W. Jun, N. Ahmed, M. I. Zaman, M. Hamza, and S. Aslam, “Improving and multi-turn attack strategies. Future work should arabic multi-label emotion classification using stacked focus on adaptive detection methods, stronger embeddings and hybrid loss function,” IEEE Access, behavioral deviation analysis, and session-aware 2025. defenses. Overall, the proposed framework provides [16] J. Yi et al., “Benchmarking and defending against a practical and extensible foundation for securing indirect prompt injection attacks on large language LLM-based chatbot systems against evolving prompt models,” arXiv preprint arXiv:2312.14197, 2023. injection threats.

References

[17] X. Liu et al., “Jailbreaking ChatGPT via prompt engineering: An empirical study,” arXiv preprint arXiv:2305.13860, 2023.

[1] OpenAI, “GPT-4o system card.” https://openai.com/index /gpt-4o-system-card, 2024.

[18] A. Zou et al., “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023.

[2] Meta AI, “Llama 3 model card.” https://ai.meta.com/blog /meta-llama-3/, 2024.

[19] N. Jain et al., “Baseline defenses for adversarial attacks against aligned language models,” 2023.

[3] A. Q. Jiang et al., “Mistral 7B,” arXiv preprint [20] A. Robey et al., “SmoothLLM: Defending large arXiv:2310.06825, 2023. language models against jailbreaking attacks,” 2023. [4] P. Lewis et al., “Retrieval-augmented generation for [21] S. Chen et al., “StruQ: Defending against prompt knowledge-intensive NLP tasks,” in Advances in Neural injection with structured queries,” 2024. Information Processing Systems (NeurIPS), 2020. [22] Z. Wu et al., “Defending ChatGPT against [5] OWASP Foundation, “OWASP top 10 for large jailbreak attack via self-reminder.” arXiv preprint language model applications.” https://owasp.org/www-pr arXiv:2310.06387, 2023. oject-top-10-for-large-language-model-applications/, 2023. [23] K. Zhu et al., “PromptBench: Towards evaluating the [6] F. Perez and M. T. Ribeiro, “Ignore previous prompt: robustness of large language models on adversarial Attack techniques for language models.” arXiv prompts,” 2023. preprint arXiv:2211.09527, 2022. [24] S. Schulhoff et al., “Ignore this title and HackAPrompt: [7] S. Willison, “Prompt injection attacks against GPT-3.” https://simonwillison.net/2022/Sep/12/prompt-injection/, 2022.

Exposing systemic vulnerabilities of LLMs through a global prompt hacking competition.” arXiv preprint arXiv:2311.16119, 2023.

[8] K. Greshake et al., “Not what you’ve signed [25] A. Shostack, Threat Modeling: Designing for Security. up for: Compromising real-world LLM-integrated Wiley, 2014. applications with indirect prompt injection,” arXiv [26] Y. Zhang et al., “Prompts should not be seen as secrets: preprint arXiv:2302.12173, 2023. Systematically measuring prompt extraction attack [9] H. Branch et al., “Evaluating the susceptibility success.” arXiv preprint arXiv:2307.06865, 2023. of pre-trained language models via handcrafted [27] A. Wan et al., “Poisoning language models during adversarial examples,” arXiv preprint arXiv:2209.02128, instruction tuning,” 2023. 2022. [28] E. Bagdasaryan and V. Shmatikov, “Abusing images [10] Y. Liu et al., “Prompt injection attacks and defenses and sounds for indirect instruction injection in in LLM-integrated applications,” arXiv preprint multi-modal LLMs,” 2023. arXiv:2310.12815, 2023. [29] Anthropic, “Constitutional AI: Harmlessness from AI [11] E. Wallace et al., “The instruction hierarchy: Training feedback.” arXiv preprint arXiv:2212.08073, 2023. LLMs to prioritise privileged instructions,” arXiv [30] V. Karpukhin et al., “Dense passage retrieval for preprint arXiv:2404.13208, 2024. open-domain question answering,” in Proceedings of [12] T. Rebedea et al., “NeMo Guardrails: A toolkit EMNLP, 2020. for controllable and safe LLM applications with [31] J. Johnson, M. Douze, and H. Jégou, “Billion-scale programmable rails.” arXiv preprint arXiv:2310.10501, similarity search with GPUs,” IEEE Transactions on Big 2023. Data, 2019. [13] H. Inan et al., “Llama Guard: LLM-based input-output [32] T. Wolf et al., “HuggingFace transformers: safeguard for human-AI conversations.” arXiv State-of-the-art natural language processing.” preprint arXiv:2312.06674, 2023. arXiv preprint arXiv:1910.03771, 2020. [14] N. Ahmed, M. I. Zaman, G. Saleem, and A. Hassan, [33] N. Reimers and I. Gurevych, “Sentence-BERT: “Do llms know they are being tested? evaluation 19

ICCK Transactions on Information Security and Cryptography

Sentence embeddings using siamese BERT-networks,” in Proceedings of EMNLP, 2019. [34] I. Loshchilov and F. Hutter, “Decoupled weight decay regularisation,” in Proceedings of ICLR, 2019. [35] Meta AI, “Llama Guard 2: Safeguarding human-AI conversations.” https://ai.meta.com/research/publications/ll ama-guard-2/, 2024. [36] T. Nguyen et al., “MS MARCO: A human generated machine reading comprehension dataset,” in Proceedings of NeurIPS Workshop on Cognitive Computation, 2016. [37] L. Zheng et al., “Judging LLM-as-a-judge with MT-Bench and chatbot arena,” in Advances in Neural Information Processing Systems (NeurIPS), 2023. [38] T. Zhang et al., “BERTScore: Evaluating text generation with BERT,” in Proceedings of ICLR, 2020. [39] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in 2017 ieee symposium on security and privacy (sp), pp. 39–57, Ieee, 2017.

Data and Code Availability The datasets, code, and related resources used in the implementation and experiments of this study are available at: GitHub Repository.

Funding This study did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Conflicts of Interest The authors are affiliated with SparkVerse AI, which contributed to this study. This affiliation may represent a potential conflict of interest. The research, analyses, and reporting were conducted objectively and in accordance with scientific and ethical standards.

Ethical Approval and Consent to Participate Not applicable. Appendix Gulshan Saleem received her Bachelor’s and Master’s degrees in Software Engineering and a Ph.D. in Computer Science. Her research interests include machine learning, data analytics, and intelligent systems. She has authored and co-authored peer-reviewed publications and contributed to research on the design and analysis of data-driven computational solutions. (Email: [email protected])

20

Nisar Ahmed (Senior Member, IEEE) received his Ph.D. and Master’s degrees in Computer Engineering and a Bachelor’s degree in Electrical Engineering. His research focuses on machine learning, deep learning, and AI-driven solutions for cybersecurity and healthcare. He has authored multiple peer-reviewed publications and contributed to research on data-driven security analytics and intelligent systems, with an emphasis on reproducibility and real-world deployment. (Email: [email protected])

Muhammad Imran Zaman Muhammad Imran Zaman received his Bachelor’s and Master’s degrees in Computer Science. He currently works in industry, focusing on practical, data-driven solutions for real-world data science problems. His interests include applied machine learning, data analytics, and the deployment of intelligent systems in production environments. (Email: [email protected])

Ali Hassan Ali Hassan received his B.S. degree in Computer Science. He is currently working in industry, focusing on the deployment of intelligent systems in production environments. He also has experience as a front-end web developer, contributing to the design and development of user-centric web applications. (Email: [email protected])