On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance–Robustness Gap Becky Mashaido and Tapadhir Das
arXiv:2605.19159v1 [cs.CR] 18 May 2026
Department of Computer Science, University of the Pacific, Stockton, USA Email: b [email protected], [email protected] Abstract—Prompt-injection attacks pose significant risks to language model safety, yet existing defenses are typically evaluated using classification performance. We show that high detection performance does not imply representational robustness. Specifically, multi-operator obfuscated prompts (combining homoglyphs, zero-width characters, and punctuation or emoji noise) can partially collapse onto the embedding manifold of clean prompts, a phenomenon we term latent embedding collapse. Results indicate that across multiple BERT-family encoders with varying depth and capacity, detectors achieve near-perfect classification performance, yet the minimal clean–obfuscated margin δ = 1.02, indicating near-overlap of obfuscated and clean embeddings. Obfuscated embeddings further exhibit elevated intra-class variance (3.33 ± 6.23), indicating severe latent-space instability despite high performance. These results reveal a substantial perf ormance–robustness gap, demonstrating that standard evaluation metrics fail to capture latent embedding collapse and underlying geometric fragility. Our findings show that increasing model capacity does not eliminate latent embedding collapse, motivating geometry-aware robustness analysis as a necessary complement to performance-based evaluation for prompt-injection defenses. Index Terms—Large language models, prompt injections, adversarial robustness, artificial intelligence, cybersecurity
I. I NTRODUCTION Large language models (LLMs) have become a ubiquitous part of our daily personal & professional lives. These are power neural network models that are trained on a wide variety of text and code to comprehend, reproduce, and handle human language [1]. Due to their widespread usage, the market size of LLMs are expected to hit $35.43 billion by 2030 [2]. However, despite their extended use, LLMs can be compromised by cyber attacks such as prompt injections [3]. Prompt-injection attacks pose significant risks to the safety of LLMs by embedding malicious instructions within seemingly benign user inputs. Such attacks can override system directives, manipulate model behavior, or exfiltrate sensitive information without explicit policy violations [4]. An illustration of prompt injection can be seen in Figure 1. As a result, a growing body of work proposes promptinjection detection mechanisms based on supervised classification, typically evaluated using accuracy, precision, recall, or F1 score [5], [6]. While these metrics capture observable misclassification behavior, they provide limited insights into the robustness of the underlying representations learned by
Regular Prompt (Plaintext)
Correct Response
User Manipulated Prompt (Unicode)
LLM
Sensitive Information
Adversary
Fig. 1: Illustration of Prompt Injection Attack on LLMs
transformers. In particular, a detector may achieve near-perfect performance while relying on fragile latent representations that can be exploited by adaptive adversaries. This raises a fundamental question: does high detection performance imply representational robustness against compositional prompt obfuscation? In this paper, we have investigated this research question and have discovered that the answer is No. For this, we demonstrate that multi-operator obfuscated prompts (constructed through combinations of Unicode homoglyphs, zerowidth characters, fragmented instructions and punctuation or emoji noise) can intrude into regions of the latent space occupied by clean prompts even when classification performance remains near-perfect. We refer to this phenomenon as latent embedding collapse, characterized by a partial manifold collapse in which obfuscated prompts intrude into regions of the latent space occupied by clean prompts without inducing misclassifications. To the best of our knowledge, this is the first research that is studying embedding-space or geometric analyses in LLM adversarial defense. This makes our focus on latent embedding collapse a novel perspective. The main contributions of this work are threefold: 1) Formalizing the concept of latent embedding collapse and connecting it to partial manifold collapse in transformer embedding spaces. 2) Introducing geometric metrics to quantify representational robustness, providing a complementary evaluation to conventional performance metrics. 3) Demonstrating empirically that increasing model depth or capacity does not resolve latent embedding collapse, indicating that the performance-robustness gap is an
inherent phenomenon rather than an artifact of limited model size. The rest of the paper is structured as follows: Section II provides our literature review; Section III introduces the system model; Section IV details the methodology of the research; Section V presents evaluation results for our project; and Section VI provides concluding remarks and future work.
Embedding Space Geometric Metrics Intra-class Variance Dataset Construction
Training Setup
Inter-class Distance Clean-Obfuscated Margin
II. R ELATED W ORK Prompt-injection attacks in LLMs have attracted increasing attention due to their potential to subtly alter model outputs [7] [8] [9] [10]. The work in [7] explored the security vulnerabilities in relation to prompt injection attacks for LLMs. Researchers in [8] introduced ChatInject, an attack that formatted malicious payloads to mimic native chat templates in LLMs. The work in [9] demonstrated that LLMs blur the lines between data and instructions during prompt injection attacks. Authors in [10] introduced Prompt Infection, where malicious prompts self-replicate across interconnected LLM agents. Prior defenses largely adopt a supervised classification approach, detecting malicious prompts using accuracy, precision, recall, or F1 score as evaluation metrics [11] [5] [6] [12].In [11], authors proposed UniGuardian, the first defense designed to detect prompt injection, backdoor attacks, and adversarial attacks in LLMs. The work in [5] investigated the underlying mechanisms of prompt injection attacks by analyzing the attention patterns within LLMs. Researchers in [6] proposed DefensiveToken, a defense with prompt injection robustness comparable to training-time alternatives. Authors in [12] introduced Jatmo, a defense mechanism for generating taskspecific models resilient to prompt injection attacks. Through these works, we observe that performance metrics demonstrate strong detection performance. However, a limitation of these works is that they provide limited insight into the geometry of the learned latent space and the robustness of underlying representations [13]. Several surveys and critiques highlight the limitations of performance-centric evaluation in this domain [14] [15]. Even recently proposed defenses, including data filtering [16] and representation-based methods [17], remain primarily evaluated on observable classification outcomes, rather than examining latent embedding structure. Observations on embedding-space or geometric analyses are rare in adversarial defense, making our focus on latent embedding collapse a novel perspective. This gap motivates our approach: rather than focusing solely on misclassification rates, we analyze the latent embedding collapse, a phenomenon in which obfuscated prompts partially intrude into regions of the latent space occupied by clean prompts without inducing misclassification. By emphasizing geometric properties of transformer representations, our method complements prior performance-based defenses and provides deeper insight into model robustness, revealing vulnerabilities that conventional metrics overlook.
Fig. 2: Proposed methodology of understanding obfuscated prompt interaction with clean embeddings
III. S YSTEM M ODEL In this section, we are providing the mathematical background for latent embedding collapse. Let ϕ(x) ∈ Rd denote the final-layer embedding of a prompt x produced by a transformer encoder. For each prompt class c ∈ {clean, prefix, suffix, obfuscated}, we define the corresponding embedding manifold Mc = {ϕ(x) | x ∼ Pc }. Latent embedding collapse occurs when obfuscated prompts occupy regions of the embedding space arbitrarily close to those of clean prompts, resulting in a loss of effective geometric separation between manifolds. We characterize this collapse through two complementary geometric conditions. First, the clean-obfuscated margin δ=
min
xc ∈Mclean , xo ∈Mobf
∥ϕ(xc ) − ϕ(xo )∥2
(1)
quantifies the minimal distance between clean and obfuscated embeddings. A small δ indicates that some obfuscated prompts lie extremely close to clean prompts in the latent space. Second, the obfuscated intra-class variance X 1 2 σobf ∥ϕ(xi ) − ϕ(xj )∥22 (2) = 2 |Mobf | xi ,xj ∈Mobf
captures the degree of dispersion among obfuscated embeddings, reflecting latent instability and spread across both benign and adversarial regions. We formalize this discrepancy as an performancerobustness gap, wherein high detection performance coexists with severe embedding-space vulnerability. By reframing prompt-injection detection as a problem of representation geometry rather than classification alone, this work highlights a fundamental limitation of existing defenses and motivates geometry-aware robustness evaluation as a necessary complement to performance-based metrics. IV. M ETHODOLOGY In this section, we describe our methodology for this research. Our goal is to understand how obfuscated prompts interact with clean embeddings, revealing phenomena such as latent embedding collapse and partial manifold collapse. Our proposed methodology is provided in Figure 2.
A. Dataset Construction We construct a dataset of 10,000 examples per label type across four classes: clean, prefix, suffix, and obfuscated. Clean sentences are generated from natural-language templates using nouns, verbs, adjectives, adverbs, and contextual extras. Prefix and suffix classes are created by prepending or appending partial or full instructional fragments, including both complete and fragmented instructions, to clean sentences. The obfuscated class is designed to probe latent embedding collapse by applying multi-layer transformations to base sentences. These transformations include: • Unicode homoglyph substitutions, e.g., Cyrillic replacements for Latin characters. • Zero-width character insertions and emoji/punctuation noise. • Embedded partial or fragmented instructional prefixes and suffixes. • Stochastic composition of all transformations, ensuring semi-ambiguous or hidden signals. Formally, for each base sentence x, the obfuscated function O(x) applies the above transformations probabilistically, creating a diverse manifold Mobf that partially intrudes into the clean manifold Mclean .
TABLE I: Classification performance across transformer architectures. Model DistilBERT BERTBase BERTMedium
Accuracy
Precision
Recall
F1
0.993 0.994 0.993
0.993 0.994 0.993
0.993 0.994 0.993
0.993 0.994 0.993
2) Inter-Class Distance: D(c1 , c2 ) =
X 1 |Mc1 ||Mc2 |
X
∥ϕ(x) − ϕ(y)∥2
(4)
x∈Mc1 y∈Mc2
The inter-class distance captures the separation between two class manifolds. Larger distances indicate well-separated classes in latent space, which generally improve robustness. Critically, when obfuscated prompts intrude into the clean manifold, D(clean, obfuscated) decreases, signaling latent embedding collapse. 3) Clean-Obfuscated Margin: δ=
min
xc ∈Mclean xo ∈Mobf
∥ϕ(xc ) − ϕ(xo )∥2
(5)
Let ϕ(x) ∈ Rd denote the final-layer embedding of prompt x. For each class c ∈ {clean, prefix, suffix, obfuscated}, define the embedding manifold
The minimum distance between clean and obfuscated embeddings highlights the most extreme cases of intrusion. A small δ reveals partial manifold collapse, where obfuscated prompts lie in regions occupied by clean prompts without triggering misclassification. This phenomenon is central to our claim: performance alone does not capture subtle vulnerabilities in latent space. These metrics collectively allow us to quantify the geometry of latent embeddings, going beyond performance-centric evaluations. By tracking intraclass variance, inter-class distances, and the clean–obfuscated margin, we can detect hidden overlap between classes (latent embedding collapse). We can also identify potential risks for LLMs to adversarial robustness, through which we can guide architecture or training modifications to improve manifold separation. This geometry-based perspective exposes vulnerabilities that traditional metrics miss and motivates the visualizations in Section V showing obfuscated prompts partially intruding into clean embedding regions.
Mc = {ϕ(x) | x ∼ Pc }.
V. R ESULTS
These metrics are designed to quantify the geometric structure of latent embeddings, exposing phenomena that are invisible to standard performance metrics, such as latent embedding collapse and partial manifold collapse. 1) Intra-Class Variance: X 1 2 ∥ϕ(xi ) − ϕ(xj )∥22 (3) σintra (c) = |Mc |2
We now analyze classification performance and latent-space geometry across our three transformer architectures (DistilBERT, BERTBase and BERTMedium) to evaluate whether high detection performance corresponds to representational robustness under multi-operator prompt obfuscation. First, we analyze the operating performance of each of the 3 observed models, as shown in Table I. We observe that all 3 models achieve near-identical classification performance, with accuracy, precision, recall, and macro F1 consistently around 99%. This indicates that the models can efficiently detect clean, suffix, prefix, and obfuscated prompts. From there, we perform a deeper analyses of the final-layer embeddings for one of the models (DistlBERT) as demonstrated in Table II,
B. Training Setup We fine-tune three transformer models: DistilBERT, BERTBase, and BERTMedium for 4-way classification. Key hyperparameters: • Dropout in classifier head: 0.1–0.2 • Weight decay: 0.01 −5 • Learning rate: 3 × 10 • Epochs: 25 with early stopping (patience 4) • Batch size: 16 per device We evaluate using accuracy, precision, recall, and macro F1. The three-model performance summary is shown in Table I. C. Embedding-Space Geometry Metrics
xi ,xj ∈Mc
This measures how tightly the embeddings of a given class 2 cluster in latent space. A small σintra indicates a compact manifold, suggesting the model consistently represents the class. Conversely, high intra-class variance may signal sensitivity to noise or obfuscation.
TABLE II: Pairwise Euclidean distances (mean ± std) between final-layer DistilBERT embeddings. Clean-obfuscated margin δ is shown below. Clean Suffix Prefix Obfuscated
Clean
Suffix
Prefix
Obfuscated
1.71 ± 0.84 25.65 ± 0.23 25.79 ± 0.17 24.34 ± 4.51
25.65 ± 0.23 1.21 ± 0.40 23.93 ± 0.18 25.05 ± 0.18
25.79 ± 0.17 23.93 ± 0.18 1.51 ± 0.60 25.15 ± 0.19
24.34 ± 4.51 25.05 ± 0.18 25.15 ± 0.19 3.33 ± 6.23
Clean-Obfuscated Margin δ = 1.02
which summarizes pairwise Euclidean distances between finallayer DistilBERT embeddings and the clean-obfuscated margin δ. Here we note that while the classification performance for DistilBERT exceeds 99%, the embedding geometry reveals a markedly different picture. We observe that the obfuscated prompts exhibit the largest intra-class variance (3.33 ± 6.23), indicating some embeddings lie extremely close to clean prompts while others are distant. This indicates that there is severe latent-space instability. We also note that inter-class distances to clean prompts have elevated standard deviation (24.34±4.51). This demonstrates partial collapse of obfuscated embeddings onto the clean manifold. Additionally, we observe that the minimal margin δ = 1.02 confirms the presence of latent embedding collapse. δ = 1.02 confirms that some adversarial inputs lie arbitrarily close to clean prompts in embedding space, despite correct predictions. These results quantify the performance-robustness gap, where performance alone would suggest robustness, but the embedding geometry indicates structural fragility. Observing this, we perform a complete evaluation of the 3 models. We analyze the final-layer embeddings of the models through two performance metrics: Principal Component Analysis (PCA) and t-Distributed Stochastic Neighbor Embedding (t-SNE). PCA and t-SNE are chosen for evaluation as they are contemporary metrics used for dimensionality reduction and data visualization in literature [18]. For PCA, the results for DistillBERT are in Figure 3, the results for BERTBase are in Figure 4, and BERTMedium are in Figure 5. In these results, we note that, despite architectural differences, all the models exhibit the same characteristics. Clean, prefix, and suffix prompts form compact and well-separated manifolds, while obfuscated prompts display elevated dispersion and partial overlap with the clean manifold. Increasing depth or hidden dimensionality does not increase the clean-obfuscated margin nor reduce obfuscated variance. Similarly for t-SNE, the results for DistillBERT are in Figure 6, the results for BERTBase are in Figure 7, and BERTMedium are in Figure 8. In these results, we note that, across all models, prefix and suffix prompts remain well separated from clean inputs, which is traditionally consistent with high classification performance. However, obfuscated prompts partially overlap with the clean cluster, providing visual evidence of latent embedding collapse and partial manifold collapse. These geometric intrusions occur even when predictions are correct, reinforcing the premise that
Fig. 3: DistillBERT PCA Projections of Prompt Embeddings
Fig. 4: BERTBase PCA Projections of Prompt Embeddings
performance alone fails to capture latent vulnerability. These results indicate that architectural scaling improves convergence and classification confidence but does not resolve the underlying geometric fragility. Latent embedding collapse persists across model families, confirming that the performance-robustness gap is not an artifact of insufficient model capacity. Taken together, the quantitative margins, cross-model comparisons, and visualizations demonstrate that high detection performance can coexist with severe geometric fragility. This confirms the central claim of this work: robust prompt-injection defense requires geometry-aware evaluation beyond performance-based metrics. VI. C ONCLUSION In this paper, we conducted a geometric analysis of transformer embeddings under multi-operator prompt-injection attacks on LLMs. For this, we formalized the concept of latent embedding collapse and connected it to partial manifold
Fig. 5: BERTMedium PCA Projections of Prompt Embeddings
Fig. 7: BERTBase t-SNE Projections of Prompt Embeddings
Fig. 6: DistillBERT t-SNE Projections of Prompt Embeddings
Fig. 8: BERTMedium t-SNE Projections of Prompt Embeddings
collapse in transformer embedding spaces. We also introduced geometric metrics to quantify representational robustness, providing a complementary evaluation to conventional performance metrics. Finally, we demonstrated empirically that increasing model depth or capacity does not resolve latent embedding collapse, indicating that the performancerobustness gap is an inherent phenomenon rather than an artifact of limited model size. Results indicate that multi-operator obfuscated prompts can partially collapse onto the clean embedding manifold, confirming latent embedding collapse, despite near-perfect classification performance. Additionally, embedding-space metrics, including pairwise distances, intraand inter-class variance, and the minimal clean-obfuscated margin (δ = 1.02), expose vulnerabilities invisible to standard evaluation, highlighting the performance-robustness gap. Finally, we noted that high classifier performance can coexist with severe structural fragility in the embedding space, emphasizing that true robustness requires geometry-aware evaluation.
For future work, we will continue this investigation in multiple directions. One direction is to explore topology-aware descriptors of embedding neighborhoods, since obfuscated prompts may preserve semantic structure while disrupting token-level geometry, causing standard distance metrics to collapse even when topological invariants remain distinct. Another direction will be formalizing the latent robustness gap, which quantifies the discrepancy between classifier confidence and embeddingspace stability. Finally, we will design training strategies that explicitly preserve geometric and topological properties under obfuscation. R EFERENCES [1] E. Kasneci, K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier et al., “Chatgpt for good? on opportunities and challenges of large language models for education,” Learning and individual differences, vol. 103, p. 102274, 2023.
[2] “Large Language Models Market Size — Industry Report, 2030 — grandviewresearch.com,” https://www.grandviewresearch.com/ industry-analysis/large-language-model-llm-market-report, [Accessed 10-01-2026]. [3] Y. Liu, G. Deng, Y. Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng et al., “Prompt injection attack against llm-integrated applications,” arXiv preprint arXiv:2306.05499, 2023. [4] J. Shi, Z. Yuan, Y. Liu, Y. Huang, P. Zhou, L. Sun, and N. Z. Gong, “Optimization-based prompt injection attack to llm-as-a-judge,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 660–674. [5] K.-H. Hung, C.-Y. Ko, A. Rawat, I.-H. Chung, W. H. Hsu, and P.-Y. Chen, “Attention tracker: Detecting prompt injection attacks in llms,” in Findings of the Association for Computational Linguistics: NAACL 2025, 2025, pp. 2309–2322. [6] S. Chen, Y. Wang, N. Carlini, C. Sitawarin, and D. Wagner, “Defending against prompt injection with a few defensivetokens,” in Proceedings of the 18th ACM Workshop on Artificial Intelligence and Security, 2025, pp. 242–252. [7] M. A. Rahman, H. Shahriar, G. Francia, F. Wu, A. Cuzzocrea, M. Rahman, M. J. H. Faruk, and S. I. Ahamed, “Fine-tuned large language models (llms): Improved prompt injection attacks detection,” in 2025 IEEE 49th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 2025, pp. 1033–1039. [8] H. Chang, Y. Jun, and H. Lee, “Chatinject: Abusing chat templates for prompt injection in llm agents,” arXiv preprint arXiv:2509.22830, 2025. [9] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llmintegrated applications with indirect prompt injection,” in Proceedings of the 16th ACM workshop on artificial intelligence and security, 2023, pp. 79–90. [10] D. Lee and M. Tiwari, “Prompt infection: Llm-to-llm prompt injection within multi-agent systems,” arXiv preprint arXiv:2410.07283, 2024. [11] H. Lin, Y. Lao, T. Geng, T. Yu, and W. Zhao, “Uniguardian: A unified defense for detecting prompt injection, backdoor attacks and adversarial attacks in large language models,” arXiv preprint arXiv:2502.13141, 2025. [12] J. Piet, M. Alrashed, C. Sitawarin, S. Chen, Z. Wei, E. Sun, B. Alomair, and D. Wagner, “Jatmo: Prompt injection defense by task-specific finetuning,” in European Symposium on Research in Computer Security. Springer, 2024, pp. 105–124. [13] S. Goyal, S. Doddapaneni, M. M. Khapra, and B. Ravindran, “A survey of adversarial defenses and robustness in nlp,” ACM Computing Surveys, vol. 55, no. 14s, pp. 1–39, 2023. [14] Q. Wang, “The comprehensive review on prompt injection attacks and defense mechanisms in large language models,” Science and Technology of Engineering, Chemistry and Environmental Protection, vol. 1, no. 3, 2025. [15] Y. Jia, Z. Shao, Y. Liu, J. Jia, D. Song, and N. Z. Gong, “A critical evaluation of defenses against prompt injection attacks,” arXiv preprint arXiv:2505.18333, 2025. [16] Y. Wang, S. Chen, R. Alkhudair, B. Alomair, and D. Wagner, “Defending against prompt injection with datafilter,” arXiv preprint arXiv:2510.19207, 2025. [17] R. Liu, Y. Lin, and J. S. Dong, “Drip: Defending prompt injection via de-instruction training and residual fusion model architecture,” arXiv e-prints, pp. arXiv–2511, 2025. [18] F. Anowar, S. Sadaoui, and B. Selim, “Conceptual and empirical comparison of dimensionality reduction algorithms (pca, kpca, lda, mds, svd, lle, isomap, le, ica, t-sne),” Computer Science Review, vol. 40, p. 100378, 2021.