HiveTraceLab
arXiv:2604.25716v1 [cs.CL] 28 Apr 2026
Cross-Lingual Jailbreak Detection via Semantic Codebooks Shirin Alanova AI Talent Hub, ITMO University [email protected] Sabrina Sadiekh HiveTraceLab [email protected]
Bogdan Minko HiveTraceLab [email protected] Evgeniy Kokuykin HiveTraceLab [email protected]
Abstract Safety mechanisms for large language models (LLMs) remain predominantly English-centric, creating systematic vulnerabilities in multilingual deployment. Prior work shows that translating malicious prompts into other languages can substantially increase jailbreak success rates, exposing a structural cross-lingual security gap. We investigate whether such attacks can be mitigated through languageagnostic semantic similarity without retraining or language-specific adaptation. Our approach compares multilingual query embeddings against a fixed English codebook of jailbreak prompts, operating as a training-free external guardrail for black-box LLMs. We conduct a systematic evaluation across four languages, two translation pipelines, four safety benchmarks, three embedding models, and three target LLMs (Qwen, Llama, GPT-3.5). Our results reveal two distinct regimes of cross-lingual transfer. On curated benchmarks containing canonical jailbreak templates, semantic similarity generalizes reliably across languages, achieving near-perfect separability (AUC up to 0.99) and substantial reductions in absolute attack success rates under strict low–false-positive constraints. However, under distribution shift—on behaviorally diverse and heterogeneous unsafe benchmarks—separability degrades markedly (AUC ≈ 0.60-0.70), and recall in the security-critical low-FPR regime drops across all embedding models.
1
Introduction
Safety alignment for large language models (LLMs) remains predominantly Englishcentric. A recent large-scale analysis of nearly 300 publications (2020–2024) by
Yong et al. [19] reveals a substantial linguistic imbalance: non-English languages are underrepresented by a factor of 5–10× even among high-resource languages such as Chinese and Arabic, while low-resource languages are largely ignored. Crucially, non-English settings are rarely studied as standalone security scenarios, but instead treated as peripheral extensions of English evaluations. This imbalance creates structural blind spots in multilingual deployment, where safety mechanisms may fail outside the linguistic domain in which they were primarily aligned. Empirical evidence confirms this vulnerability. Li et al.[10] demonstrate that translating malicious prompts into other languages systematically increases jailbreak success rates across advanced LLMs. Similarly, Shen et al.[18] show that refusal behaviors learned in English often fail to transfer reliably across languages. While fine-tuning on multilingual jailbreak datasets can substantially reduce attack success rates, such approaches require model access, additional data curation, and language-specific adaptation, limiting their applicability in black-box or resource-constrained settings. These findings raise a practical question: can cross-lingual jailbreak attempts be mitigated without retraining, additional data collection, or language-specific tuning? In this work, we investigate whether harmful intent can be detected through languageagnostic semantic similarity after translation. Specifically, we ask: Can a fixed English semantic codebook detect cross-lingual jailbreak attempts through multilingual embeddings, without retraining or translation-based preprocessing? To answer this question, we construct a training-free detection framework that compares multilingual query embeddings against a fixed English codebook of jailbreak prompts. We systematically evaluate this approach across four languages, two translation pipelines, four safety benchmarks, three embedding models, and three target LLMs. Our contributions are as follows: • We propose a training-free cross-lingual guardrail based on semantic similarity to a fixed English codebook, deployable as an external filter for black-box LLMs • We provide a systematic empirical evaluation of cross-lingual transfer across multiple benchmarks, identifying settings where similarity-based detection generalizes reliably and settings where performance degrades under distribution shift • We analyze performance in the security-critical low-FPR regime and demonstrate that embedding choice critically affects detection robustness • We position semantic codebooks as a lightweight first-line filter for multilingual safety, highlighting both their practical utility and their limitations 2
2
Related Works
2.1
Cross-lingual security gaps
LLM security remains extremely English-centric. Yong et al [19], confirmed that over 90% of publications ignore non-English languages even in multilingual models. This has real-world implications: Li et al. [10] demonstrate that translating jailbreak requests into other languages systematically bypasses filters even in GPT-4 and LLaMA. Shen et al. [18] confirm that models trained to reject harmful requests in English often fail in other languages.
2.2
Attack detection methods
Existing approaches are divided into internal and external. Internal methods (RLHF, DPO, PEFT) require access to the model and retraining [7, 14, 16], which is not applicable to black-box APIs. External methods work as preprocessing, but most of them are based on English: rule-based filters can be readily bypassed by rephrasing [11], and classifiers such as Prompt-Guard-86M [12] lose their effectiveness on translated queries [18].
2.3
Semantic similarity as a signal
A more promising approach is to compare query embeddings with a codebook of known attacks using cosine similarity. Hypothesis: malicious intent forms a connected region in semantic space, which allows detecting zero-shot attacks when using aligned multilingual embeddings [13]. This is empirically confirmed: [5] and [1] show that frozen embeddings with classical ML classifiers outperform fine-tuned transformers at FPR < 1%.
2.4
The role of embedders
Not all multilingual models are the same. Bell et al. [2] find that the “translate → classify” strategy outperforms direct multilingual classifiers in toxicity tasks, especially for low-resource languages. However, for jailbreaks, semantic drift during translation (e.g., via Google Translate) remains a problem. BGE-M3 [4] demonstrates state-of-the-art alignment, but its effectiveness for security in realistic settings has not been tested. Theoretically, cross-language translation should be supported if the embedding spaces are well-aligned, that is, if they exhibit isomorphism, isometry, and isotropy [15]. However, in practice, even the most powerful models suffer from translation artifacts, especially in the case of typologically distant languages such as Arabic or Chinese. Alternative architectures, such as SPIRE [17], attempt to address this issue by dynamically indexing conflicting fragments, but they remain sensitive to tokenization and require manual curation. 3
Our work fills this critical gap. We systematically evaluate whether a fixed English semantic code book combined with BGE-M3 can reliably prevent cross-language jailbreaks – without retraining, without language-specific tuning, and without dependence on translation quality. This is not just another detector, it is an attempt to address the fundamental blindness of LLMs to their own vulnerabilities outside of English.
3
Methodology
3.1
Detection Framework Overview
Our approach is a training-free cross-lingual guardrail based on semantic similarity to a fixed English codebook of jailbreak prompts. Given an incoming query in any language, we encode it using a multilingual sentence embedding model and compare it to a set of pre-computed embeddings of unsafe English prompts. Detection is performed via nearest-neighbor similarity without retraining, translation-based preprocessing, or language-specific adaptation. Figure 1 illustrates the overall detection pipeline.
Figure 1: Overview of the proposed cross-lingual semantic filtering framework. Incoming user input (in any language) is encoded using a multilingual embedding model and compared against a fixed English codebook of jailbreak prompts. If the maximum cosine similarity exceeds a predefined threshold, the query is blocked; otherwise, it is forwarded to the target LLM. The approach operates as a training-free external guardrail and does not require translation or model fine-tuning.
3.2
Codebook Construction
The codebook is a fixed set of English-language unsafe prompts embedded into a shared vector space. We construct the codebook from the training split of jayavibhav/prompt-injectionsafety. Starting from 15,237 unsafe examples, we remove duplicates, extremely short prompts, and malformed inputs. To reduce annotation noise, each candidate prompt is additionally checked by two external guard models (Prompt-Guard-86M [12] and Qwen3Guard-Gen-4B [20]). A prompt is retained if at least one model confirms it as unsafe. 4
The final codebook contains 13,811 unique jailbreak prompts. The codebook remains fixed throughout all experiments.
3.3
Embedding and Detection Rule
All prompts are encoded using multilingual sentence embeddings (BGE-M3 unless otherwise stated). Let f (x) ∈ R1024 denote the L2-normalized embedding of query x. Let C = {c1 , . . . , cN } be the codebook embeddings. For a query x, we compute the maximum cosine similarity: s(x) = max f (x)⊤ c. c∈C
The query is classified as unsafe if s(x) ≥ τ , where τ is selected on validation data. This yields a nearest-neighbor similarity detector operating entirely in embedding space and requiring no model fine-tuning. The threshold τ is selected on a held-out validation partition derived from the training split of jayavibhav/prompt-injection-safety. Specifically, we perform a grid search over candidate thresholds to maximize TPR subject to the hard operational constraint FPR≤ 1%. This optimization protocol reflects deployment settings where false positives incur disproportionate filtering costs. The resulting per-language thresholds and their corresponding operating points are reported in Appendix B.
3.4
Evaluation Metrics
We report standard binary classification metrics (TPR, FPR, AUC). Because deployment-grade safety systems must operate under strict false-positive constraints, we focus particularly on the low-FPR regime (e.g., TPR at FPR ≤ 1%). To assess real-world mitigation impact, we additionally measure Attack Success Rate (ASR), defined as the proportion of unsafe prompts that elicit harmful outputs from a target LLM. We report both baseline ASR (without filtering) and ASR after applying the semantic filter.
3.5
Cross-Lingual Setup
To simulate realistic cross-lingual attacks, English prompts are translated into Russian, Chinese, and Arabic using two translation pipelines: (i) Google Translate (commercial system), (ii) M2M100 (facebook/m2m100_418M). No post-editing or back-translation is applied. This reflects realistic attacker behavior and preserves potential translation artifacts. 5
3.6
Evaluation Datasets
We evaluate our approach on four publicly available binary-labeled safety datasets covering prompt injection, jailbreak behaviors, and broader unsafe content: • jayavibhav/prompt-injection-safety[8] (used for codebook construction and testing), • xTRam1/safe-guard-prompt-injection[9], • JailbreakBench/JBB-Behaviors[3], • nvidia/Aegis-AI-Content-Safety-Dataset-2.0[6]. For large datasets, we sample balanced subsets of 1,000 prompts (500 safe / 500 unsafe). JailbreakBench (200 prompts) is used in full as a stress-test benchmark. Evaluation sets are not manually re-cleaned beyond deduplication in order to preserve realistic noise conditions.
3.7
Benchmark Characteristics
The four benchmarks differ in structure and difficulty. Benchmarks 1–2 correspond to jayavibhav/prompt-injection-safety and xTRam1/safe-guard-prompt-injection, respectively, and primarily contain prompt-injection and canonical jailbreak-style instructions with relatively homogeneous attack patterns. These settings resemble templated or pattern-driven attacks. In contrast, JailbreakBench (Benchmark 3) and Aegis-AI-Content-Safety-Dataset-2.0 (Benchmark 4) exhibit substantially greater behavioral diversity. They include heterogeneous unsafe categories (e.g., harmful instructions, sensitive topics, adversarial rephrasings) and contain prompts that do not necessarily follow canonical jailbreak templates. This increased diversity and distributional variability makes similarity-based detection more challenging and provides a stress-test for cross-lingual transfer. We report results separately across benchmarks to capture these differences in structure and difficulty.
4
Experiments and Results
We structure the evaluation around three research questions. Across all experiments, a consistent pattern emerges: similarity-based detection exhibits two distinct behavioral regimes depending on benchmark structure. On curated benchmarks dominated by canonical jailbreak templates, cross-lingual transfer is strong and stable. In contrast, on behaviorally diverse and heterogeneous unsafe benchmarks, performance degrades substantially. The following research questions quantify this effect at three levels: separability, operating constraints, and end-to-end mitigation. 6
RQ1 (Cross-lingual separability). Does an English-only semantic codebook transfer across languages in embedding space, i.e., can it separate unsafe vs. safe prompts after translation? RQ2 (Security-critical operating point). Is there a practical single-threshold operating regime with low false-positive rate (FPR) that still catches a meaningful fraction of attacks across languages and benchmarks? RQ3 (End-to-end mitigation). Does the detector reduce jailbreak success when used as an external pre-filter for black-box LLMs, and how does this depend on the target model and translation method? RQ1: Cross-lingual separability. We quantify cross-lingual transfer using ROCAUC across all benchmark–language pairs (Table 1). Results reveal a clear benchmark-dependent pattern. On curated prompt-injection benchmarks (Benchmarks 1–2), separability remains high across languages. In particular, Benchmark 2 achieves near-perfect AUC in English (0.993) and maintains strong performance under translation (0.847–0.884), indicating that similarity to an English codebook transfers reliably when attacks follow canonical jailbreak patterns. In contrast, on behaviorally diverse benchmarks (Benchmarks 3–4), separability degrades substantially. AUC drops to 0.618–0.703 on Benchmark 3 and to 0.593–0.627 on Benchmark 4 across translations. These datasets contain more heterogeneous unsafe behaviors and less templated attack patterns, making similarity-based discrimination more challenging. Figure 2 illustrates this contrast by comparing ROC curves from a cleaner setting (Benchmark 1) and a heterogeneous setting (Benchmark 4). Table 1: AUC-ROC scores across benchmarks and input languages (BGE-M3 embedder).
bench1 bench2 bench3 bench4
eng
rum2m
rugt
zhm2m
zhgt
arm2m
argt
0.829 0.993 0.618 0.615
0.785 0.854 0.675 0.614
0.806 0.866 0.694 0.627
0.781 0.855 0.694 0.593
0.792 0.884 0.703 0.617
0.765 0.855 0.660 0.605
0.782 0.847 0.703 0.609
RQ2: Low-FPR regime and thresholding. Deployment-grade safety systems must operate under strict false-positive constraints. In Table 2, we report the securitycritical metric TPR at fixed FPR≤ 1%, computed by identifying the per-language operating point that satisfies the constraint on the validation partition. To evaluate deployability without language-specific routing, we additionally fix a single global threshold τfinal = 0.66, calibrated on the English validation split to approximate the FPR≤ 1% regime across translated queries. This global configuration enables zero-configuration inference while preserving strict false-positive bounds in the majority of cross-lingual settings. Under this constraint, the contrast between the two regimes becomes even more pronounced. On Benchmark 2, similarity-based detection remains effective, achieving TPR between 78.5% and 91.9% across languages with BGE-M3. In this setting, 7
Figure 2: Representative ROC curves illustrating cross-lingual transfer (English vs. Russian; M2M100 vs. Google Translate) on Benchmark 1 (cleaner) and Benchmark 4 (noisier).
canonical jailbreak templates remain sufficiently close to the English codebook even after translation. On Benchmark 4, however, recall collapses to single digits (3.3–6.4%) across all languages and embedding models. When FPR is tightly constrained, similarity-only filtering fails to capture the majority of unsafe prompts in this heterogeneous setting. Table 2: Security-critical metric (FPR ≤ 1%): TPR at fixed FPR≤ 1% by embedder, language/translation, and benchmark. Embedder
Eng
Ru (m2m)
Ru (gt) Zh (m2m) Zh (gt) Ar (m2m)
Ar (gt)
Benchmark 1 bge-m3 25.6% multilingual-e5-large 24.4% jina-embeddings-v3 12.4%
22.2% 14.7% 9.0%
22.2% 17.0% 8.8%
21.6% 4.8% 10.1%
22.8% 7.9% 9.9%
20.1% 14.9% 7.0%
21.1% 15.9% 7.0%
80.1% 74.8% 85.5%
87.7% 76.8% 91.3%
78.5% 81.6% 79.3%
82.3% 82.3% 85.6%
17.0% 9.0% 10.0%
21.0% 7.0% 13.0%
17.0% 10.0% 11.0%
21.0% 5.0% 16.0%
4.5% 1.7% 5.7%
5.3% 2.0% 6.0%
6.1% 6.4% 6.6%
6.4% 2.9% 7.3%
Benchmark 2 bge-m3 91.9% multilingual-e5-large 95.2% jina-embeddings-v3 93.8%
82.9% 82.3% 85.5%
85.3% 82.8% 88.6% Benchmark 3
bge-m3 23.0% multilingual-e5-large 15.0% jina-embeddings-v3 19.0%
14.0% 10.0% 6.0%
18.0% 6.0% 6.0% Benchmark 4
bge-m3 multilingual-e5-large jina-embeddings-v3
3.3% 3.9% 3.8%
5.2% 6.1% 3.9%
5.8% 5.0% 6.5%
RQ3: End-to-end jailbreak mitigation on target LLMs. We evaluate practical security impact by comparing successful jailbreaks without filtering (Attacks0 ) and 8
with the semantic codebook pre-filter (Attacks1 ) across three target LLMs. Full per-language results are provided in Appendix C; here we report aggregated statistics. Table 3: Average relative reduction in successful jailbreaks (%) across models and languages. Mean and standard deviation are computed over all model–language pairs within each benchmark.
Benchmark
Mean Reduction (%)
Std
Benchmark 1 Benchmark 2 Benchmark 3 Benchmark 4
96.2 50.0 43.7 18.6
± 2.6 ± 17.4 ± 21.6 ± 13.8
Table 3 summarizes the average relative reduction in successful jailbreaks across models and languages within each benchmark. On canonical prompt-injection benchmarks (Benchmarks 1–2), the semantic filter removes a substantial fraction of successful attacks, achieving a mean reduction of 96.2% on Benchmark 1 and 50.0% on Benchmark 2. Under distribution shift, mitigation weakens significantly: the mean reduction drops to 43.7% on Benchmark 3 and further to 18.6% on the most heterogeneous benchmark (Benchmark 4). The increasing standard deviation across Benchmarks 2–4 indicates growing instability across models and languages in harder settings. These results closely mirror the separability and low-FPR findings from RQ1–RQ2: end-to-end mitigation is strong in canonical regimes but degrades sharply under heterogeneous unsafe distributions. Note: ASR1 is computed over the subset of unsafe prompts that pass the filter. Consequently, ASR1 may increase even when the absolute number of successful jailbreaks decreases. Absolute reduction therefore provides the more informative security metric. Supporting analysis: codebook size and embedder choice. We additionally analyze (i) the sensitivity to codebook size and (ii) the sensitivity to the embedding model. Table 4 shows that larger codebooks tend to increase TPR but also substantially increase FPR, indicating a practical trade-off between coverage and false alarms. Tables 5 and 2 show that while AUC can be similar across embedders on some benchmarks, differences become pronounced in the low-FPR regime; for example, on Benchmark 1 (Chinese, m2m), BGE-M3 achieves TPR 21.6% at FPR≤ 1% versus 4.8% for multilingual-e5-large. On Benchmark 4, all embedders collapse to single-digit TPR at FPR≤ 1%, reinforcing the limitation of similarity-only filtering in the hardest setting.
5
Conclusion
We examined whether cross-lingual jailbreak attacks can be mitigated using a fixed English semantic codebook combined with multilingual sentence embeddings. Our experiments reveal a clear two-regime pattern. 9
Table 4: Codebook subsampling on Benchmark 1 (unsafe class). Larger codebooks improve TPR but increase FPR, highlighting a coverage–false-alarm trade-off. Codebook Size
Language
TPR
FPR
TNR
FNR
25%
English (native) Russian (m2m) Chinese (m2m) Arabic (m2m) Russian (gt) Chinese (gt) Arabic (gt)
0.9476 0.6452 0.6195 0.5200 0.7713 0.7438 0.6618
0.6316 0.2318 0.2166 0.1558 0.3284 0.3196 0.2731
0.3684 0.7682 0.7834 0.8442 0.6716 0.6804 0.7269
0.0524 0.3548 0.3805 0.4800 0.2287 0.2562 0.3382
50%
English (native) Russian (m2m) Chinese (m2m) Arabic (m2m) Russian (gt) Chinese (gt) Arabic (gt)
0.9638 0.7200 0.6886 0.5952 0.8336 0.8101 0.7432
0.7100 0.3045 0.2753 0.2034 0.4196 0.4009 0.3383
0.2900 0.6955 0.7247 0.7966 0.5804 0.5991 0.6617
0.0362 0.2800 0.3114 0.4048 0.1664 0.1899 0.2568
75%
English (native) Russian (m2m) Chinese (m2m) Arabic (m2m) Russian (gt) Chinese (gt) Arabic (gt)
0.9713 0.7568 0.7182 0.6311 0.8651 0.8399 0.7717
0.7368 0.3506 0.3071 0.2313 0.4618 0.4435 0.3787
0.2632 0.6494 0.6929 0.7687 0.5382 0.5565 0.6213
0.0287 0.2432 0.2818 0.3689 0.1349 0.1601 0.2283
100%
English (native) Russian (m2m) Chinese (m2m) Arabic (m2m) Russian (gt) Chinese (gt) Arabic (gt)
0.9814 0.8162 0.7798 0.6987 0.9077 0.8862 0.8346
0.8012 0.4385 0.3790 0.2904 0.5547 0.5244 0.4583
0.1988 0.5615 0.6210 0.7096 0.4453 0.4756 0.5417
0.0186 0.1838 0.2202 0.3013 0.0923 0.1138 0.1654
On curated benchmarks dominated by canonical jailbreak templates, similarity-based detection transfers reliably across languages, achieving near-perfect separability (AUC up to 0.993) and high recall under strict false-positive constraints (TPR 78– 92% at FPR ≤ 1%). When deployed as an external pre-filter, it consistently reduces the absolute number of successful jailbreaks across target LLMs and translation pipelines. Under distribution shift, however, performance degrades substantially. On behaviorally diverse benchmarks, AUC drops to 0.59–0.63 and recall in the low-FPR regime falls to single digits. These results indicate that semantic similarity to a fixed English codebook effectively captures canonical attack patterns but does not generalize reliably to heterogeneous or adversarially rephrased unsafe content. 10
Table 5: Overall discriminatory power (AUC-ROC). AUC by embedder and language on each benchmark. Embedder
Eng
Ru (m2m) Ru (gt) Zh (m2m) Zh (gt)
Ar (m2m)
Ar (gt)
Benchmark 1 bge-m3 0.830 multilingual-e5-large 0.852 jina-embeddings-v3 0.767
0.785 0.763 0.707
0.810 0.765 0.718
0.781 0.733 0.726
0.792 0.730 0.727
0.765 0.712 0.709
0.782 0.720 0.698
0.981 0.970 0.987
0.991 0.980 0.991
0.977 0.982 0.972
0.984 0.992 0.9833
0.693 0.687 0.713
0.702 0.669 0.733
0.661 0.705 0.698
0.704 0.734 0.749
0.595 0.535 0.565
0.619 0.510 0.599
0.610 0.618 0.574
0.612 0.587 0.587
Benchmark 2 bge-m3 0.993 multilingual-e5-large 0.998 jina-embeddings-v3 0.995
0.985 0.973 0.9864
0.988 0.975 0.989 Benchmark 3
bge-m3 0.719 multilingual-e5-large 0.708 jina-embeddings-v3 0.700
0.673 0.685 0.717
0.693 0.713 0.737 Benchmark 4
bge-m3 0.620 multilingual-e5-large 0.635 jina-embeddings-v3 0.591
0.618 0.643 0.571
0.631 0.627 0.606
Our analysis further demonstrates that embedding choice and codebook size introduce critical trade-offs between coverage and false-alarm rates, particularly in the security-critical operating regime (Appendix E).While the proposed framework relies on a static English codebook—a design choice that ensures deployment stability but inherently limits adaptability to novel attack morphologies or language-specific harm formulations—it establishes a scalable, training-free first-line defense for black-box multilingual LLMs. Similarity-based filtering alone is insufficient for comprehensive protection and should be integrated into a broader, multi-layered safety pipeline.
6
Future Work
Our findings identify several specific directions for extending semantic guardrails in multilingual settings. Codebook maintenance and adaptation. To mitigate the static nature of the English codebook, future systems should implement a continuous update pipeline: (i) production false negatives are clustered in embedding space to identify emerging semantic centroids; (ii) candidate entries are validated via ensemble LLM adjudication and cross-lingual consistency checks; (iii) periodic codebook refreshes preserve coverage while controlling concept drift. This active-learning loop enables adaptation to evolving jailbreak tactics without full detector retraining. Native multilingual codebooks. Extending the framework to incorporate languagespecific templates—via controlled translation, semantic deduplication, and isotropy11
aware alignment—may improve recall for typologically distant languages (e.g., Arabic, Chinese) and reduce reliance on translation-induced semantic drift. Hybrid and cascaded architectures. Given the performance degradation under distribution shift, similarity-based filtering should be combined with orthogonal signals such as perplexity-based anomaly detection, syntactic pattern analysis, or LLM-verified secondary review. Cascaded pipelines can compensate for similarity degradation while maintaining strict operational FPR constraints. Input sensitivity and prompt structure. Systematic analysis of how detection performance varies with input length, contextual complexity, and syntactic obfuscation will clarify the operational boundaries of embedding-based filtering and guide prompt-aware normalization strategies. Longitudinal deployment evaluation. Real-world assessment on live traffic is necessary to quantify resilience to evolving attack distributions, language drift, and production constraints (latency, throughput, cost), ultimately establishing practical deployment guidelines for semantic guardrails.
12
References [1] Md. Ahsan Ayub and Subhabrata Majumdar. Embedding-based classifiers can detect prompt injection attacks. arXiv preprint arXiv:2410.22284, 2024. [2] Samuel J. Bell, Eduardo Sánchez, David Dale, Pontus Stenetorp, Mikel Artetxe, and Marta R. Costa-jussà. Translate, then detect: Leveraging machine translation for cross-lingual toxicity classification. arXiv preprint arXiv:2509.14493, 2025. [3] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. JailbreakBench: An open robustness benchmark for jailbreaking large language models. arXiv preprint arXiv:2404.01318, 2024. [4] Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. BGE-M3-embedding: Multi-lingual, multi-functionality, multigranularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216, 2024. [5] Erick Galinkin and Martin Sablotny. Improved large language model jailbreak detection via pretrained embeddings. arXiv preprint arXiv:2412.01547, 2024. [6] Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. AEGIS: Online adaptive AI content safety moderation with ensemble of LLM experts. arXiv preprint arXiv:2404.05993, 2024. [7] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. [8] Jayavibhav. Prompt injection safety dataset. https://huggingface.co/dat asets/jayavibhav/prompt-injection-safety, 2024. Accessed: 2026. [9] Hao Li and Xiaogeng Liu. InjecGuard: Benchmarking and mitigating overdefense in prompt injection guardrail models. arXiv preprint arXiv:2410.22770, 2025. [10] Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue. A cross-language investigation into jailbreak attacks in large language models. arXiv preprint arXiv:2401.16765, 2024. [11] Xing Li, Hui-Ling Zhen, Lihao Yin, Xianzhi Yu, Zhenhua Dong, and Mingxuan Yuan. What matters for safety alignment? arXiv preprint arXiv:2601.03868, 2026. [12] Meta. Prompt-guard-86m. https://huggingface.co/meta-llama/Promp t-Guard-86M, 2024. Accessed: 2026. [13] Andrianos Michail, Simon Clematide, and Rico Sennrich. Examining multilingual embedding models cross-lingually through LLM-generated adversarial examples. arXiv preprint arXiv:2502.08638, 2025. 13
[14] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. [15] Filippo Pallucchini, Lorenzo Malandri, Fabio Mercorio, and Mario Mezzanzanica. Lost in alignment: A survey on cross-lingual alignment methods for contextualized representation. ACM Computing Surveys, 58(5), 2025. [16] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2024. [17] Segal. SPIRE: Detecting prompt injection in zero-day using semantic matching. Engineering @ ActiveFence Blog, 2025. URL https://engineering.acti vefence.com. [18] Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. The language barrier: Dissecting safety challenges of LLMs in multilingual contexts. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2668–2680, Bangkok, Thailand, 2024. Association for Computational Linguistics. [19] Zheng-Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen Bach, and Julia Kreutzer. The state of multilingual llm safety research: From measuring the language gap to mitigating it. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15856–15871, 2025. [20] Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, et al. Qwen3Guard technical report. arXiv preprint arXiv:2510.14276, 2025.
14
A
Dataset Statistics Table 6: Datasets details by classes
Dataset
Total
Safe
Unsafe
Lang.
jayavibhav/prompt-injection-safety xTRam1/safe-guard-prompt-injection JailbreakBench/JBB-Behaviors nvidia/Aegis-AI-Content-Safety-Dataset-2.0
60,000 10,300 200 33,416
27,971 7,150 100 13,773
32,029 3,150 100 19643
EN EN EN EN
B
Metrics dependence by threshold
Table 7: Threshold selection and performance metrics on Benchmark №1 across input languages and translation methods for three optimisation criteria. Input Language
Threshold TPR (%)
FPR (%)
TNR (%) FNR (%)
Thresholds for FPR ≤ 1% English (native)
0.790
27.2
1.3
98.6
72.8
Russian (model) Russian (google)
0.740 0.750
24.3 24.2
1.4 1.3
98.6 98.6
75.6 75.7
Chinese (model) Chinese (google)
0.740 0.750
23.0 24.7
1.5 1.5
98.4 98.4
76.9 75.3
Arabic (model) Arabic (google)
0.730 0.740
23.5 24.3
1.7 1.7
98.2 98.2
76.5 75.6
Best thresholds for maximising TPR & TNR English (native)
0.725
78.0
28.4
71.6
22.0
Russian (model) Russian (google)
0.670 0.691
75.5 70.8
34.5 25.5
65.5 74.5
24.5 29.2
Chinese (model) Chinese (google)
0.665 0.677
74.6 78.5
33.5 36.4
66.5 63.6
25.4 21.5
Arabic (model) Arabic (google)
0.656 0.670
72.8 76.6
32.5 36.5
67.5 63.5
27.2 23.4
Best thresholds for minimising FPR & FNR English (native)
0.729
75.3
28.4
71.6
22.0
Russian (model) Russian (google)
0.675 0.691
71.1 70.8
29.5 25.5
70.5 74.5
28.9 29.2
Chinese (model) Chinese (google)
0.670 0.686
70.8 72.3
29.4 29.4
70.6 70.6
29.2 27.7
Arabic (model) Arabic (google)
0.659 0.677
71.0 71.0
30.4 30.4
69.6 69.6
29.0 29.0
15
Table 8: Threshold selection and performance metrics on Benchmark №2 across input languages and translation methods for three optimisation criteria.
Input Language
Threshold
TPR (%)
FPR (%)
TNR (%)
FNR (%)
Thresholds for FPR ≤ 1% English (native)
0.600
98.7
0.0
99.6
1.3
Russian (model) Russian (google)
0.700 0.700
70.9 73.2
0.1 0.1
99.9 99.9
29.1 26.8
Chinese (model) Chinese (google)
0.700 0.700
71.0 76.8
0.3 0.2
99.7 99.8
29.0 23.2
Arabic (model) Arabic (google)
0.700 0.700
71.0 69.4
0.3 0.2
99.7 99.8
29.0 30.6
Best thresholds for maximising TPR & TNR English (native)
0.600
98.7
0.4
99.6
1.3
Russian (model) Russian (google)
0.700 0.700
70.9 73.2
0.1 0.1
99.9 99.9
29.1 26.8
Chinese (model) Chinese (google)
0.700 0.700
71.0 76.8
0.3 0.2
99.7 99.8
29.0 23.2
Arabic (model) Arabic (google)
0.700 0.700
71.0 69.4
0.3 0.2
99.7 99.8
29.0 30.6
Best thresholds for minimising FPR & FNR English (native)
0.600
98.7
0.0
99.6
1.3
Russian (model) Russian (google)
0.700 0.700
70.9 73.2
0.1 0.1
99.9 99.9
29.1 26.8
Chinese (model) Chinese (google)
0.700 0.700
71.0 76.8
0.3 0.2
99.7 99.8
29.0 23.2
Arabic (model) Arabic (google)
0.700 0.700
71.0 69.4
0.3 0.2
99.7 99.8
29.0 30.6
16
Table 9: Threshold selection and performance metrics on Benchmark №3 across input languages and translation methods for three optimisation criteria.
Input Language
Threshold
TPR (%)
FPR (%)
TNR (%)
FNR (%)
Thresholds for FPR ≤ 1% English (native)
0.790
20.0
1.0
99.0
80.0
Russian (model) Russian (google)
0.780 0.750
12.0 18.0
1.0 1.0
99.0 99.0
88.0 82.0
Chinese (model) Chinese (google)
0.770 0.780
15.0 19.0
1.0 1.0
99.0 99.0
85.0 81.0
Arabic (model) Arabic (google)
0.760 0.750
13.0 16.0
0.0 0.0
100.0 100.0
87.0 84.0
Best thresholds for maximising TPR & TNR English (native)
0.714
41.0
17.0
83.0
59.0
Russian (model) Russian (google)
0.638 0.675
73.0 56.0
45.0 28.0
55.0 72.0
27.0 44.0
Chinese (model) Chinese (google)
0.637 0.665
77.0 67.0
47.0 37.0
53.0 63.0
23.0 33.0
Arabic (model) Arabic (google)
0.647 0.668
64.0 64.0
39.0 33.0
61.0 67.0
36.0 36.0
Best thresholds for minimising FPR & FNR English (native)
0.714
41.0
17.0
83.0
59.0
Russian (model) Russian (google)
0.652 0.675
64.0 56.0
38.0 28.0
62.0 72.0
36.0 44.0
Chinese (model) Chinese (google)
0.661 0.666
64.0 66.0
36.0 36.0
64.0 64.0
36.0 34.0
Arabic (model) Arabic (google)
0.648 0.668
63.0 64.0
38.0 33.0
62.0 67.0
37.0 36.0
17
Table 10: Threshold selection and performance metrics on Benchmark №4 across input languages and translation methods for three optimisation criteria.
Input Language
Threshold
TPR (%)
FPR (%)
TNR (%)
FNR (%)
Thresholds for FPR ≤ 1% English (native)
0.780
5.1
1.0
98.6
94.9
Russian (model) Russian (google)
0.730 0.720
5.9 8.6
1.4 1.5
98.5 98.5
94.1 91.4
Chinese (model) Chinese (google)
0.740 0.730
5.4 8.4
1.3 1.5
98.2 98.5
94.6 91.6
Arabic (model) Arabic (google)
0.720 0.720
7.1 7.4
1.3 1.2
98.7 98.7
92.9 92.5
Best thresholds for maximising TPR & TNR English (native)
0.619
49.9
29.3
70.7
50.1
Russian (model) Russian (google)
0.622 0.615
36.6 44.7
17.5 24.4
82.5 75.6
63.4 55.3
Chinese (model) Chinese (google)
0.603 0.630
48.5 39.8
33.5 20.2
66.5 79.8
51.5 60.2
Arabic (model) Arabic (google)
0.617 0.620
37.6 39.6
20.4 20.4
79.6 79.6
62.4 60.4
Best thresholds for minimising FPR & FNR English (native)
0.619
49.9
29.3
70.7
50.1
Russian (model) Russian (google)
0.622 0.615
36.6 44.7
17.5 24.4
82.5 75.6
63.4 55.3
Chinese (model) Chinese (google)
0.615 0.629
42.8 40.8
27.3 21.5
72.7 78.5
57.2 59.2
Arabic (model) Arabic (google)
0.617 0.618
37.6 41.3
20.4 21.5
79.6 78.5
62.4 58.7
18
C
Full ASR Results
Table 11: Benchmark 1. End-to-end mitigation results. Attacks0 denotes successful jailbreaks without filtering; Attacks1 denotes successful jailbreaks under defense. ∆Attacks = Attacks0 - Attacks1 . Attacks0
Attacks1
∆ Attacks
ASR0
ASR1
English (native) Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
14 214 60 106 48 140 36
Qwen3-4B 0 5 1 4 0 3 2
14 209 59 102 48 137 34
2.8% 42.8% 12.0% 21.2% 9.6% 28.0% 7.2%
0% 62.5% 25.0% 66.7% 0% 21.4% 20.0%
English (native) Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
Llama-3.2-3B-Instruct 45 0 45 131 6 125 38 0 38 119 5 114 67 2 65 161 7 154 90 4 86
9.0% 26.2% 7.6% 23.8% 13.4% 32.2% 18.0%
0% 75.0% 0% 83.3% 40.0% 50.0% 40.0%
English (native) Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
61 92 92 76 76 161 74
Input
gpt-3.5-turbo 0 5 1 2 0 7 2
19
61 87 91 74 76 154 72
12.2% 0% 18.4% 62.0% 18.4% 25.0% 15.2% 33.3% 15.2% 0% 32.2% 50.0% 14.8% 20.0%
Table 12: Benchmark 2. End-to-end mitigation results. Attacks0 denotes successful jailbreaks without filtering; Attacks1 denotes successful jailbreaks under defense. ∆Attacks = Attacks0 - Attacks1 . Attacks0
Attacks1
∆ Attacks
ASR0
ASR1
English (native) Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
38 94 49 73 41 70 37
Qwen3-4B 5 44 23 29 23 30 20
33 50 26 44 18 40 17
7.6% 18.8% 9.8% 14.6% 8.2% 14.0% 7.4%
18.5% 31.0% 16.9% 21.2% 43.9% 21.9% 12.8%
English (native) Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
Llama-3.2-3B-Instruct 75 3 72 106 36 70 69 28 41 114 41 73 77 25 52 136 49 87 53 12 41
15.0% 21.2% 13.8% 22.8% 15.4% 27.2% 10.6%
11.1% 23.4% 20.6% 29.9% 21.9% 35.8% 7.7%
English (native) Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
45 41 41 31 31 136 39
gpt-3.5-turbo 9 20 19 13 9 49 19
9.0% 8.2% 8.2% 6.2% 6.2% 27.2% 7.8%
33.3% 14.1% 14.0% 9.5% 7.9% 35.8% 12.2%
Input
20
36 21 22 18 22 87 20
Table 13: Benchmark 3. End-to-end mitigation results. Attacks0 denotes successful jailbreaks without filtering; Attacks1 denotes successful jailbreaks under defense. ∆Attacks = Attacks0 - Attacks1 . Attacks0
Attacks1
∆ Attacks
ASR0
ASR1
English Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
6 41 24 28 16 37 17
Qwen3-4B 5 18 14 13 7 24 6
1 23 10 15 9 13 11
1.2% 8.2% 4.8% 5.6% 3.2% 7.4% 3.4%
8.8% 42.9% 38.9% 36.1% 23.3% 46.2% 17.1%
English Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
Llama-3.2-3B-Instruct 10 5 5 32 11 21 24 14 10 39 13 26 16 7 9 39 21 18 17 6 11
2.0% 6.4% 4.8% 7.8% 3.2% 7.8% 3.4%
8.8% 26.2% 38.9% 36.1% 23.3% 40.4% 17.1%
English Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
22 25 25 34 34 39 21
gpt-3.5-turbo 14 13 12 15 13 21 10
4.4% 5.0% 5.0% 6.8% 6.8% 7.8% 4.2%
24.6% 31.0% 33.3% 41.7% 43.3% 40.4% 28.6%
Input
21
8 12 13 19 21 18 11
Table 14: Benchmark 4. End-to-end mitigation results. Attacks0 denotes successful jailbreaks without filtering; Attacks1 denotes successful jailbreaks under defense. ∆Attacks = Attacks0 - Attacks1 . Attacks0
Attacks1
∆ Attacks
ASR0
ASR1
English (native) Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
46 119 66 107 44 69 74
Qwen3-4B 35 92 51 88 36 60 61
11 27 15 19 8 9 13
9.2% 23.8% 13.2% 21.4% 8.8% 13.8% 14.8%
10.1% 23.4% 13.2% 23.2% 9.9% 15.0% 16.2%
English (native) Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
Llama-3.2-3B-Instruct 46 35 11 131 111 20 66 51 15 107 88 19 44 36 8 115 60 55 74 61 13
9.2% 26.2% 13.2% 21.4% 8.8% 23.0% 14.8%
10.0% 28.2% 13.2% 23.2% 9.9% 22.8% 16.2%
English (native) Russian (model) Russian (google) Chinese (model) Chinese (google) Arabic (model) Arabic (google)
39 37 37 34 34 115 35
gpt-3.5-turbo 26 26 27 22 22 91 23
7.8% 7.4% 7.4% 6.8% 6.8% 23.0% 7.0%
7.5% 6.6% 7.0% 5.8% 6.0% 22.8% 6.1%
Input
22
13 11 10 12 12 24 12
D
Results of the Codebook Sizes
Table 15: Detection performance across codebook subsampling ratios on Benchmark 2. Codebook Size
Language
TPR
FPR
TNR
FNR
25%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.8242 0.6107 0.6328 0.5668 0.6377 0.7037 0.6193
0.0072 0.0030 0.0051 0.0037 0.0032 0.0042 0.0028
0.9928 0.9970 0.9949 0.9963 0.9968 0.9958 0.9972
0.1758 0.3893 0.3672 0.4332 0.3623 0.2963 0.3807
50%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.8906 0.7176 0.7283 0.6865 0.7525 0.7934 0.7205
0.0116 0.0048 0.0077 0.0051 0.0044 0.0062 0.0046
0.9884 0.9952 0.9923 0.9949 0.9956 0.9938 0.9954
0.1094 0.2824 0.2717 0.3135 0.2475 0.2066 0.2795
75%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.9082 0.7451 0.7643 0.7102 0.7746 0.8295 0.7508
0.0113 0.0060 0.0083 0.0063 0.0056 0.0062 0.0058
0.9887 0.9940 0.9917 0.9937 0.9944 0.9938 0.9942
0.0918 0.2549 0.2357 0.2898 0.2254 0.1705 0.2492
100%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.9402 0.7086 0.7102 0.7102 0.8402 0.7684 0.6943
0.0183 0.0012 0.0026 0.0026 0.0088 0.0025 0.0016
0.9817 0.9988 0.9974 0.9974 0.9912 0.9975 0.9984
0.0598 0.2914 0.2898 0.2898 0.1598 0.2316 0.3057
23
Table 16: Detection performance across codebook subsampling ratios on Benchmark 3. Codebook Size
Language
TPR
FPR
TNR
FNR
25%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.6200 0.4300 0.4900 0.3400 0.4700 0.4900 0.4800
0.2800 0.1800 0.1700 0.1200 0.1300 0.2000 0.1300
0.7200 0.8200 0.8300 0.8800 0.8700 0.8000 0.8700
0.3800 0.5700 0.5100 0.6600 0.5300 0.5100 0.5200
50%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.6800 0.4700 0.4900 0.3700 0.5200 0.6100 0.5100
0.4100 0.1900 0.2700 0.2200 0.2400 0.3300 0.2500
0.5900 0.8100 0.7300 0.7800 0.7600 0.6700 0.7500
0.3200 0.5300 0.5100 0.6300 0.4800 0.3900 0.4900
75%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.7300 0.5100 0.5700 0.4100 0.5800 0.6400 0.5500
0.4400 0.2300 0.3000 0.2200 0.2800 0.3700 0.2600
0.5600 0.7700 0.7000 0.7800 0.7200 0.6300 0.7400
0.2700 0.4900 0.4300 0.5900 0.4200 0.3600 0.4500
100%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.8000 0.5800 0.6400 0.4800 0.6400 0.7000 0.6500
0.5500 0.3600 0.3700 0.3200 0.3900 0.4600 0.3700
0.4500 0.6400 0.6300 0.6800 0.6100 0.5400 0.6300
0.2000 0.4200 0.3600 0.5200 0.3600 0.3000 0.3500
24
Table 17: Detection performance across codebook subsampling ratios on Benchmark 4. Codebook Size
Language
TPR
FPR
TNR
FNR
25%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.1686 0.1156 0.1359 0.1108 0.1350 0.1621 0.1417
0.0550 0.0327 0.0329 0.0257 0.0258 0.0352 0.0246
0.9450 0.9673 0.9671 0.9743 0.9742 0.9648 0.9754
0.8314 0.8844 0.8641 0.8892 0.8650 0.8379 0.8583
50%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.2158 0.1484 0.1796 0.1435 0.1641 0.1990 0.1699
0.0819 0.0515 0.0599 0.0398 0.0516 0.0704 0.0434
0.9181 0.9485 0.9401 0.9602 0.9484 0.9296 0.9566
0.7842 0.8516 0.8204 0.8565 0.8359 0.8010 0.8301
75%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.2331 0.1638 0.1893 0.1522 0.1816 0.2136 0.1845
0.0901 0.0561 0.0599 0.0374 0.0552 0.0681 0.0493
0.9099 0.9439 0.9401 0.9626 0.9448 0.9319 0.9507
0.7669 0.8362 0.8107 0.8478 0.8184 0.7864 0.8155
100%
English (Original) Russian (Model) Chinese (Model) Arabic (Model) Russian (Google) Chinese (Google) Arabic (Google)
0.2881 0.2129 0.2350 0.2004 0.2350 0.2612 0.2301
0.1181 0.0749 0.0869 0.0667 0.0739 0.0951 0.0751
0.8819 0.9251 0.9131 0.9333 0.9261 0.9049 0.9249
0.7119 0.7871 0.7650 0.7996 0.7650 0.7388 0.7699
25
E
Results of the Embedders Analysis
Table 18: Benchmark №2. Detailed threshold metrics for all models (separate translation methods). Part 1 Language
Model
AUC
Goal
Threshold
TPR
FPR
TNR
Precision
J/F1
English
bge-m3
0.9929
Youden’s J Fixed FPR F1-Max
0.6492 0.6789 0.6658
95.3% 91.9% 93.5%
2.6% 1.0% 1.4%
97.4% 99.0% 98.6%
94.1% 97.6% 96.6%
J=0.927 — F1=0.950
e5-large
0.9976
Youden’s J Fixed FPR F1-Max
0.8672 0.8736 0.8779
96.9% 95.2% 94.5%
1.9% 1.0% 0.4%
98.1% 99.0% 99.6%
95.7% 97.7% 99.0%
J=0.950 — F1=0.967
jina-v3
0.9953
Youden’s J Fixed FPR F1-Max
0.6916 0.7209 0.6966
97.0% 93.8% 96.4%
2.1% 1.0% 1.7%
97.9% 99.0% 98.3%
95.3% 97.6% 96.1%
J=0.950 — F1=0.963
bge-m3
0.9850
Youden’s J Fixed FPR F1-Max
0.6139 0.6568 0.6274
93.1% 82.9% 90.7%
4.6% 1.0% 3.0%
95.4% 99.0% 97.0%
89.7% 97.4% 92.9%
J=0.885 — F1=0.918
e5-large
0.9773
Youden’s J Fixed FPR F1-Max
0.8425 0.8533 0.8449
89.3% 82.3% 87.9%
3.6% 1.0% 2.6%
96.4% 99.0% 97.4%
91.3% 97.3% 93.5%
J=0.857 — F1=0.906
jina-v3
0.9864
Youden’s J Fixed FPR F1-Max
0.6542 0.6871 0.6542
93.1% 85.5% 93.1%
3.4% 1.0% 3.4%
96.6% 99.0% 96.6%
92.1% 97.4% 92.1%
J=0.896 — F1=0.926
bge-m3
0.9876
Youden’s J Fixed FPR F1-Max
0.6112 0.6558 0.6292
94.8% 85.3% 91.6%
5.2% 1.0% 2.7%
94.8% 99.0% 97.3%
88.6% 97.4% 93.5%
J=0.896 — F1=0.925
e5-large
0.9749
Youden’s J Fixed FPR F1-Max
0.8370 0.8508 0.8453
90.4% 82.8% 85.9%
5.2% 0.9% 1.9%
94.8% 99.1% 98.1%
88.2% 97.4% 95.1%
J=0.852 — F1=0.903
jina-v3
0.9890
Youden’s J Fixed FPR F1-Max
0.6424 0.6794 0.6573
95.7% 88.6% 92.8%
4.1% 1.0% 2.5%
95.9% 99.0% 97.5%
90.9% 97.5% 94.2%
J=0.916 — F1=0.935
bge-m3
0.9810
Youden’s J Fixed FPR F1-Max
0.6178 0.6674 0.6240
93.0% 80.1% 91.4%
5.7% 1.0% 4.5%
94.3% 99.0% 95.5%
87.5% 97.2% 89.7%
J=0.873 — F1=0.906
e5-large
0.9696
Youden’s J Fixed FPR F1-Max
0.8447 0.8543 0.8481
88.8% 74.8% 84.1%
8.0% 1.0% 4.3%
92.0% 99.0% 95.7%
82.7% 97.0% 89.3%
J=0.808 — F1=0.866
jina-v3
0.9871
Youden’s J Fixed FPR F1-Max
0.6678 0.7026 0.6747
94.1% 85.5% 93.0%
4.4% 1.0% 3.3%
95.6% 99.0% 96.7%
90.3% 97.4% 92.3%
J=0.898 — F1=0.927
Russian (model)
Russian (google)
Chinese (model)
26
Table 19: Benchmark №2. Detailed threshold metrics for all models (separate translation methods). Part 2 Language
Model
AUC
Goal
Threshold
TPR
FPR
TNR
Precision
J/F1
Chinese (google)
bge-m3
0.9906
Youden’s J Fixed FPR F1-Max
0.6277 0.6617 0.6424
94.7% 87.7% 92.2%
3.6% 1.0% 2.0%
96.4% 99.0% 98.0%
91.9% 97.4% 95.1%
J=0.911 — F1=0.936
e5-large
0.9796
Youden’s J Fixed FPR F1-Max
0.8449 0.8545 0.8487
91.3% 76.8% 85.8%
7.1% 1.0% 3.7%
92.9% 99.0% 96.3%
84.7% 97.1% 90.9%
J=0.843 — F1=0.883
jina-v3
0.9913
Youden’s J Fixed FPR F1-Max
0.6785 0.6976 0.6818
95.2% 91.3% 95.0%
2.4% 1.0% 2.1%
97.6% 99.0% 97.9%
94.4% 97.5% 95.0%
J=0.928 — F1=0.950
bge-m3
0.9773
Youden’s J Fixed FPR F1-Max
0.6186 0.6577 0.6262
90.0% 78.5% 88.2%
5.3% 1.0% 3.9%
94.7% 99.0% 96.1%
87.9% 97.2% 90.8%
J=0.847 — F1=0.894
e5-large
0.9822
Youden’s J Fixed FPR F1-Max
0.8400 0.8546 0.8445
93.0% 81.6% 90.7%
5.1% 1.0% 3.3%
94.9% 99.0% 96.7%
88.7% 97.3% 92.1%
J=0.879 — F1=0.914
jina-v3
0.9724
Youden’s J Fixed FPR F1-Max
0.6343 0.6778 0.6549
90.5% 79.3% 85.7%
6.3% 1.0% 2.7%
93.7% 99.0% 97.3%
86.0% 97.2% 93.1%
J=0.841 — F1=0.893
bge-m3
0.9844
Youden’s J Fixed FPR F1-Max
0.6211 0.6578 0.6248
92.2% 82.3% 91.4%
3.9% 1.0% 3.4%
96.1% 99.0% 96.6%
91.0% 97.3% 92.1%
J=0.883 — F1=0.917
e5-large
0.9915
Youden’s J Fixed FPR F1-Max
0.8361 0.8558 0.8449
96.7% 84.8% 92.7%
5.5% 1.0% 2.5%
94.5% 99.0% 97.5%
88.4% 97.4% 94.1%
J=0.913 — F1=0.934
jina-v3
0.9833
Youden’s J Fixed FPR F1-Max
0.6402 0.6719 0.6524
92.1% 85.6% 90.2%
3.9% 1.0% 2.2%
96.1% 99.0% 97.8%
91.1% 97.4% 94.5%
J=0.883 — F1=0.923
Arabic (model)
Arabic (google)
27
Table 20: Benchmark №3. Detailed threshold metrics for all models (separate translation methods). Language English
Russian (model)
Russian (google)
Chinese (model)
Chinese (google)
Arabic (model)
Arabic (google)
Model
AUC
Goal
Threshold
TPR
FPR
TNR
Precision
J/F1
bge-m3
0.7191
Youden’s J Fixed FPR F1-Max
0.7148 0.8055 0.6181
56.0% 23.0% 93.0%
22.0% 1.0% 68.0%
78.0% 99.0% 32.0%
71.8% 95.8% 57.8%
J=0.340 — F1=0.713
e5-large
0.7083
Youden’s J Fixed FPR F1-Max
0.8980 0.9286 0.8826
59.0% 15.0% 80.0%
22.0% 0.0% 51.0%
78.0% 100.0% 49.0%
72.8% 100.0% 61.1%
J=0.370 — F1=0.693
jina-v3
0.7000
Youden’s J Fixed FPR F1-Max
0.7282 0.8414 0.6250
60.0% 19.0% 95.0%
28.0% 1.0% 78.0%
72.0% 99.0% 22.0%
68.2% 95.0% 54.9%
J=0.320 — F1=0.696
bge-m3
0.6732
Youden’s J Fixed FPR F1-Max
0.6380 0.7759 0.6192
73.0% 14.0% 82.0%
45.0% 1.0% 57.0%
55.0% 99.0% 43.0%
61.9% 93.3% 59.0%
J=0.280 — F1=0.686
e5-large
0.6846
Youden’s J Fixed FPR F1-Max
0.8548 0.9006 0.8548
81.0% 10.0% 81.0%
52.0% 1.0% 52.0%
48.0% 99.0% 48.0%
60.9% 90.9% 60.9%
J=0.290 — F1=0.695
jina-v3
0.7173
Youden’s J Fixed FPR F1-Max
0.6691 0.7944 0.6037
53.0% 6.0% 84.0%
17.0% 0.0% 49.0%
83.0% 100.0% 51.0%
75.7% 100.0% 63.2%
J=0.360 — F1=0.721
bge-m3
0.6933
Youden’s J Fixed FPR F1-Max
0.6764 0.7620 0.6085
54.0% 18.0% 87.0%
26.0% 0.0% 63.0%
74.0% 100.0% 37.0%
67.5% 100.0% 58.0%
J=0.280 — F1=0.696
e5-large
0.7127
Youden’s J Fixed FPR F1-Max
0.8706 0.9119 0.8519
58.0% 6.0% 89.0%
21.0% 0.0% 63.0%
79.0% 100.0% 37.0%
73.4% 100.0% 58.6%
J=0.370 — F1=0.706
jina-v3
0.7369
Youden’s J Fixed FPR F1-Max
0.6397 0.7896 0.6350
69.0% 6.0% 70.0%
25.0% 1.0% 26.0%
75.0% 99.0% 74.0%
73.4% 85.7% 72.9%
J=0.440 — F1=0.714
bge-m3
0.6925
Youden’s J Fixed FPR F1-Max
0.6374 0.7688 0.5880
77.0% 17.0% 93.0%
47.0% 1.0% 70.0%
53.0% 99.0% 30.0%
62.1% 94.4% 57.1%
J=0.300 — F1=0.707
e5-large
0.6866
Youden’s J Fixed FPR F1-Max
0.8388 0.8723 0.8369
77.0% 9.0% 82.0%
48.0% 1.0% 55.0%
52.0% 99.0% 45.0%
61.6% 90.0% 59.9%
J=0.290 — F1=0.692
jina-v3
0.7129
Youden’s J Fixed FPR F1-Max
0.6334 0.8114 0.6058
81.0% 10.0% 92.0%
47.0% 0.0% 64.0%
53.0% 100.0% 36.0%
63.3% 100.0% 59.0%
J=0.340 — F1=0.719
bge-m3
0.7021
Youden’s J Fixed FPR F1-Max
0.6661 0.7794 0.5818
66.0% 21.0% 97.0%
36.0% 1.0% 77.0%
64.0% 99.0% 23.0%
64.7% 95.5% 55.7%
J=0.300 — F1=0.708
e5-large
0.6685
Youden’s J Fixed FPR F1-Max
0.8477 0.8808 0.8203
55.0% 7.0% 99.0%
23.0% 0.0% 95.0%
77.0% 100.0% 5.0%
70.5% 100.0% 51.0%
J=0.320 — F1=0.674
jina-v3
0.7333
Youden’s J Fixed FPR F1-Max
0.6769 0.7906 0.6185
64.0% 13.0% 89.0%
29.0% 1.0% 57.0%
71.0% 99.0% 43.0%
68.8% 92.9% 61.0%
J=0.350 — F1=0.724
bge-m3
0.6608
Youden’s J Fixed FPR F1-Max
0.6480 0.7575 0.5472
63.0% 17.0% 98.0%
37.0% 0.0% 84.0%
63.0% 100.0% 16.0%
63.0% 100.0% 53.8%
J=0.260 — F1=0.695
e5-large
0.7053
Youden’s J Fixed FPR F1-Max
0.8622 0.8938 0.8509
61.0% 10.0% 85.0%
29.0% 1.0% 53.0%
71.0% 99.0% 47.0%
67.8% 90.9% 61.6%
J=0.320 — F1=0.714
jina-v3
0.6981
Youden’s J Fixed FPR F1-Max
0.6725 0.7761 0.5952
48.0% 11.0% 87.0%
16.0% 1.0% 63.0%
84.0% 99.0% 37.0%
75.0% 91.7% 58.0%
J=0.320 — F1=0.696
bge-m3
0.7038
Youden’s J Fixed FPR F1-Max
0.6682 0.7426 0.6248
64.0% 21.0% 84.0%
33.0% 1.0% 57.0%
67.0% 99.0% 43.0%
66.0% 95.5% 59.6%
J=0.310 — F1=0.697
e5-large
0.7338
Youden’s J Fixed FPR F1-Max
0.8620 0.9082 0.8582
71.0% 5.0% 79.0%
29.0% 0.0% 43.0%
71.0% 100.0% 57.0%
71.0% 100.0% 64.8%
J=0.420 — F1=0.712
jina-v3
0.7494
Youden’s J Fixed FPR F1-Max
0.6196 0.7415 0.5800
75.0% 16.0% 90.0%
31.0% 1.0% 56.0%
69.0% 99.0% 44.0%
70.8% 94.1% 61.6%
J=0.440 — F1=0.732
28
Table 21: Benchmark №4. Detailed threshold metrics for all models (separate translation methods). Language English
Russian (model)
Russian (google)
Chinese (model)
Chinese (google)
Arabic (model)
Arabic (google)
Model
AUC
Goal
Threshold
TPR
FPR
TNR
Precision
J/F1
bge-m3
0.6199
Youden’s J Fixed FPR F1-Max
0.6186 0.8050 0.4734
49.9% 3.3% 98.9%
29.2% 0.9% 95.8%
70.8% 99.1% 4.2%
67.4% 81.0% 55.6%
J=0.207 — F1=0.712
e5-large
0.6345
Youden’s J Fixed FPR F1-Max
0.8496 0.9031 0.7978
50.5% 3.9% 99.8%
29.5% 0.9% 98.2%
70.5% 99.1% 1.8%
67.5% 83.7% 55.2%
J=0.210 — F1=0.711
jina-v3
0.5907
Youden’s J Fixed FPR F1-Max
0.6785 0.8131 0.4842
31.8% 3.8% 99.1%
13.5% 0.9% 97.4%
86.5% 99.1% 2.6%
74.2% 83.0% 55.3%
J=0.183 — F1=0.710
bge-m3
0.6178
Youden’s J Fixed FPR F1-Max
0.6222 0.7366 0.4627
36.6% 5.2% 98.7%
17.2% 0.9% 95.4%
82.8% 99.1% 4.6%
72.1% 87.1% 55.7%
J=0.194 — F1=0.712
e5-large
0.6428
Youden’s J Fixed FPR F1-Max
0.8322 0.8705 0.7532
50.7% 6.1% 100.0%
26.9% 0.9% 98.9%
73.1% 99.1% 1.1%
69.6% 88.7% 55.1%
J=0.238 — F1=0.711
jina-v3
0.5708
Youden’s J Fixed FPR F1-Max
0.6440 0.7612 0.4798
30.2% 3.9% 99.4%
16.8% 0.9% 98.4%
83.2% 99.1% 1.6%
68.5% 83.7% 55.1%
J=0.133 — F1=0.709
bge-m3
0.6305
Youden’s J Fixed FPR F1-Max
0.6222 0.7435 0.4736
40.9% 5.8% 98.7%
20.2% 0.9% 95.8%
79.8% 99.1% 4.2%
71.0% 88.2% 55.5%
J=0.207 — F1=0.710
e5-large
0.6265
Youden’s J Fixed FPR F1-Max
0.8371 0.8737 0.7486
42.8% 5.0% 100.0%
22.7% 0.9% 99.1%
77.3% 99.1% 0.9%
69.6% 86.4% 55.0%
J=0.202 — F1=0.709
jina-v3
0.6063
Youden’s J Fixed FPR F1-Max
0.6551 0.7455 0.4621
29.5% 6.5% 100.0%
11.2% 0.9% 98.7%
88.8% 99.1% 1.3%
76.2% 89.3% 55.1%
J=0.184 — F1=0.710
bge-m3
0.5952
Youden’s J Fixed FPR F1-Max
0.6342 0.7554 0.4348
33.9% 4.5% 99.7%
17.8% 0.8% 98.6%
82.2% 99.2% 1.4%
69.7% 86.8% 55.0%
J=0.160 — F1=0.709
e5-large
0.5346
Youden’s J Fixed FPR F1-Max
0.8220 0.8675 0.7778
62.9% 1.7% 100.0%
54.9% 0.8% 99.2%
45.1% 99.2% 0.8%
58.1% 70.8% 54.9%
J=0.080 — F1=0.709
jina-v3
0.5647
Youden’s J Fixed FPR F1-Max
0.6771 0.7433 0.4902
21.6% 5.7% 99.8%
10.2% 0.9% 98.8%
89.8% 99.1% 1.2%
71.8% 88.1% 55.0%
J=0.113 — F1=0.709
bge-m3
0.6194
Youden’s J Fixed FPR F1-Max
0.6301 0.7556 0.4525
39.9% 5.3% 99.6%
20.2% 0.7% 98.5%
79.8% 99.3% 1.5%
70.5% 90.2% 55.0%
J=0.197 — F1=0.709
e5-large
0.5059
Youden’s J Fixed FPR F1-Max
0.8134 0.8751 0.7743
79.2% 2.0% 100.0%
74.6% 0.8% 99.5%
25.4% 99.2% 0.5%
56.2% 75.0% 54.8%
J=0.046 — F1=0.708
jina-v3
0.5990
Youden’s J Fixed FPR F1-Max
0.6260 0.7610 0.5039
52.0% 6.0% 98.9%
37.4% 0.8% 97.1%
62.6% 99.2% 2.9%
62.7% 89.9% 55.2%
J=0.146 — F1=0.709
bge-m3
0.6102
Youden’s J Fixed FPR F1-Max
0.6296 0.7272 0.4349
32.7% 6.1% 99.6%
15.0% 0.9% 97.9%
85.0% 99.1% 2.1%
72.6% 88.7% 55.3%
J=0.177 — F1=0.711
e5-large
0.6180
Youden’s J Fixed FPR F1-Max
0.8285 0.8659 0.7446
44.4% 6.4% 100.0%
26.2% 0.8% 98.4%
73.8% 99.2% 1.6%
67.3% 90.4% 55.2%
J=0.182 — F1=0.712
jina-v3
0.5744
Youden’s J Fixed FPR F1-Max
0.6472 0.7251 0.4377
25.4% 6.6% 100.0%
11.0% 0.9% 99.9%
89.0% 99.1% 0.1%
73.7% 89.5% 54.9%
J=0.144 — F1=0.709
bge-m3
0.6121
Youden’s J Fixed FPR F1-Max
0.6184 0.7273 0.4457
41.3% 6.4% 99.2%
21.4% 0.8% 97.4%
78.6% 99.2% 2.6%
70.0% 90.4% 55.2%
J=0.199 — F1=0.709
e5-large
0.5871
Youden’s J Fixed FPR F1-Max
0.8258 0.8783 0.7520
47.7% 2.9% 99.7%
33.5% 0.8% 98.4%
66.5% 99.2% 1.6%
63.3% 81.1% 55.1%
J=0.142 — F1=0.710
jina-v3
0.5871
Youden’s J Fixed FPR F1-Max
0.6414 0.7271 0.4456
31.8% 7.3% 100.0%
16.2% 0.9% 99.8%
83.8% 99.1% 0.2%
70.4% 90.4% 54.8%
J=0.157 — F1=0.708
29