C OMPARISON OF M ODERN M ULTILINGUAL T EXT E MBEDDING T ECHNIQUES FOR H ATE S PEECH D ETECTION TASK
arXiv:2604.14907v1 [cs.CL] 16 Apr 2026
A P REPRINT Evaldas Vaičiukynas* 1 , Paulius Danėnas† 2 , Linas Ablonskis‡ 1 , Algirdas Šukys§ 1 , Edgaras Dambrauskas¶ 2 , Voldemaras Žitkus|| 1 , Rita Butkienė** 1 , and Rimantas Butleris†† 1,2 1 Department of Information Systems, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania 2 Centre of Information Systems Design Technologies, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania
April 17, 2026
A BSTRACT Online hate speech and abusive language pose a growing challenge for content moderation, especially in multilingual settings and for low-resource languages such as Lithuanian. This paper investigates to what extent modern multilingual sentence embedding models can support accurate hate speech detection in Lithuanian, Russian, and English, and how their performance depends on downstream modeling choices and feature dimensionality. We introduce LtHate, a new Lithuanian hate speech corpus derived from news portals and social networks, and benchmark six modern multilingual encoders (potion, gemma, bge, snow, jina, e5) on LtHate, RuToxic, and EnSuperset using a unified Python pipeline. For each embedding, we train both a one class HBOS anomaly detector and a two class CatBoost classifier, with and without principal component analysis (PCA) compression to 64-dimensional feature vectors. Across all datasets, two class supervised models consistently and substantially outperform one class anomaly detection, with the best configurations achieving up to 80.96% accuracy and AUC ROC of 0.887 in Lithuanian (jina), 92.19% accuracy and AUC ROC of 0.978 in Russian (e5), and 77.21% accuracy and AUC ROC of 0.859 in English (e5 with PCA). PCA compression preserves almost all discriminative power in the supervised setting, while showing some negative impact for the unsupervised anomaly detection case. These results demonstrate how modern multilingual sentence embeddings combined with gradient boosted decision trees provide robust soft-computing solutions for multilingual hate speech detection applications. Keywords Hate speech detection · Anomaly detection · Sentence embeddings · Dimensionality reduction
1
Introduction
The rapid proliferation of digital communication platforms has fundamentally transformed the way societies interact, debate, and share information. While social media, online news portals, and public forums have democratized access to public discourse, they have simultaneously become vectors for the dissemination of hate speech — language that attacks, demeans, or incites violence against individuals or groups on the basis of protected characteristics such as race, ethnicity, religion, gender, sexual orientation, or national origin [1, 2]. The scale and urgency of this phenomenon are ∗
[email protected] [email protected] ‡ [email protected] § [email protected] ¶ [email protected] ∥ [email protected] ∗∗ [email protected] †† [email protected] †
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
staggering: a 2023 global survey conducted by UNESCO and Ipsos across 16 countries found that 67% of internet users have personally encountered hate speech online, with the prevalence rising to 74% among users under the age of 35 [3]. Similarly, a European Union survey has reported that approximately 80% of respondents in the EU have encountered hate speech in online environments [4]. These figures underscore that online hate speech is not a marginal or isolated problem but a pervasive feature of the contemporary digital landscape that affects billions of users worldwide. The societal consequences of unchecked online hate speech extend far beyond individual psychological harm. At the individual level, exposure to hateful content has been associated with increased anxiety, depression, social withdrawal, and, in extreme cases, suicidal ideation [4, 5]. At the community level, persistent hate speech fosters social polarization, normalizes bigotry, and erodes the quality of public discourse [4]. More alarmingly, a growing body of empirical research has established direct links between the spread of online hate speech and real-world violence. Müller and Schwarz [6] demonstrated that anti-refugee hate speech on Facebook causally predicted violent crimes against refugees in German municipalities, with the effect disappearing during major platform outages. The role of social media in facilitating mass atrocities has been tragically illustrated in Myanmar, where Facebook was identified by a United Nations fact-finding mission as an instrument for those seeking to spread hate against the Rohingya Muslim minority, contributing to a campaign of ethnic cleansing that displaced over 700,000 people [7]. In light of such evidence, the need for scalable and reliable automated hate speech detection systems has become a pressing concern for governments, platform operators, and civil society alike. The regulatory environment has evolved accordingly. The European Union’s Digital Services Act (DSA), which became fully applicable in February 2024, imposes explicit obligations on large online platforms to address illegal content — including hate speech — through a combination of automated detection, human review, and transparent reporting mechanisms [8]. In January 2025, the European Commission integrated the revised Code of Conduct on countering illegal hate speech online into the DSA framework, further strengthening the requirements for proactive content moderation. These regulatory developments have intensified the demand for automated content moderation tools that are accurate across linguistic and cultural contexts, scalable to the volume of user-generated content (which exceeds millions of posts per minute on major platforms), and robust against adversarial evasion strategies [9]. From a natural language processing (NLP) perspective, hate speech detection is typically framed as a supervised text classification problem. The field has evolved rapidly from early approaches based on handcrafted features and classical machine learning algorithms — such as bag-of-words representations paired with support vector machines or logistic regression [10, 11] — to deep learning architectures leveraging word embeddings [12, 13] and, more recently, pretrained transformer-based language models [14–16]. Fine-tuned variants of BERT [17] and its multilingual counterpart mBERT, as well as cross-lingual models such as XLM-RoBERTa [18], have achieved state-of-the-art performance on several English-language hate speech benchmarks and have demonstrated the ability to transfer detection capabilities across languages via shared multilingual representation spaces [19–21]. Despite these advances, several critical challenges remain. First, the overwhelming majority of hate speech detection research has concentrated on English-language data, leaving most of the world’s approximately 7,000 languages without adequate detection tools or annotated resources [1, 2, 9]. Recent surveys have catalogued over 60 publicly available hate speech training datasets, yet the vast majority are English-centric, and only a handful cover languages from Central and Eastern Europe, the Baltics, or other underrepresented linguistic communities [9, 22]. For low-resource languages, the scarcity of annotated corpora, the absence of language-specific NLP pre-processing tools, and the limited coverage of pre-trained models compound the difficulty of building effective detection systems [23, 24]. Lithuanian is a prototypical example of such a low-resource scenario: despite the existence of active online communities and documented instances of online hate speech in the Lithuanian digital sphere, systematic studies on automated hate speech detection in Lithuanian have only recently begun to appear in the literature [25]. Second, modern multilingual sentence embedding models have emerged as a promising paradigm for cross-lingual and multilingual text classification tasks. Models such as Multilingual E5 [26], Jina Embeddings [27], Snowflake Arctic [28], and BGE-M3 [29] encode texts from dozens or hundreds of languages into a shared vector space, enabling downstream classifiers to operate on fixed-dimensional representations without requiring language-specific fine-tuning. However, systematic comparisons of these modern off-the-shelf embedding models for hate speech detection — particularly in multilingual settings that include low-resource languages — remain scarce. Most existing studies either focus on a single encoder, employ task-specific fine-tuning that obscures the contribution of the base representation, or evaluate only on well-resourced languages. This gap motivates a rigorous, controlled comparison of modern embedding techniques across diverse linguistic settings under a unified experimental protocol. Third, practical deployment of hate speech detection systems in real-world content moderation pipelines imposes stringent constraints on computational efficiency, memory footprint, and inference latency. Dimensionality reduction techniques, such as Principal Component Analysis (PCA), offer a straightforward mechanism for compressing highdimensional embeddings while potentially preserving discriminative information. Understanding the trade-off between
2
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
compression and detection performance is critical for designing systems that can operate at scale without prohibitive computational costs, yet this aspect has received limited systematic attention in the hate speech detection literature. In this paper, we address these gaps by investigating the effectiveness of several recent multilingual text embedding models for hate speech detection in three languages: Lithuanian, Russian, and English. We focus on sentence-level vector representations produced by six modern multilingual encoders — Potion, Gemma, BGE, Snowflake Arctic, Jina Embeddings (v3), and Multilingual E5 (large-instruct) — and evaluate them in combination with a two-class CatBoost supervised classifier and a one-class HBOS anomaly detection approach across three binary hate speech datasets, with and without PCA-based dimensionality reduction. Our main scientific contributions are as follows: • We devise a unified experimental framework for benchmarking multilingual sentence embeddings on multiple hate speech datasets. • We prepare and release LtHate, a 12k-comment Lithuanian hate speech corpus with topical and severity annotations. • We report a systematic comparison of six recent multilingual embedding models across Lithuanian, Russian and English hate speech corpora, with and without PCA-based compression. • We provide practical recommendations for model and embedding selection under computational constraints in multilingual moderation systems.
2
Related work
Section overviews attempts at using multilingual text embeddings for hate speech detection in various languages. 2.1
Hate speech and offensive language detection
Hate speech detection has gained a lot of attention from the research community since core natural language processing techniques were established and applied. Schmidt & Wiegand [10] provide one of the first structured overviews of early automatic hate speech detection methods using NLP, covering classical machine learning approaches and highlighting early challenges and features (e.g., bag-of-words, lexicons). One of the more important milestones in the area of hate speech research was benchmark dataset and taxonomy introduction by Davidson et al. in 2017 [11]. Initial results were obtained by training a suite of traditional classifiers, such as logistic regression, naïve Bayes, decision trees, random forests, and support vector machines, resulting in best overall F1-score of 0.90 achieved by logistic regression while revealing a persistent confusion between hate speech and offensive content. Twitter tweets were another source for hate speech corpus [30], which led to introducing first deep learning systems based on convolutional neural network (CNN) and multiple representations [12]. The best model used word2vec semantic embeddings achieved an F1-score of 78.3% on a four-class problem, demonstrating that learned distributed representations substantially outperform surface-form features for hate speech classification. This was further confirmed in [13] by comparing CNN, LSTM, and FastText architectures against traditional TF-IDF and bag-of-words vector baselines on a benchmark of 16,000 annotated tweets. Their results showed that LSTM models whose embeddings were subsequently used to train gradient-boosted classifiers significantly outperformed outperformed state-of-the-art character and word n-gram methods. The introduction of transformer architecture [14] led to substantial improvements over prior sequence models as it relies exclusively on self-attention mechanisms to model dependencies between arbitrary positions in a sequence in constant time. The proposed architecture also enabled creating fine-tuned versions of the original model which usually outperformed the original. Mozafari et al. [15] were among the first to systematically investigate BERT fine-tuning for hate speech classification in a peer-reviewed open-access venue using BERT-Base to both Davidson [11] and Waseem and Hovy [30] datasets. By proposing a regularisation-based reweighting mechanism applied during fine-tuning, their model substantially outperformed prior deep learning baselines and helped to mitigate systematic racial biases. Further, [16] followed this direction with HateBERT pretrained on a large-scale corpus of Reddit comments containing offensive, abusive, or hateful content. Comparative experiments across three English benchmarks for abusive language detection (OffensEval, AbusEval, and HatEval) showed that HateBERT consistently outperformed the corresponding general BERT model on each of them. The OffensEval dataset [31] has since been adopted as one of de facto standards for evaluating offensive language, and later extended to multilingual settings in OffensEval-2020 [19], which introduced parallel datasets in Arabic, Danish, Greek, and Turkish. This led to evaluation of multilingual transformer architectures such as mBERT and XLM-RoBERTa. Across all languages, the dominant strategy among top-performing systems was fine-tuning XLM-R on the target-language training data, either alone or in combination with language-specific pre-trained models confirming the model’s cross-lingual generalisation capability. 3
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
Beyond shared tasks, a growing body of work has explored the application of multilingual transformer models to heterogeneous, real-world multilingual detection settings. [21] addressed the HASOC 2020 challenge on hate speech and offensive content identification in English, German, and Hindi using a two-stage hierarchical classification architecture built on mBERT and XLM-R. Their system first identified if content is hateful or offensive versus non-offensive, then classified detected toxic content into hate speech, offensive language, or profanity, exploiting the shared multilingual representation space of the underlying encoder across all three languages simultaneously. Results demonstrated that multilingual transformer encoders could successfully transfer information across typologically distinct languages within a single fine-tuned model. The authors additionally found that incorporating hashtag and emoji-aware tokenisation improved performance on Twitter data, where non-verbal signals frequently modify or intensify the meaning of hateful text. Performance of multilingual transformer models also inspired research in cross-lingual transfer as a substitute for in-language annotation in settings where labelled hate speech data is scarce or entirely absent. Bigoulaeva et al. [23] evaluated cross-lingual transfer from high-resource source-language English to German for which only limited labelled examples were available. Using bilingual word embeddings that aligned the representation spaces of the two languages, they demonstrated that zero-shot transfer from English to German was feasible and competitive with in-language supervised baselines when training data was minimal. They also showed that using the transferred model’s predictions to generate pseudo-labels for unlabelled German data and then training on those labels substantially improved over the zero-shot baseline, establishing a practical and low-cost pathway to extending detection to new languages without full annotation campaigns. However, a subsequent and more comprehensive study [24] identified structural and cultural divergence between source and target language hate speech conventions — including differences in target group definitions, slur conventions, and expression of implicit versus explicit hatred - as the primary bottleneck for effective zero-shot transfer, as well as proposed targeted data selection strategies to partially mitigate these effects. Nevertheless, further research also confirmed the capability to project knowledge from English to other languages with cross-lingual contextual embeddings, reducing dependence on language-specific labeled data and enabling transfer to low-resource settings [20], as well as outperforming general-purpose embeddings in cross-lingual classification scenarios using domain-specific multilingual hate speech embeddings [32]. Recurrent neural architectures on FastText embeddings were also adapted to multilingual hate speech detection task across English, Italian, and German languages in [33]. Awal et al. proposed HateMAML, a meta-learning framework to improve cross-lingual transfer of hate speech classifiers in low-resource languages by adapting pretrained language models to new languages with limited data. The adoption of cross-lingual transfer in low-resource scenarios has been extensively researched in the context of multiple languages, including Arabic [34], Turkish [34], Indian languages [35–37]. Lithuanian language has only recently gained attention, with the latest study exploring transfer learning and transformer-based architectures on newly created annotated corpora of 27358 user-generated comments [25]. Research results indicated that multilingual transformer models, like Multilingual BERT, LitLat BERT or Electra, can reach competitive accuracy and F1-scores. 2.2
Multilingual transformer models
Multilingual embeddings encode semantic information from words, sentences, paragraphs or documents across multiple languages in a shared vector space, allowing models to compare and transfer linguistic knowledge between languages. This is foundational to multilingual Natural Language Processing (NLP) tasks such as cross-lingual retrieval, semantic similarity, or machine translation. Early embedding techniques like word2vec [38] demonstrated the power of dense vector representations for capturing semantic relationships within a single language, as well as revealed capabilities to learn semantic relationships and vector arithmetic analogies from distributional statistics of text corpora. Cross-lingual and multilingual embeddings extend this approach by representing words from multiple languages in a common space where semantically equivalent words are close together regardless of the language. Early cross-lingual methods often used linear mappings trained with bilingual dictionaries or parallel corpora to align monolingual embeddings. These mapping-based models were simple and computationally efficient, and they enabled cross-lingual lexical tasks such as bilingual lexicon induction and document classification. The early survey by Ruder et al. [39] provides a comprehensive taxonomy cross-lingual word embedding models, classifying techniques for word, sentence and document level alignments, as well as optimization objectives. These techniques were later extended to multilingual settings, handling many languages in a unified space rather than just pairs. Other researchers moved beyond mapping approaches, by developing unsupervised neural language models that jointly train on raw multilingual corpora and exploit structural similarities across languages to align representations in a common space [40][41]. This idea was also extended to capture semantics across a wide variety of languages simultaneously, thus learning rich multilingual representations. The rise of pre-trained multilingual transformer models completely changed the landscape of the existing multilingual embedding techniques and shifted this domain toward large, open-source transformer-based models with broad language coverage and flexible training objectives. Models like multilingual BERT (mBERT) [17] and XLM-R [18] leverage large amounts of text data from dozens or hundreds of languages to learn contextual embeddings that generalize 4
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
across languages. These models are trained with self-supervised objectives (e.g., masked language modelling) on multilingual corpora and often yield strong zero-shot transfer performance. LaBSE (Language-agnostic BERT Sentence Embedding) [42] extends multilingual transformer pre-training to sentence-level representations trained on parallel translation data, enabling semantically comparable sentence embeddings across more than 100 languages. Multilingual E5 models [43] extend the original E5 recipe by contrastive pre-training on billions of multilingual sentence pairs followed by supervised fine-tuning and instruction tuning, yielding strong retrieval and similarity performance across many languages; the core architecture remains encoder-focused with contrastive objectives, which makes them efficient and robust but sometimes less adaptable in long-context scenarios. Qwen3-Embedding series [44] builds on the Qwen3 LLM backbone with a dense transformer architecture and multi-stage training including synthetic weak supervision, supervised fine-tuning, and model merging; these models support 100+ languages, benefitting from instruction-aware embedding generation and strong cross-lingual capabilities. Jina-Embeddings (v3/v4) use a transformer foundation with task-specific LoRA adapters and Matryoshka Representation Learning to produce flexible, high-quality embeddings for retrieval, clustering, and long-context tasks. Approaches combining or fusing representations from multiple pre-trained models indicate that embedding choice and combination strategy significantly affect hate speech detection performance, although fusion can yield only modest gains relative to its computational cost. Other studies consider multilingual and multimodal settings, where text is combined with images and cultural context to improve detection of hateful memes and visually grounded content. In contrast to prior work that either fine-tunes a single multilingual encoder or constructs custom domain-specific embeddings, our study compares several modern off-the-shelf sentence embedding models (Potion, Snowflake, Jina, Multilingual-E5) across multiple languages, using a fixed downstream classifier and a standardized evaluation protocol.
3
Hate speech datasets
Diversity of hate speech datasets enables us to assess whether the same embedding methods and downstream machine learning models are effective across various languages and dataset sizes. Lithuanian corpora - LtHate LtHate [45] is a new hate speech corpus for the Lithuanian language. It consists of public media comments taken from Litis [46] corpus and other public media sources. Comments from Litis corpus are sourced from two of the biggest Lithuanian online news portals in years 2010 to 2014. Comments from other media sources are spanning years 2021 to 2024 and were sourced from various social media platforms in the Lithuanian language and Lithuanian news portals. The topical composition of the corpus was inspired by the methodology described in [47]. We have chosen five subjects of hate speech: a) ethnicity, nationality and race; b) gender and sexual orientation; c) country and state; d) political views and e) religion. For each subject category sets of neutral and loaded samples were collected. For each loaded sample we have noted a target of the hate and the level of the hate. The levels are 1 to 4 with 1 corresponding to expressions of contempt, 4 corresponding to outright incitement to violence against the target and 2,3 being in between. Since other corpora used in this research do not have the same structure, for the experiment described in this paper, we reduced LtHate corpus to binary labels of neutral or hate speech only and aggregated entries that represent multiple labelings of the same comment due to multiple distinct targets being present. Resulting corpus contains 5577 neutral and 6477 loaded comments in 5 categories of subjects (see Table 1) with a total of 12054 comments, slightly skewed towards hateful class, corresponding to 53.73% of corpus samples. Table 1: Distribution of LtHate corpus with respect to subject categories. Subject category
Neutral
Hate speech
Total comments
ethnicity, nationality and race gender and sexual orientation country and state political views religion
2143 177 433 1858 966
1796 865 463 2516 837
3939 1042 896 4374 1803
Russian corpora - RuToxic. RuToxic is a publicly available Russian-language dataset of 163187 user comments annotated for toxicity, which has been introduced in offensive or toxic language detection research [48]. Target class (toxic/hateful comments) here comprise 19.25% of the dataset, therefore some weak class imbalance exist. RuToxic provides a complementary perspective on Slavic-language hate speech and toxicity, allowing us to test whether multilingual embeddings can capture similar patterns across Lithuanian and Russian data. 5
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
English corpora - EnSuperset. The English dataset EnSuperset aggregates many public hate speech and offensive language corpora into a unified binary classification benchmark [22], containing 360493 comments. Texts originate from English social media and online platforms, with annotations indicating presence or absence of hateful or offensive content. After harmonization of label schemes, we use a binary label 0/1 and retain only relevant textual fields ("text", "labels"). Target class encompass 27.04% of a dataset. EnSuperset is substantially larger than LtHate and RuToxic, providing a high-resource English setting against which multilingual embeddings can be evaluated.
4
Methodology
Machine learning pipeline in Python was devised with text pre-processing, vectorization and training of models using 10-fold stratified cross-validation strategy on various hate speech datasets, and is available as open-source code at https://github.com/evavaic/KTU-Misijos-HIPSTer. Optionally, dimensionality reduction with PCA is applied independently for each cross-validation training split. 4.1
Text pre-processing
All datasets investigated for hate speech detection task are processed using an identical and shared pipeline implemented in Python. Each text comment is first passed through a fix_punctuation function that removes exclamation marks and: • normalizes encoding using ftfy package; • removes hyperlinks; • collapses repeated punctuation marks while preserving limited emphasis; • standardizes spacing around punctuation and numbers, and • replaces emojis with shortcode text using the emoji package. The resulting cleaned texts are then processed with text vectorization technique to obtain feature vectors suitable for machine learning model training and testing. 4.2
Sentence embeddings
Table 2: Description of selected multilingual embedding models. Note: MRL corresponds to Matryoshka Representation Learning, which allows truncation to lower dimensionality for resulting original feature vectors. Embedding name potion gemma bge snow jina e5
Dimensions
MRL
Model size
Model link (in huggingface.co platform)
256 768 1024 1024 1024 1024
No Yes No Yes Yes No
128 M 308 M 569 M 568 M 572 M 560 M
minishlab/potion-multilingual-128M google/embeddinggemma-300m BAAI/bge-m3 Snowflake/snowflake-arctic-embed-l-v2.0 jinaai/jina-embeddings-v3 intfloat/multilingual-e5-large-instruct
We compare six multilingual sentence embedding techniques for text vectorization (see Table 2). Embedding models are loaded via SentenceTransformer package, with Jina embeddings requiring trust_remote_code=True setting. Texts are encoded in batches with configurable batch size, and embeddings are concatenated into a matrix with 256 dimensionality for potion, 768 for gemma and and 1024 for the remaining vectorization techniques. minishlab/potion-multilingual-128M [49], [50] is a 0.128B parameter multilingual text embedding model distilled from BAAI/bge-m3 [51] and refined on the C4 dataset [52], [53] by applying TokenLearn method [54]. Its notable features are high speed in comparison to larger models and unrestricted input length, due to internal averaging of embeddings of individual tokens. It has an output dimensionality of 256. google/embeddinggemma-300m [55], [56] is a 0.3B parameter multilingual text embedding model. The composition of dataset that the model is trained on is not disclosed, however the model does well on MMTEB [57] benchmark which includes Lithuanian language. The model uses RoPE positional encodings [58] and supports inputs up to 2048 tokens. It has an output dimensionality of 768 with matryoshka [59] points at 512, 256 and 128 dimensions. 6
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
BAAI/bge-m3 [29], [51] is a 0.56B parameter text embedding model specifically trained for retrieval tasks. It is trained and refined on datasets [60], [61] that include most widespread European languages. The model supports inputs up to 8194 tokens. It has an output dimensionality of 1024. Snowflake/snowflake-arctic-embed-l-v2.0 [28], [62] is a 0.6B parameter multilingual text embedding model based on transformer encoder architecture and trained on MIRACL dataset [63] which contains 18 languages. The model uses RoPE positional encodings [58] and supports inputs up to 8194 tokens. It has an output dimensionality of 1024, with a single matryoshka [59] point at 256 dimensions. jinaai/jina-embeddings-v3 [27], [64] is a 0.6B parameter multilingual text embedding model internally combining a transformer based encoder with 5 task specific LoRA [65] adapters. The model is trained on 100 languages and fine-tuned on 30 languages. Interestingly, one of those 30 languages is Latvian, which is highly similar to Lithuanian. The model uses RoPE positional encodings [58] and supports inputs up to 8194 tokens. It has an output dimensionality of 1024 with matryoshka [59] points at 32, 64, 128, 256, 512 and 768 dimensions. intfloat/multilingual-e5-large-instruct [26], [66] is a 0.6B parameter multilingual text embedding model based on transformer encoder architecture with weights initialized from XLM-RoBERTa large [18], [67] which is trained on 100 languages, including Lithuanian. The model then was additionally trained on datasets coming from public media and fine tuned on curated high quality datasets including one used in [68]. The model supports inputs up to 514 tokens. It has an output dimensionality of 1024. 4.3
Dimensionality reduction
To analyze the trade-off between performance and compactness, we train models on both original embeddings and their compressed variants. Principal Component Analysis (PCA) [69] is a linear dimensionality reduction technique that finds a new set of orthogonal axes (principal components) that capture as much of the variance in the original feature space as possible in descending order of importance, thus providing a compressed projection of the original data. By retaining only the first 64 principal components after PCA, we obtain a compact feature vector representation that concentrates most of the variance of the original embeddings. This reduces storage and computational costs for downstream machine learning models while also acting as a form of linear noise filtering: directions with very low variance, which often correspond to noise or redundant information, are discarded. PCA in experiments here is fitted only to the training data of each cross-validation split to avoid information leakage and consequently the learned transformation is applied to compress both training and test embeddings within that CV split. 4.4
Detection models
In experiments we consider two types of downstream models for detection task: 1. One-class (1c) anomaly detection. Histogram-based outlier score (HBOS) [70] model from PyOD (Python Outlier Detection) package, used as a one-class approach trained on the target class (hate speech) examples only. HBOS method models each feature independently using a histogram and due to its linear complexity is suitable for very large datasets, being significantly faster than many other multivariate outlier detection methods. The outlier score is estimated based on density corresponding to histogram bin each feature falls into with lower density values indicating anomalous instance. Contamination rate was fixed at 0.01, and output from model was rescaled by dividing from 10000 (for original feature vectors) or from 100 (for PCA-transformed feature vectors) to get a score resembling class probability prediction. 2. Two-class (2c) supervised classification. Gradient boosting CatBoost classifier [71] with 500 maximum iterations, learning rate 0.05, depth 8, LogLoss loss function, and scale_pos_weight set to the ratio of negatives to positives in the training data (to address class imbalance). Within each cross-validation iteration, 80% of the training set is used to fit the model and 20% is held out for early stopping with a patience of 30 iterations where the best CatBoost model with respect to validation AUC ROC metric is retained. Although datasets had binary annotation of both target class (hate speech) and non-target class (neutral speech) examples, which is required to evaluate detection success on test folds, difference between model variants used was in training step where construction of model used data from both classes (2c case) or only data from a target class (1c case). Such selection of methods allows a direct comparison of text embedding quality under a strong two-class (2c) supervised and a weaker one-class (1c) pseudo-unsupervised setting. In practice this would correspond to the scope of annotation efforts where for the one-class case collection of only hate speech examples should be sufficient to create a detector. 7
Multilingual Text Embeddings for Hate Speech Detection
4.5
A P REPRINT
Evaluation metrics
Machine learning experiment success was assessed using k-fold stratified cross-validation (CV) using 10 folds (k=10) and stratified by target attribute, where machine learning model is trained on all except one fold with that one fold left out to test model inference. After pooling model outputs for all test folds and comparing them to ground-truth class labels various accuracy metrics were calculated in a micro-average fashion. To summarize detection performance, the following metrics were used: • Area under the receiver operating characteristic curve (AUC ROC), which corresponds to the probability that a randomly chosen non-target class instance will have a smaller estimated target class probability than a randomly chosen target class instance [72]. In short, AUC summarizes the probability of correctly ranking a (neutral, hate speech) pair of text examples based on detector’s output and is directly related to the Wilcoxon Mann-Whitney U statistic. • Precision-Recall curve (PRC) also allows calculating the area under the curve (AUC PRC) and for the case of large class imbalance, the PRC is recommended over ROC [73] when choosing a better performing detector. • Overall accuracy (Accuracy) - the most known evaluation metric, calculating proportion of correct predictions to the ground-truth classes: TP + TN (1) TP + FP + TN + FN where TP is true positives (correct target class predictions), TN is true negatives (correct non-target class predictions), FP is false positives (incorrect predictions of target class examples), and FN is false negatives (incorrect predictions of non-target class examples). All these counts correspond to absolute frequences from the confusion matrix, obtained after applying treshold to model’s prediction (output probability of target class). • Kappa [74, 75] - accuracy, corrected for class imbalance: p0 − p e 1 − pe
(2)
where pe – the sum of probabilities of predictions agreeing with the ground-truth by chance, p0 – overall accuracy of the model. According to the academic literature, the Kappa value of 0.21 - 0.40 indicates a fair agreement and 0.41 - 0.60 a moderate agreement. Higher values correspond to a substantial and almost perfect agreement result. Plot-based AUC ROC and AUC PRC metrics are calculated using model’s raw outputs before thresholding. To calculate the remaining metrics, one needs to obtain a confusion matrix by using a threshold on model’s raw outputs to convert soft decision (class probability) to hard decision (class prediction). Since for the ad-hoc choice of 0.5 is usually suboptimal, we’ve selected a more effective - equal error rate - operating point where ROC curve intersects with diagonal and class recall metrics become equal, namely, specificity becomes approximately equal to sensitivity and, consequently, to overall accuracy.
5
Experimental results
In this section we outline the main findings organized by dataset. Detailed ROC and PRC curves and accuracy metrics tables — for original feature vectors and for feature vectors after PCA transformation — are presented here with an overview of the results. 5.1
Lithuanian dataset results
Machine learning for Lithuanian language hate speech dataset results are in Table 3 and Figures 1-2. One-class classification (see top part of Table 3) resulted in 52.34% – 63.57% accuracy for original and 50.02% – 60.52% accuracy for PCA-compressed embeddings. Two-class classification (see bottom part of Table 3) resulted in 74.37% – 80.96% accuracy for original and 73.64% – 79.60% accuracy for PCA-compressed embeddings. Two-class supervised classification clearly outperformed one-class anomaly detection with only a negligible differences between original and PCA-compressed embeddings in the two-class case. Detection effectiveness for Lithuanian hate speech, as measured by ROC/PRC curves and accuracy metrics, was highest for Jina embeddings, resulting in AUC ROC of 0.887 and AUC PRC of 0.895 for two-class classification (see Fig. 8
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
Table 3: Summary of hate speech detection success for Lithuanian language dataset. Method
Accuracy (%) Orig. PCA
Kappa Orig. PCA
AUC ROC Orig. PCA
AUC PRC Orig. PCA
1c potion 1c gemma 1c bge 1c snow 1c jina 1c e5
52.34 54.46 56.53 57.88 63.57 60.02
50.02 54.11 55.13 56.41 60.52 57.64
0.047 0.089 0.130 0.157 0.270 0.200
0.001 0.082 0.102 0.128 0.209 0.152
0.529 0.561 0.588 0.609 0.689 0.640
0.508 0.560 0.574 0.588 0.645 0.605
0.552 0.588 0.622 0.629 0.707 0.662
0.543 0.580 0.596 0.605 0.655 0.623
2c potion 2c gemma 2c bge 2c snow 2c jina 2c e5
75.13 74.37 78.68 79.08 80.96 79.07
73.92 73.64 78.08 77.95 79.60 78.05
0.501 0.486 0.572 0.580 0.618 0.580
0.477 0.471 0.560 0.558 0.591 0.560
0.830 0.826 0.874 0.874 0.887 0.876
0.818 0.815 0.865 0.863 0.877 0.864
0.843 0.840 0.886 0.885 0.895 0.887
0.831 0.829 0.878 0.876 0.887 0.875
Figure 1: Lithuanian language hate speech detection curves using original embeddings: ROC (left) and PRC (right).
1) and PCA transformation did not affect this result noticeably (see Fig. 2). Other very competitive embeddings in two-class case were bge, snow and e5. However, gemma embeddings demonstrated surprisingly bad result and even were slightly outperformed by lower dimensional potion embeddings in two-class case. Slightly above moderate agreement between predicted class and ground truth (best Kappa=0.62 for jina embeddings) demonstrates average success in hate speech detection for Lithuanian language dataset. 5.2
Russian dataset results
Machine learning for Russian language hate speech dataset results are in Table 4 and Figures 3-4. One-class classification (see top part of Table 4) resulted in 67.71% – 80.82% accuracy for original and 59.40% – 75.28% accuracy for PCAcompressed embeddings. Two-class classification (see bottom part of Table 4) resulted in 85.41% – 92.19% accuracy for original and 84.97% – 91.74% accuracy for PCA-compressed embeddings. Two-class supervised classification clearly outperformed one-class anomaly detection with only a negligible differences between original and PCA-compressed embeddings in two-class case. 9
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
Figure 2: Lithuanian hate speech detection curves using compressed embeddings: ROC (left) and PRC (right). Table 4: Summary of hate speech detection success for Russian language dataset. Method
Accuracy (%) Orig. PCA
Kappa Orig. PCA
AUC ROC Orig. PCA
AUC PRC Orig. PCA
1c potion 1c gemma 1c bge 1c snow 1c jina 1c e5
67.71 79.00 74.50 69.18 76.15 80.82
59.40 70.87 67.87 66.12 69.87 75.28
0.254 0.462 0.374 0.279 0.405 0.500
0.126 0.308 0.257 0.228 0.291 0.389
0.741 0.869 0.816 0.767 0.841 0.890
0.626 0.776 0.741 0.720 0.763 0.831
0.422 0.666 0.569 0.467 0.622 0.712
0.290 0.462 0.436 0.380 0.445 0.579
2c potion 2c gemma 2c bge 2c snow 2c jina 2c e5
85.41 89.41 90.99 91.04 91.07 92.19
84.97 88.73 90.33 90.52 90.62 91.74
0.601 0.698 0.739 0.740 0.741 0.770
0.591 0.681 0.722 0.727 0.729 0.759
0.934 0.961 0.970 0.971 0.972 0.978
0.930 0.957 0.967 0.968 0.969 0.975
0.814 0.885 0.906 0.912 0.909 0.931
0.802 0.873 0.895 0.903 0.901 0.924
Detection effectiveness for Russian hate speech, as measured by ROC/PRC curves and accuracy metrics, was highest for E5 embeddings, resulting in AUC ROC of 0.978 and AUC PRC of 0.931 for two-class classification (see Fig. 3) and PCA transformation did not affect it noticeably (see Fig. 4). Other embeddings (jina, snow, bge) also were very competitive to e5 in two-class case, with the gemma embedding performing only slightly worse. Worst performance was for snow and potion embeddings. Substantial agreement between predicted class and ground truth (best Kappa=0.77 for e5 embeddings) demonstrates high success in hate speech detection for Russian language dataset. 5.3
English dataset results
Machine learning for English language hate speech dataset results are in Table 5 and Figures 5-6. One-class classification (see top part of Table 5) resulted in 55.27% – 64.22% accuracy for original and 51.32% – 56.63% accuracy for PCAcompressed embeddings. Two-class classification (see bottom part of Table 5) resulted in 70.14% – 76.95% accuracy for original and 69.55% – 77.21% accuracy for PCA-compressed embeddings (see bottom part of Table 5). Two-class 10
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
Figure 3: Russian hate speech detection curves using original embeddings: ROC (left) and PRC (right).
Figure 4: Russian language hate speech detection curves using compressed embeddings: ROC (left) and PRC (right).
supervised classification clearly outperformed one-class anomaly detection with only a negligible differences between original and PCA-compressed embeddings in two-class case. Detection effectiveness for English hate speech, as measured by ROC/PRC curves and accuracy metrics, was highest for E5 embeddings, resulting in AUC ROC of 0.855 and AUC PRC of 0.698 for two-class classification (see Fig. 5) and PCA transformation even improved this result slightly (see Fig. 6). Other very competitive embeddings in two-class case were gemma, snow and jina, with surprisingly good results for lower-dimensional gemma embeddings. Moderate agreement between predicted class and ground truth (best Kappa=0.48 for e5 embeddings) demonstrates below average success in hate speech detection for English language dataset. 11
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
Table 5: Summary of hate speech detection success for English language dataset. Method
Accuracy (%) Orig. PCA
Kappa Orig. PCA
AUC ROC Orig. PCA
AUC PRC Orig. PCA
1c potion 1c gemma 1c bge 1c snow 1c jina 1c e5
55.27 66.69 60.25 60.98 61.94 64.22
51.32 58.01 52.03 55.14 55.84 56.63
0.085 0.283 0.169 0.182 0.198 0.239
0.021 0.131 0.032 0.083 0.094 0.108
0.572 0.731 0.647 0.663 0.671 0.702
0.516 0.613 0.526 0.575 0.583 0.593
0.319 0.484 0.379 0.417 0.416 0.464
0.289 0.376 0.291 0.328 0.343 0.338
2c potion 2c gemma 2c bge 2c snow 2c jina 2c e5
70.14 76.37 74.50 75.89 75.35 76.95
69.55 76.67 74.10 76.00 75.02 77.21
0.347 0.468 0.431 0.459 0.448 0.480
0.336 0.474 0.423 0.461 0.442 0.485
0.777 0.848 0.828 0.842 0.837 0.855
0.770 0.851 0.824 0.844 0.834 0.859
0.570 0.679 0.650 0.668 0.657 0.698
0.558 0.683 0.644 0.670 0.655 0.705
Figure 5: English language hate speech detection curves using original embeddings: ROC (left) and PRC (right). 5.4
Overview of all results
Across the three hate speech datasets and six multilingual embedding models, several consistent patterns can be observed. First, two-class (2c) supervised CatBoost classifiers systematically and substantially outperform one-class (1c) HBOS anomaly detectors in terms of accuracy, Kappa, AUC ROC, and AUC PRC for all languages and embedding families. For example, on Lithuanian LtHate, accuracy increases from 63.57% (best 1c, jina) to 80.96% (best 2c, jina), while on Russian RuToxic it increases from 80.82% (best 1c, e5) to 92.19% (best 2c, e5) and on English EnSuperset from 66.69% (best 1c, gemma) to 76.95% (best 2c, e5). These results suggest that even when high-quality negative examples are more expensive to annotate strong supervised approach should be preferred whenever a reasonably balanced labeled dataset can be constructed. Second, detection effectiveness depends strongly on both language and embedding model. For Lithuanian LtHate, the best performing configuration combines Jina embeddings with the 2c CatBoost classifier, reaching accuracy of 80.96%, Kappa of 0.618, AUC ROC of 0.887, and AUC PRC of 0.895. Russian RuToxic achieves the highest overall scores in the study, with e5 + 2c CatBoost attaining 92.19% accuracy, Kappa of 0.770, AUC ROC of 0.978, and AUC PRC of 0.931. For English EnSuperset, e5 again provides the strongest results in the 2c setting with 76.95% accuracy, Kappa of 0.480, AUC ROC of 0.855, and AUC PRC of 0.698, slightly improving further when PCA-compressed embeddings are 12
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
Figure 6: English language hate speech detection curves using compressed embeddings: ROC (left) and PRC (right). used. Overall, modern large multilingual encoders (jina, e5, snow, bge, gemma) consistently outperform the compact potion model, although potion remains surprisingly competitive given its much smaller dimensionality and parameter count. Third, the relative ranking of embedding models is not fully consistent across languages, reflecting differences in training data coverage and linguistic similarity. Jina embeddings are clearly superior for Lithuanian LtHate in the 2c setting, whereas e5 dominates on both Russian RuToxic and English EnSuperset. Gemma embeddings behave somewhat atypically, yielding only mid-tier performance on Lithuanian (even slightly worse than potion in the 2c case) while becoming one of the top-performing models on English EnSuperset and achieving strong 1c performance on RuToxic. This observation highlights that embedding models with excellent global multilingual benchmarks do not necessarily transfer uniformly across all lower-resource target languages and domains. Fourth, PCA-based dimensionality reduction to 64 components preserves most of the discriminative information for the 2c CatBoost classifiers. Across all datasets and embedding models, accuracy, Kappa, AUC ROC, and AUC PRC values for original and PCA-compressed representations differ only marginally in the 2c setting, often within one percentage point. For example, for Lithuanian LtHate with jina embeddings, AUC ROC drops only from 0.887 to 0.877 and AUC PRC from 0.895 to 0.887, while for Russian RuToxic with e5 the AUC ROC remains at 0.978 vs. 0.975 and AUC PRC at 0.931 vs. 0.924. However, in the 1c HBOS scenario PCA compression can noticeably degrade performance, particularly for RuToxic and EnSuperset, indicating that fine-grained density information is more important for histogram-based anomaly scoring than for gradient-boosted decision trees. Finally, there are clear differences in achievable performance across languages and datasets. Russian RuToxic, which is relatively large and has a moderate class imbalance, yields the highest scores (best Kappa = 0.770), suggesting that current multilingual embeddings can model Russian toxic language patterns particularly well under a supervised 2c setup. Lithuanian LtHate attains lower but still competitive results (best Kappa = 0.618), reflecting both its smaller size and the increased difficulty of modeling a newly constructed low-resource language hate speech corpus. English EnSuperset yields intermediate performance (best Kappa = 0.485 with PCA-compressed e5 embeddings in 2c case), which is slightly lower than might be expected for English but may be explained by the heterogeneity of source corpora and label schemes that were harmonized into a single binary benchmark.
6
Discussion & Conclusions
In this paper, we presented a comparative study of six modern multilingual sentence embedding models — potion, gemma, bge, snow, jina, and e5 — for hate speech detection in Lithuanian, Russian, and English. We introduced LtHate, a new Lithuanian hate speech corpus with detailed topical and severity annotations that we reduced to a 13
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
binary classification setting for experiments, and we evaluated all embedding models in both one-class (HBOS) and two-class (CatBoost) configurations with and without PCA-based dimensionality reduction. Experimental results show that contemporary multilingual encoders combined with a simple gradient-boosted classifier can achieve moderate to substantial agreement with human annotations across all three languages, with the strongest performance observed for Russian and competitive results for the newly created Lithuanian dataset. From a practical perspective, the experiments suggest several recommendations for multilingual hate speech detection systems. Whenever it is feasible to obtain labeled examples for both hateful and non-hateful categories, a two-class supervised setup with CatBoost (or similar gradient-boosting methods) should be preferred over purely one-class anomaly detection, as the latter consistently lags behind across datasets and metrics. Among the embedding models, jina embeddings appear to be the most suitable choice for Lithuanian LtHate, whereas e5 embeddings provide the best overall performance for Russian RuToxic and English EnSuperset; other embeddings (snow, bge, gemma) remain competitive alternatives. Additionally, our findings indicate that applying PCA to reduce embeddings to 64 principal components yields almost no loss in 2c classification performance, offering a straightforward way to lower memory and computation costs in real-world deployments. At the same time, several limitations of the current study point to directions for future work. First, we focused exclusively on off-the-shelf sentence encoders without any task-specific fine-tuning, meaning that further gains are likely achievable via supervised or contrastive adaptation on in-domain hate speech corpora. Second, our experiments considered only binary hate vs. neutral labels, whereas LtHate and many existing datasets provide richer taxonomies (e.g., fine-grained target groups, severity levels), and modeling these distinctions may be necessary for more nuanced moderation decisions. Third, the current setup is text-only and does not incorporate multi-modal information such as images, emojis beyond textual normalisation, or conversation context, which are often crucial in real-world hateful or abusive content. Therefore, an important direction for further research would be systematic evaluation of instruction-tuned large language models in zero-shot and few-shot classification regimes, as well as hybrid architectures combining frozen multilingual embedding encoders with lightweight adapters fine-tuned on hate speech and toxicity detection tasks. Also, integrating explainability techniques and bias assessments into the evaluation protocol will be essential for understanding and mitigating potential harms when deploying multilingual hate speech detectors in high-stakes, real-world moderation scenarios. Code availability All code used to implement the unified Python pipeline for pre-processing, multilingual sentence embeddings, PCA compression, and model training is publicly available at https://github.com/evavaic/KTU-Misijos-HIPSTer.
Acknowledgements This work was conducted as part of the execution of the project "Mission-driven Implementation of Science and Innovation Programs" (No. 02-002-P-0001), funded by the Economic Revitalization and Resilience Enhancement Plan "New Generation Lithuania".
References [1] Paula Fortuna and Sérgio Nunes. A survey on automatic detection of hate speech in text. ACM Computing Surveys, 51(4):85:1–85:30, 2018. doi:10.1145/3232676. [2] Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation, 55:477–523, 2021. doi:10.1007/s10579-020-09502-8. [3] UNESCO and Ipsos. Survey on the impact of online disinformation and hate speech, 2023. URL https: //www.unesco.org/sites/default/files/medias/fichiers/2023/11/unesco_ipsos_survey.pdf. [4] María Antonia Paz, Julio Montero-Díaz, and Alicia Moreno-Delgado. Hate speech: A systematized review. SAGE Open, 13(1):1–18, 2023. doi:10.1177/21582440231181311. [5] James Hawdon, Atte Oksanen, and Pekka Räsänen. Exposure to online hate in four nations: A cross-national consideration. Deviant Behavior, 38(3):254–266, 2017. doi:10.1080/01639625.2016.1196985. [6] Karsten Müller and Carlo Schwarz. Fanning the flames of hate: Social media and hate crime. Journal of the European Economic Association, 19(4):2131–2167, 2021. doi:10.1093/jeea/jvaa045. 14
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
[7] United Nations Human Rights Council. Report of the independent international fact-finding mission on Myanmar, 2018. URL https://www.ohchr.org/en/hr-bodies/hrc/myanmar-ffm/index. A/HRC/39/64. [8] European Parliament and Council of the European Union. Regulation (EU) 2022/2065 of the European Parliament and of the Council on a single market for digital services and amending Directive 2000/31/EC (Digital Services Act), 2022. URL https://eur-lex.europa.eu/eli/reg/2022/2065/oj. Official Journal of the European Union, L 277, 1–102. [9] Bertie Vidgen and Leon Derczynski. Directions in abusive language training data, a systematic review: Garbage in, garbage out. PLOS ONE, 15(12):e0243300, 2020. doi:10.1371/journal.pone.0243300. [10] Anna Schmidt and Michael Wiegand. A survey on hate speech detection using natural language processing. In Lun-Wei Ku and Cheng-Te Li, editors, Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, SocialNLP@EACL 2017, Valencia, Spain, April 3, 2017, pages 1–10. Association for Computational Linguistics, 2017. doi:10.18653/V1/W17-1101. [11] Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the Eleventh International Conference on Web and Social Media, ICWSM 2017, Montréal, Québec, Canada, May 15-18, 2017, pages 512–515. AAAI Press, 2017. URL https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15665. [12] Björn Gambäck and Utpal Kumar Sikdar. Using convolutional neural networks to classify hate-speech. In Zeerak Waseem, Wendy Hui Kyong Chung, Dirk Hovy, and Joel R. Tetreault, editors, Proceedings of the First Workshop on Abusive Language Online, ALW@ACL 2017, Vancouver, BC, Canada, August 4, 2017, pages 85–90. Association for Computational Linguistics, 2017. doi:10.18653/V1/W17-3013. [13] Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. Deep learning for hate speech detection in tweets. In Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich, editors, Proceedings of the 26th International Conference on World Wide Web Companion, Perth, Australia, April 3-7, 2017, pages 759–760. ACM, 2017. doi:10.1145/3041021.3054223. [14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017. URL http://arxiv.org/abs/1706.03762. [15] Marzieh Mozafari, Reza Farahbakhsh, and Noël Crespi. Hate speech detection and racial bias mitigation in social media based on bert model. PLOS ONE, 15(8):1–26, 08 2020. doi:10.1371/journal.pone.0237861. [16] Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. Hatebert: Retraining bert for abusive language detection in english. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 17–25. Association for Computational Linguistics, 2021. doi:10.18653/v1/2021.woah-1.3. [17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. doi:10.48550/arXiv.1810.04805. [18] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116, 2019. doi:10.48550/arXiv.1911.02116. [19] Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). In Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider, Jonathan May, and Ekaterina Shutova, editors, Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1425–1447, Barcelona (online), December 2020. International Committee for Computational Linguistics. doi:10.18653/v1/2020.semeval-1.188. URL https://aclanthology.org/2020.semeval-1.188/. [20] Tharindu Ranasinghe and Marcos Zampieri. Multilingual offensive language identification with cross-lingual embeddings. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5838–5844, Online, November 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.emnlp-main.470. [21] Sayar Ghosh Roy, Ujwal Narayan, Tathagata Raha, Zubair Abid, and Vasudeva Varma. Leveraging multilingual transformers for hate speech detection. In Parth Mehta, Thomas Mandl, Prasenjit Majumder, and Mandar Mitra, editors, Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, Hyderabad, India, December 16-20, 2020, volume 2826 of CEUR Workshop Proceedings, pages 128–138. CEUR-WS.org, 2020. URL https://ceur-ws.org/Vol-2826/T2-4.pdf. [22] Manuel Tonneau, Diyi Liu, Samuel Fraiberger, Ralph Schroeder, Scott A. Hale, and Paul Röttger. From languages to geographies: Towards evaluating cultural bias in hate speech datasets. In Yi-Ling Chung, Zeerak Talat, Debora 15
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
Nozza, Flor Miriam Plaza-del Arco, Paul Röttger, Aida Mostafazadeh Davani, and Agostina Calabrese, editors, Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), pages 283–311, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.woah-1.23. [23] Irina Bigoulaeva, Viktor Hangya, and Alexander Fraser. Cross-lingual transfer learning for hate speech detection. In Bharathi Raja Chakravarthi, John P. McCrae, Manel Zarrouk, Rajeev K. Bali, and Paul Buitelaar, editors, Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, LT-EDI@EACL 2021, Online, April 19, 2021, pages 15–25. Association for Computational Linguistics, 2021. URL https: //www.aclweb.org/anthology/2021.ltedi-1.3/. [24] Irina Bigoulaeva, Viktor Hangya, Iryna Gurevych, and Alexander Fraser. Addressing the challenges of crosslingual hate speech detection. CoRR, abs/2201.05922, 2022. doi:10.48550/arxiv.2201.05922. [25] Justina Mandravickaitė, Eglė Rimkienė, Mindaugas Petkevičius, Milita Songailaitė, Eimantas Zaranka, and Tomas Krilavičius. Exploring hate speech detection models for Lithuanian language. In Agostina Calabrese, Christine de Kock, Debora Nozza, Flor Miriam Plaza-del Arco, Zeerak Talat, and Francielle Vargas, editors, Proceedings of the The 9th Workshop on Online Abuse and Harms (WOAH), pages 206–218, Vienna, Austria, August 2025. Association for Computational Linguistics. ISBN 979-8-89176-105-6. [26] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual E5 text embeddings: A technical report. arXiv, 2024. doi:10.48550/arXiv.2402.05672. [27] Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and Han Xiao. jina-embeddings-v3: Multilingual embeddings with task lora. arXiv, 2024. doi:10.48550/arXiv.2409.10173. [28] Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. Arctic-embed 2.0: Multilingual retrieval without compromise. arXiv, 2024. doi:10.48550/arXiv.2412.04506. [29] Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3-embedding: Multilinguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv, 2025. doi:10.48550/arXiv.2402.03216. [30] Zeerak Waseem and Dirk Hovy. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the Student Research Workshop, SRW@HLT-NAACL 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 88–93. The Association for Computational Linguistics, 2016. doi:10.18653/V1/N16-2013. [31] Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). In Jonathan May, Ekaterina Shutova, Aurélie Herbelot, Xiaodan Zhu, Marianna Apidianaki, and Saif M. Mohammad, editors, Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2019, Minneapolis, MN, USA, June 6-7, 2019, pages 75–86. Association for Computational Linguistics, 2019. doi:10.18653/V1/S19-2010. [32] Ayme Arango Monnar, Jorge Perez Rojas, and Barbara Polete Labra. Cross-lingual hate speech detection using domain-specific word embeddings. PLOS ONE, 19(7):e0306521, July 2024. ISSN 1932-6203. doi:10.1371/journal.pone.0306521. [33] Michele Corazza, Stefano Menini, Elena Cabrio, Sara Tonelli, and Serena Villata. A multilingual evaluation for online hate speech detection. arXiv, 20(2), March 2020. ISSN 1533-5399. doi:10.1145/3377323. [34] Kriti Singhal and Jatin Bedi. Transformers at hsd-2lang 2024: Hate speech detection in arabic and turkish tweets using BERT based architectures. In Ali Hürriyetoglu, Hristo Tanev, Surendrabikram Thapa, and Gökçe Uludogan, editors, Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text, CASE 2024, St. Julians, Malta, March 22, 2024, pages 190–194. Association for Computational Linguistics, 2024. URL https://aclanthology.org/2024.case-1.26. [35] Akshay Singh and Rahul Thakur. Generalizable multilingual hate speech detection on low resource indian languages using fair selection in federated learning. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 7211–7221. Association for Computational Linguistics, 2024. doi:10.18653/V1/2024.NAACLLONG.400. [36] Sayan Ghosh and Suman Kumar Senapati. Hate speech detection in low-resourced indian languages: An analysis of transformer-based monolingual and multilingual models with cross-lingual experiments. Natural Language Engineering, 2024. doi:10.1017/S1351324924000281. 16
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
[37] Krishan Chavinda and Uthayasanker Thayasivam. A dual contrastive learning framework for enhanced hate speech detection in low-resource languages. In Kengatharaiyer Sarveswaran, Ashwini Vaidya, Bal Krishna Bal, Sana Shams, and Surendrabikram Thapa, editors, Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025), pages 115–123, Abu Dhabi, UAE, January 2025. International Committee on Computational Linguistics. URL https://aclanthology.org/2025.chipsal-1.11/. [38] Tomas Mikolov, I. Sutskever, Kai Chen, G. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. Neural Information Processing Systems, 2013. doi:10.48550/arXiv.1310.4546. [39] Sebastian Ruder, Ivan Vulic, and Anders Søgaard. A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 2017. doi:10.1613/jair.1.11640. [40] H. Kanayama, Trevor Cohn, Tengfei Ma, Steven Bird, and Long Duong. Multilingual training of crosslingual word embeddings. Conference of the European Chapter of the Association for Computational Linguistics, 2017. doi:10.18653/V1/E17-1084. [41] Xilun Chen and Claire Cardie. Unsupervised multilingual word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 261–270. Association for Computational Linguistics, 2018. doi:10.18653/v1/d18-1024. [42] Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891. Association for Computational Linguistics, 2022. doi:10.18653/v1/2022.acllong.62. [43] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916. Association for Computational Linguistics, 2024. doi:10.18653/v1/2024.acl-long.642. [44] Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025. doi:10.48550/arXiv.2506.05176. [45] Rita Butkienė, Dambrauskas Edgaras, Šukys Algirdas, and Žitkus Voldemaras. Lithuanian hate speech corpus v.1, 2025. URL http://hdl.handle.net/20.500.11821/69. CLARIN-LT digital library in the Republic of Lithuania. [46] Darius Amilevičius and Mažvydas Petkevičius. LITIS v.1, 2016. URL http://hdl.handle.net/20.500. 11821/11. CLARIN-LT digital library in the Republic of Lithuania. [47] Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Marco Stranisci. An Italian Twitter corpus of hate speech against immigrants. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga, editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https://aclanthology.org/L18-1443/. [48] Daryna Dementieva, Daniil Moskovskiy, Varvara Logacheva, David Dale, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. Methods for detoxification of texts for the russian language. Multimodal Technologies and Interaction, 5(9), 2021. ISSN 2414-4088. doi:10.3390/mti5090054. [49] Stephan Tulkens and Thomas van Dongen. Model2vec: Fast state-of-the-art static embeddings, 2024. URL https://github.com/MinishLab/model2vec. [50] minishlab/potion-multilingual-128M at Hugging Face, 2025. URL https://huggingface.co/minishlab/ potion-multilingual-128M. [51] BAAI/bge-m3 at Hugging Face, 2024. URL https://huggingface.co/BAAI/bge-m3. [52] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv, 2023. doi:10.48550/arXiv.1910.10683. [53] allenai/C4 datasets at Hugging Face, 2020. URL https://huggingface.co/datasets/allenai/c4. [54] Stephan Tulkens and Thomas van Dongen. POTION: bag of tricks leads to better models, 2024. URL https: //minishlab.github.io/tokenlearn_blogpost/. 17
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
[55] Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, Daniel Cer, Alice Lisak, Min Choi, Lucas Gonzalez, Omar Sanseviero, Glenn Cameron, Ian Ballantyne, Kat Black, Kaifeng Chen, Weiyi Wang, Zhe Li, Gus Martins, Jinhyuk Lee, Mark Sherwood, Juyeong Ji, Renjie Wu, Jingxiao Zheng, Jyotinder Singh, Abheesht Sharma, Divyashree Sreepathihalli, Aashi Jain, Adham Elarabawy, AJ Co, Andreas Doumanoglou, Babak Samari, Ben Hora, Brian Potetz, Dahun Kim, Enrique Alfonseca, Fedor Moiseev, Feng Han, Frank Palma Gomez, Gustavo Hernández Ábrego, Hesen Zhang, Hui Hui, Jay Han, Karan Gill, Ke Chen, Koert Chen, Madhuri Shanbhogue, Michael Boratko, Paul Suganthan, Sai Meher Karthik Duddu, Sandeep Mariserla, Setareh Ariafar, Shanfeng Zhang, Shijie Zhang, Simon Baumgartner, Sonam Goenka, Steve Qiu, Tanmaya Dabral, Trevor Walker, Vikram Rao, Waleed Khawaja, Wenlei Zhou, Xiaoqi Ren, Ye Xia, Yichang Chen, Yi-Ting Chen, Zhe Dong, Zhongli Ding, Francesco Visin, Gaël Liu, Jiageng Zhang, Kathleen Kenealy, Michelle Casbon, Ravin Kumar, Thomas Mesnard, Zach Gleicher, Cormac Brick, Olivier Lacombe, Adam Roberts, Qin Yin, Yunhsuan Sung, Raphael Hoffmann, Tris Warkentin, Armand Joulin, Tom Duerig, and Mojtaba Seyedhosseini. Embeddinggemma: Powerful and lightweight text representations. arXiv, 2025. doi:10.48550/arXiv.2509.20354. [56] google/embeddinggemma-300m at Hugging Face, 2025. embeddinggemma-300m.
URL https://huggingface.co/google/
[57] Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ömer Çağatan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafał Poświata, Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Björn Plüster, Jan Philipp Harries, Loïc Magne, Isabelle Mohr, Mariya Hendriksen, Dawei Zhu, Hippolyte Gisserot-Boukhlef, Tom Aarsen, Jan Kostkan, Konrad Wojtasik, Taemin Lee, Marek Šuppa, Crystina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang, Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan Dey, Dipam Vasani, Pranjal Chitale, Simone Tedeschi, Nguyen Tai, Artem Snegirev, Michael Günther, Mengzhou Xia, Weijia Shi, Xing Han Lù, Jordan Clive, Gayatri Krishnakumar, Anna Maksimova, Silvan Wehrli, Maria Tikhonova, Henil Panchal, Aleksandr Abramov, Malte Ostendorff, Zheng Liu, Simon Clematide, Lester James Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Alessia Borghini, Federico Cassano, Hongjin Su, Jimmy Lin, Howard Yen, Lasse Hansen, Sara Hooker, Chenghao Xiao, Vaibhav Adlakha, Orion Weller, Siva Reddy, and Niklas Muennighoff. MMTEB: Massive multilingual text embedding benchmark. arXiv, 2025. doi:10.48550/arXiv.2502.13595. [58] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi:10.1016/j.neucom.2023.127063. [59] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30233–30249. Curran Associates, Inc., 2022. [60] Shitao/MLDR datasets at Hugging Face, 2025. URL https://huggingface.co/datasets/Shitao/MLDR. [61] Shitao/bge-m3-data datasets at Hugging Face, 2024. URL https://huggingface.co/datasets/Shitao/ bge-m3-data. [62] Snowflake/snowflake-arctic-embed-l-v2.0 at Hugging Face, 2025. Snowflake/snowflake-arctic-embed-l-v2.0.
URL https://huggingface.co/
[63] Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11:1114–1131, 09 2023. ISSN 2307-387X. doi:10.1162/tacl_a_00595. [64] jinaai/jina-embeddings-v3 at Hugging Face, jina-embeddings-v3.
2024.
URL https://huggingface.co/jinaai/
[65] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv, 2021. doi:10.48550/arXiv.2106.09685. [66] intfloat/multilingual-e5-large-instruct at Hugging Face, 2023. URL https://huggingface.co/intfloat/ multilingual-e5-large-instruct. [67] FacebookAI/xlm-roberta-large at Hugging Face, 2019. xlm-roberta-large. 18
URL https://huggingface.co/FacebookAI/
Multilingual Text Embeddings for Hate Speech Detection
A P REPRINT
[68] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, 2024. [69] H. Hotelling. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6):417–441, September 1933. ISSN 0022-0663. doi:10.1037/h0071325. [70] Markus Goldstein and Andreas Dengel. Histogram-based outlier score (HBOS): A fast unsupervised anomaly detection algorithm. KI-2012: poster and demo track, 1:59–63, 2012. URL https://www.dfki.de/fileadmin/ user_upload/import/6431_HBOS-poster.pdf. [71] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: unbiased boosting with categorical features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://papers.nips.cc/paper/ 7898-catboost-unbiased-boosting-with-categorical-features. [72] Jin Huang and C.X. Ling. Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 17:299–310, March 2005. doi:10.1109/tkde.2005.50. [73] Andreas Beger. Precision-recall curves. SSRN Electronic Journal, 2016. doi:10.2139/ssrn.2765419. [74] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20: 37–46, 1960. doi:10.1177/001316446002000104. [75] Marry L. McHugh. Interrater reliability: the kappa statistic. Biochemia Medica, pages 276–282, 2012. doi:10.11613/bm.2012.031.
19