Evaluation of Adversarial Robustness in Arabic Language Models Anwar Alajmi1,2*, Ayed Salman1 and Imtiaz Ahmad1 1
2
Department of Computer Engineering, Kuwait University, Kuwait. Department of Computer Science, College of Business Studies, Public Authority of Applied Education and Training, Kuwait.
arXiv:2607.25814v1 [cs.CL] 28 Jul 2026
*Corresponding author(s). E-mail(s): [email protected]; Contributing authors: [email protected]; [email protected]; Abstract The emergence of the recent outstanding capabilities of Arabic Language Models has opened doors for exposing their vulnerabilities. One of the major security risks associated with such Natural Language Processing models is adversarial attacks. These attacks can deceive the model into the wrong prediction, raising critical model security and safety concerns. This study aims to assess the robustness of five state-of-the-art Arabic Language Models under a distinct set of Arabic adversarial attacks applied at various levels of granularity and using different example generation strategies. We also explore a defense technique based on adversarial training to enhance model robustness. The results show that insertion of diacritics can reduce the accuracy of some models by 92% while maintaining a low perturbation distance. For word-level attacks, manipulating Arabic conjunctions preserves high semantic similarity scores, low perturbation distance, and leads to an accuracy degradation of up to 58%. For sentence-level attacks, paraphrasing proves its effectiveness by an average reduction of 76% percent in the victim models’ performance. While adversarial training improves overall resilience, with MARBERT being the most robust and AraBERT showing the greatest relative gains, challenges persist, particularly against character-level noise. These findings highlight both the potential and limitations of current defense strategies in morphologically rich languages like Arabic. Keywords: Natural Language Processing, Adversarial Attacks, Robustness, Arabic Language Models
1
1 Introduction Natural Language Processing (NLP) models achieved remarkable success in a wide variety of applications that integrate with vital aspects of human life including education, medicine, and media. However, recent attacks on these models exposed serious safety concerns. Language Models (LMs) susceptibility to such attacks have critical implications that include the security of the model, the misuse of the compromised models to harm others, leakage of private or sensitive data, as well as economic and social effects [1]. Attacks that exploit the vulnerabilities of machine learning (ML) models are referred to as adversarial attacks. Adversaries craft malicious adversarial examples through input modifications to intentionally lead the model into a false prediction [2]. Textual adversarial attacks on NLP models were introduced through a Computer Vision (CV) attack called Fast Gradient Sign Method (FGSM) [3]. Due to feature space differences, CV adversarial attacks generate poor adversarial samples that don’t satisfy the attack constraints such as grammatical correctness and semantic similarity in NLP. Hence, NLP-specific attack strategies were introduced throughout the past years to further examine the weaknesses of LMs [4]. Textual adversarial attacks are commonly categorized based on the modified object [5]. Character-level attacks occur when a character is inserted, removed, or replaced. Word/token-level attacks happen when the same operations are performed on words/tokens, whereas sentence-level attacks are generated through paraphrasing, substituting, removing, or injecting sentences in the input sequence. Sometimes attacks composed of two or more of the previous granularity levels are referred to as multilevel attacks [4]. Further categorization is based on the adversarial example-generation strategies, which include edit-based, and importance-based methods [6]. In the editbased techniques, no features or attributes of the modified objects are considered. Meanwhile, importance-based methods usually focus on perturbing object with the most importance to the prediction of the victim model. Figure 1 provides a taxonomy of textual adversarial attacks. Adversarial attacks raise critical ethical concerns, particularly when LMs are deployed in critical applications such as healthcare and legal systems. The exploitation of these models vulnerabilities can lead to data theft, business disruption, misinformation, or harm. The attacks can be designed based on the adversary’s full access to the victim model’s parameters (white-box) or without any model-specific knowledge (black-box) [7]. Usually such attacks occur in the real-world under black-box theme [4, 5]. Moreover, targeted attacks occur when the adversary expects a specific misprediction, whereas non-targeted attacks happen when the adversary aims for a general incorrect output [7–9]. Adversarial robustness evaluation must be a crucial measure to consider before the release of any NLP model to mitigate these issues. While this field gained significant attention among researches worldwide, it is still very understudied when it comes to Arabic models. The Arabic language is the 5th most spoken language globally, with over 334 million speakers, making it one of the most popular Semitic languages [10]. Only few limited studies discuss this area due to Arabic language-specific reasons such as low resources, the complexity of the language, and its various dialects [11]. Arabic words are complex 2
Fig. 1: NLP Attacks Taxonomy.
in structure where a single word can convey the meaning, tense, number, and gender. Furthermore, different vowels (diacritics) on an Arabic word could completely change its meaning [12, 13]. Hence, from a computational linguistics and NLP perspective, Arabic presents unique challenges compared to other languages, particularly in areas like adversarial robustness, which require comprehensive evaluations across diverse linguistic nuances. Despite significant advancements in NLP, Arabic remains underrepresented when it comes to adversarial robustness research due to its linguistic complexity, dialectical diversity, and resource limitations. These challenges have left a critical gap in evaluating how state-of-the-art Arabic and multilingual NLP models perform under adversarial conditions. Therefore, this study addresses this gap by exploring the effects of non-targeted, black-box adversarial attacks on leading models such as AraBERT [14], MARBERT [15], CaMeLBERT [16], mBERT [17], and XLM-T [18]. The key contributions of this paper are listed as follows:
• This work is the first of its kind to evaluate the adversarial robustness of Arabic NLP models, providing novel insights into their performance under adversarial conditions. • The application of a diverse set of Arabic adversarial attacks on all levels of granularity (character-level, word-level, and sentence-level) using various example-generation strategies (edit-based, importance-based, and paraphrase-based). • A thorough evaluation process that includes measures of accuracy, attack success rate, perturbation rate, and semantic similarity. The paper is organized as follows: Section 2 reviews related studies in the field of LMs adversarial robustness. Sections 3 explores the methodology, while Section 4 3
demonstrates and discusses the experimental results. Finally, the conclusion and future work are presented in Section 5.
2 Literature Review Adversarial attacks are carefully crafted input examples designed to intentionally manipulate NLP models into producing incorrect outputs. Some of these perturbed examples may not be easily detectable by humans but have the potential to deceive LMs. Textual attacks are usually categorized based on the semantic granularity of the perturbed object (character, word, token, sentence) [19]. The study by [20] further classifies the attacks based on the strategy employed for generating examples. Semantic granularity categories have example generation strategies as sub-categories: gradient-based, optimization-based, importance-based, paraphrase-based, and generative model-based (GAN). Certain criteria must be met when generating adversarial examples that includes grammatical errors that are equal to or less than the original example as well as semantic similarity [21]. Numerous methods exist to check the previous requirements such as grammar checking tools, and semantic similarity measures (cosine similarity, universal sentence encoders (USE) [22], and Sentence-BERT [23]). Attacks on the character level involve perturbation of characters that includes adding, removing, swapping, or substituting characters within a given sequence. While these strategies are notably efficient and hard to observe, their detection can be straightforward using spell-checking tool [24]. Character-level attacks under blackbox theme focus on adding natural or synthetic noise such as punctuation, diacritics, special characters, and numbers. While attack methods proposed in [25, 26] are edit-based, the works by [27, 28] depend on importance-based character transformations. The authors in [27] leverage local post-hoc explanation, SHapley Additive exPlanations (SHAP) [29] for importance-based adversarial example generation. On the other hand, [28] uses a Word Importance Rank (WIR) to find the most important words for character manipulation. Moreover, DeepWordBug and TextBugger [30, 31] obtain the importance of the words within a sequence through greedy WIR algorithm, then apply multiple character-level perturbations to the most important word. On the other hand, TextAttack [32] offers a broader framework where character-level manipulations are a part of its toolkit. It also proposes word-level and semantic-level transformations, including paraphrasing and other higher-level alterations. For word-level attacks, [33–35] employ WIR to craft black-box word replacements. While [33] replaces the word with the highest importance by its synonym, [34] uses BERT Masked Language Model (MLM) to predict the synonym. On the other hand, [35] perturbs words to violate Arabic grammatical rules in order to deceive victim models including BERT, CNN, and LSTM. In a similar approach to [34], [36] uses BERT MLM to replace words with the highest scores. However, instead of conventional scoring functions, SHAP is used to find the sensitivity of the model’s prediction to each word within the original sequence, which is similar to the word-attacks in [37]. The work in [38] evaluates the robustness of AraBERT and CaMeLBERT under editbased and gradient-based word substitutions. AraBERT’s accuracy decreased by 7%
4
(edit-based) and 37% (gradient-based), while CaMeLBERT’s accuracy was reduced by 10% and 36% after edit-based and gradient-based attacks, respectively. The previously reviewed attack studies are summarized in Table 1. LMs adversarial robustness is typically assessed by evaluating their performance in the face of carefully crafted attacks. It is a measure of how well a model performs under adversarial attacks [4, 5]. To improve the ability of NLP models to resist adversarial attacks, researchers have proposed data-driven approaches that rely on data augmentation. This means training the models on a mixture of real data and synthetic data that has been modified in a way to create adversarial examples. This helps the models to learn to be less sensitive to small changes in the input data, which can make them more robust to adversarial attacks. The work in [39] utilized data augmentation with less cost. The study presents a simple vanilla adversarial training process (A2T) on BERT and RoBERTa to reduce the computational time and complexity of generating attacks and therefore reduce overall adversarial training time. In general, the approach consists of generating white-box gradient-based attacks under DistilBERT [40] similarity constraint. Another active defense mechanism is to use a ML attack detector to identify and remove adversarial examples from the training data as proposed by [41, 42]. In [41] a defense method against importance-based attacks based on deletion-based scoring is proposed. The method calculates the words importance for the original and adversarial sequences. Then, entropy is calculated for both to identify an adversarial example. However, [42] uses SHAP for word importance calculations, and then TextAttack to craft word-level adversarial attacks. After that, a detection model is trained on the generated examples. Similarly, [43] calculates the word frequency in the adversarial example and original example to detect word-level perturbations in text classification models. On the other hand, [44] proposes a method to discriminate and restore perturbed tokens semantics through searching the corpus token embedding space using k-nearest neighbors (kNN). In addition to adversarial training, representation learning is also considered to be an effective and active defense mechanism. This involves using techniques such as randomizing inputs, and unifying input representation to create more robust representations of the input data. An example of this is Dirichlet Neighborhood Ensemble approach by [45]. The authors focus on improving adversarial robustness through exposing the target model to broader forms of the input samples including possible synonyms. This is done through forming virtual sentences by randomized sampling within the convex hull that is spanned by the sequence word embeddings and its synonyms. The process is done for every word within the original input sequence. Then, the model is trained on the augmented data with preserved semantics. Finally, there are a number of passive defense tactics that can be used to protect NLP models from adversarial attacks, such as misspelling check tools used in [30, 31] to defend against character-level attacks [20]. A list of the defense methods reviewed is shown in Table 2. Authors in [46] apply word-level synonym replacement on WordCNN [47], and WordLSTM [48], and AraBERTBASE. Most important words are identified using a ranking greedy algorithm and then top k replacement words are generated from
5
AraBERTBASE v024 as MLM model. WordCNN and WordLSTM accuracy decreased by almost 5% on MSDA dataset, whereas AraBERT’s accuracy decreased by 26.93%. The introduction of transformers [49], have revolutionized the NLP field through the attention mechanism, which allowed the models to enhance the text context understanding by processing different parts of the input simultaneously. One of the most successful transformer-based models is BERT or Bidirectional Encoder Representations from Transformers which leverages contextualization by its bidirectional input processing approach [50]. Hence, this study only focuses evaluating BERT-based models to examine the capabilities of compromising such powerful models. Since NLP adversarial robustness is limited in the Arabic literature, this study is the first to evaluate a variety of customized Arabic attacks on all levels of granularity on Arabic and multilingual models. Furthermore, it provides insights on the success of the attacks by incorporating a variety of evaluation metrics for measuring the perturbation amount and the semantic similarity of the generated adversarial examples.
3 Methodology This section discusses the victim models, the task, and the dataset for fine-tuning the models. In addition, the adversarial setting as well as the character-level, word-level, and sentence-level attacks will be explored.
3.1 Victim Models Due to the state-of-the-art performance of BERT on NLP tasks, specifically for text classification, the adversarial robustness of the most successful Arabic and multilingual BERT-based models will be examined. Hence, the victim models are as follows:
• AraBERT: Pre-trained on a large Modern Standard Arabic (MSA) corpus, AraBERT is one of the first BERT models dedicated for Arabic NLP tasks. Due to the complexities within the Arabic language as well as its variant dialects, AraBERT incorporates customized pre-processing and tokenization techniques to comprehend Arabic text [14]. • MARBERT: While AraBERT’s pre-training data mainly consist of MSA, MARBERT [15] extends AraBERT’s approach through including dialectal Arabic. • CaMeLBERT: Based on BERT architecture, CaMeLBERT (mix) [16] extends AraBERT and MARBERT by pre-training on large datasets that consist of three types of Arabic text: classical Arabic, MSA, and dialectal Arabic. • mBERT: Extended from BERT, mBERT or Multilingual BERT is pre-trained on a large multilingual corpus of 104 languages including the Arabic language. • XLM-T: XLMs or Cross-lingual Language Models [54] differ from BERT in their enhanced capabilities by transferring learning between languages. Hence, these type of models work very well on multiple languages, including low resource languages like Arabic. XLM-T [18], which is a variant that is trained on Twitter data was selected due to the nature of the used dataset samples.
6
7
Year 2018 2018 2019 2020 2020 2020 2021 2022 2023 2023 2024
Reference
[30]
[31] [35]
[33] [34] [25]
[28]
[36]
[26]
[27]
[46]
Importance-based
Importance-based
Edit-based
Importance-based
Importance-based
Importance-based Importance-based Edit-based
Importance-based Importance-based
Importance-based
Strategy
BERT, RoBERTa [51], and DistilBERT [52] GRU, LSTM, CNN, and BERT LSTM, CNN, and AraBERT
BERT, Bi-LSTM, XLNet, and CNN BERT
CNN and LSTM Bi-LSTM, CNN, XLNet, and BERT BERT BERT RoBERTa
CNN
Victim Model
Character replacement through LIME [53] and SHAP [29] Synonym replacement through AraBERT MLM
Character insertion, deletion, swapping, and replacement Insertion, deletion, swapping, and replacement Word replacement with Arabic adjectives perturbations BERT MLM word substitution BERT MLM word insertion and substiution 10 attacks based on character deletion, swapping, and substitution Substituting Arabic characters with visually similar ones BERT MLM word substitution based on SHAP score Punctuation insertion
Method
Table 1: Literature review summary of black-box adversarial attacks on NLP models.
Word
Character
Character
Word
Character
Word Word Character
Character Word
Character
Attack level
8 2020
[45]
[43]
2022
2020
[44]
[42]
2019
[31]
2021 2023
2018
[30]
[39] [41]
Year 2018
Reference
Attack level
Victim Model
CNN, LSTM, and bag-of-words
Bi-LSTM
BERT and RoBERTa BERT and LSTM
CNN, LSTM, and RoBERTa
BERT
CNN and LSTM
CNN
Check misspelling to avoid character-level perturbations through Python autocorrect package Misspelling checking through a context-aware service Proposes a KNN based embedding estimator to recover the semantics of perturbed words Using word frequency to detect adversarial sequences examples Enhanced adversarial training technique Detection of adversarial attacks through calculating the entropy between the original sample and the adversarial sample Train Bi-LSTM in order to detect word-level attacks Representation learning through randomized word synonyms substitutions
Method
Table 2: Reviewed works on adversarial robustness under black-box attacks.
Word-level
Word-level from [32]
Word-level from [19] Word-level
Word-level
Word-level
Character-level
Character-level
3.2 Task and Dataset The selected task for generating and evaluating the adversarial attacks is binary textclassification. In specific, classical sentiment analysis is selected due to its broad and common uses, without delving into more specific tasks that can cause the victim models to differ in their performances. The dataset used [55] is diverse as it is constructed from the Arabic sentiment tweets datasets in [56, 57], which combine tweets in MSA and dialectal Arabic. In the pre-processing phase, unnecessary special characters, including numbers, non-Arabic characters, punctuation, and emojis, are neglected. Note that Arabic diacritics (tashkeel) were not present in the raw dataset and are not affected by this step.”. In addition, Farasa segmentation [58] is applied to break down Arabic words into stems, prefixes, and suffixes, which will improve performance by reducing redundancy. Finally, the processed text is tokenized using SentencePiece [59]. The dataset includes roughly 16K samples in total are labeled as negative or positive due to the nature of the task (binary). The training set includes 14.7K samples, while the testing set has 1.08K samples. Due to performance issues associated with the imbalanced number of classes, sampling techniques such as down-sampling and up-sampling were carried out to ensure the accuracy and objectivity of the models’ prediction. The distribution and length of the sample classes, as well as the training and test sets, are shown in Figures 2a and 2b.
(a) Classes statistics within the dataset.
(b) Training and test subsets statistics.
Fig. 2: The Used Sentiment Analysis Dataset Information.
3.3 Adversarial Setting and Objective Our approach evaluates the adversarial robustness of victim models against nontargeted black-box attacks. Adversarial robustness refers to a model’s ability to maintain performance when exposed to perturbed (adversarial) input samples. In the black-box setting, the attacker has no access to the internal architecture or parameters of the victim model and can only observe its outputs. The goal behind the attacks is to generate adversarial examples from the constructed dataset in a way that will lead to an incorrect output by the unfortified victim models. Having an original example
9
x, where the model correctly classifies the input f (x) = y , each attack’s algorithm generates a perturbed example x′ aimed to lead the victim models to missclassification such that f (x′ ) ̸= y . The framework of the attacks is demonstrated in Figure 3. The models are trained on the pre-processed dataset to perform binary classification. Then, a subset of the dataset is manipulated through six attacks strategies to generate adversarial samples. The perturbed samples are tested on each model to evaluate the adversarial robustness.
Fig. 3: Attacks Framework.
3.4 Attacks Six types of customized black-box and non-targeted Arabic attacks from [60] will be tested on each of the victim models. Three of the attacks are on a character-level (edit-based), two of them are on a word-level (importance-based), and one is on a sentence-level (paraphrase-based). The edit-based attacks do not take regard any character importance information in the perturbation process, whereas importance-based calculates the importance of every word to the prediction of the model and perturbs in a descending order of importance score. This is done through the use of SHapley Additive exPlanations (SHAP) that is based on game theory to interpret and score the words within the input sequence that lead to a model’s decision [29]. To generate word-level adversarial attacks, SHAP is used to compute the importance scores of individual words (see examples in Appendix A). Specifically, the shap.Explainer
10
class was used with a token masker and fixed context=1, which masks one token at a time while preserving the rest of the sentence. This implicitly defines the baseline input as the unmasked context, eliminating the need for an explicit reference input. Once SHAP values are obtained for each token, they are sorted by magnitude and the most influential token is selected as the perturbation target, following the methodology of [37]. This procedure yields targeted, minimally invasive adversarial examples. Figure 4 demonstrates the attacks techniques using an illustrative example, while Table 3 provides an example of each attack method. The details of constructing the adversarial samples are as follows:
• Diacritics: Arabic diacritics are vowels that can shift the meaning of words, where the insertion of different diacritics can cause one word to multiple meanings. For instance, are words with the same letters but denote three entirely different meanings in Arabic, highlighting the complexity of the language. This attack incorporates the insertion of a variety of Arabic diacritics to deceive the victim models. • Visual: The Arabic language consists of distinct characters that are visually similar such as which are only differentiated by one dot. Hence, the attacks substitutes the input characters with visually similar ones. • Phonetic: This attack targets the dialectal differences across Arabic-speaking regions. With roughly 25 dialects in the Arab world, a word with the same semantic meaning can be written differently in various countries, particularly when it comes to male and female pronouns. The strategy involves performing character replacements to change the dialect of the original sequence. • AraBERT MLM: Performs word substitutions with synonyms generated by AraBERT Masked Language Model (MLM), where the substitution begins with the word that has the highest SHAP score. • Conjunctions: Replacement of Arabic conjunctions with semantically similar ones starting with the ones that have higher SHAP scores. • Paraphrase: Rephrasing Arabic sentences similar to [37], while preserving the semantics of the original sequence using the abstractive summarization AraT5 model in [61].
4 Experimental Results and Analysis After data pre-processing and class balancing techniques, the victim models were trained on the constructed dataset for binary sentiment analysis through a unified training approach and hyperparameters selection for objective evaluation. The pretrained models are loaded from huggingface and finetuned with early stopping to prevent them from overfitting. In order to properly evaluate the adversarial robustness of the unfortified models the following metrics will be incorporated:
• Accuracy : The number of correct predictions over the model’s total predictions as shown in Equation 1, where TP denotes the number of true positives, and TN denotes the number of true negatives, and FP represent the number of false positives,
11
12
Attack Level Character-level Character-level Character-level Word-level Word-level Sentence-level
Attack
No Attack Diacritic Visual Phonetic AraBERT MLM Conjunctions Paraphrase
Table 3: Examples of the attacks.
Edit-based Edit-based Edit-based Importance-based Importance-based Edit-based
Perturbation Strategy
Example
Fig. 4: Flowchart Demonstrating The Character-level and Word-level Attacks Strategies.
whereas FN denotes the number of false negatives.
Accuracy =
TP + TN TP + TN + FP + FN
(1)
• Attack Success Rate (ASR) : A metric that is commonly used to evaluate the effectiveness of adversarial attacks. As represented in Equation 2, ASR measures the percentage of successful adversarial samples (Nsuccessful ) over the total number of adversarial samples (Ntotal ). In our experiments, (Nsuccessful ) denotes the number of adversarial examples for which the model’s prediction on the perturbed input differs from its original correct prediction on the clean input. Samples where the clean-input prediction was already incorrect are excluded from ASR computation. ASR =
Nsuccessful Ntotal
13
× 100%
(2)
• Perturbation Rate : To measure the perturbation distance between the original and perturbed examples, Levenshtein distance is selected due to its efficiency in including all perturbation operations on sequences of different sizes. It is shown in Equation 3 where the Levenshtein distance is di,j between the original example A with the first character xi and the perturbed example B with the first character yj . The function I returns 1 if xi ̸= yj [62]. i j
if j = 0 if i = 0
(3) di−1,j + 1, min di,j−1 + 1, di−1,j−1 + I (xi ̸= yj ) • Cosine Similarity : The cosine angle between the original sequence vector A and perturbed sequence vector B (Equation 4). The cosine value increases with higher similarities between the original and adversarial examples. dx,y (i, j ) =
cos(θ) =
A·B ||A||||B||
(4)
• Universal Sentence Encoder (USE) : A more recent approach that extends Cosine Similarity by using arccos to find the angular distance between the original sequence vector A and perturbed sequence vector B [22] as shown in Equation 5. High USE score indicates similar original and adversarial samples, which means that the semantic similarity is preserved. sim(A, B) = 1 −
arccos (cos(θ)) π
(5)
4.1 Adversarial Attacks Results The accuracies of the models on the test set are: 99% for MARBERT, 95% for CaMeLBERT, 94% for AraBERT and XLM-T, and 87% for mBERT. Table 4 demonstrates the effects of diacritics insertions on the performance of the victim models. It is shown that MARBERT is the most robust against this type of attacks with 94% post-attack accuracy. On the other hand, AraBERT is the most vulnerable to diacritics manipulation with an accuracy decrease of 92% followed by CaMeLBERT (80% decrease). When it comes to perturbation rate, mBERT model adversarial samples modify an average of only 10% of the original samples, while retaining the highest cosine and USE similarity scores of 90% and 68%, respectively. The attack success rates plotted against the perturbation rates of the victim models are shown in Figure 5 as well as the pre-attack and post-attack accuracies. The experimental results of diacritics manipulation proves that character-level insertions attacks are feasible. This kind of attacks is not affected by the limitations of character or word perturbations, especially that it is hard to be detected by passive defense approaches like grammar checkers, while preserving the semantics on the input [26].
14
15
Pre-attack Accuracy 94% 99% 95% 87 % 94%
Model
AraBERT MARBERT CaMeLBERT mBERT XLM-T
92% 5% 80% 42% 29%
Decrease 15% 28.5% 15.5 % 10% 33%
Perturbation Rate
Table 4: Diacritics Attacks Evaluation Results
2% 94% 15% 45% 65%
Post-attack Accuracy
84% 57% 79% 90% 53%
Cosine Similarity
56% 55% 55% 68% 52%
USE Similarity
16
Pre-attack Accuracy 94% 99% 95% 87 % 94%
Model
AraBERT MARBERT CaMeLBERT mBERT XLM-T
94% 13% 78% 39% 21%
Decrease 31% 45% 38% 18% 43%
Perturbation Rate
Table 5: Visual Attacks Evaluation Results
0% 86% 17% 48% 73%
Post-attack Accuracy
26% 12% 31% 53% 27%
Cosine Similarity
56% 50% 63% 72% 58%
USE Similarity
Fig. 5: The results of diacritics manipulation on the models.
Fig. 6: The victim models’ performance evaluation after the visual attack.
The results of applying visual attack on the victim models are indicated in Table 5. Again, AraBERT is the weakest to visual replacement with 0% post-attack accuracy, while MARBERT is the most robust with 86% post-attack accuracy as shown in Figure 6. MARBERT may be less susceptible to visual replacement attacks due to its diverse and extensive pre-training data, which includes a wide range of Arabic dialects. The pre-training corpus also included tweets that contained at least three Arabic words but did not exclude non-Arabic content. As a result, the model was exposed to a variety of mixed-script text. This exposure to noisy and heterogeneous data during pre-training likely enhanced MARBERT’s ability to handle such perturbations, contributing to its robustness. Moreover, this kind of attack results in high perturbation rate for all the models due to the character swapping operation which is more expensive in terms of the adversary budget (character deletion and insertion) than the diacritics insertion attack. However, it only takes a perturbation rate of 18% for mBERT to drop its accuracy by 39%, while maintaining the highest cosine and USE similarity scores. As for the last character-level attack, Table 6 and Figure 7 surprisingly show that CaMeLBERT’s performance proves to be the lowest among the victim models after performing phonetic character substitutions (31% drop in accuracy). Although this model is pre-trained on a datasets that include a variety of dialects, this finding reveals a generalization problem when it comes to unseen Arabic data. On the other 17
18
Pre-attack Accuracy 94% 99% 95% 87 % 94%
Model
AraBERT MARBERT CaMeLBERT mBERT XLM-T
3% 2% 31% 22% 14%
Decrease 28% 37% 33% 17% 36%
Perturbation Rate
Table 6: Phonetic Attacks Evaluation Results
91% 97% 64% 65% 80%
Post-attack Accuracy
33% 33% 30% 50% 34%
Cosine Similarity
62% 63% 61% 70% 64%
USE Similarity
Fig. 7: The effects of phonetic character substitution on the victim models.
hand, MARBERT, shows good generalization capability as it is barely affected with the phonetic attack with only 2% decrease in its pre-attack accuracy. In addition, the second most vulnerable model to dialect manipulations is mBERT, in which the examples sustain the highest cosine and USE similarity results, while having the lowest perturbation rate of 17%. This is relevant to mBERT’s pre-training data which consists only of MSA with no dialectal Arabic inclusion. With the use of SHAP and AraBERT MLM to generate replacements for the most significant words within the original sample, the impact on the target models can be seen in Table 7. The attack was successful the most on XLM-T, which exhibited an overall accuracy decrease of 44% with the highest perturbation rate, followed by mBERT (highest Cosine Similarity) and MARBERT, which both approximately have the same amount of drop in accuracy by 39%. Meanwhile, AraBERT had the highest post-attack accuracy of 81% revealing that it is less vulnerable to similar word substitution than the other tested models. Applying conjunctions manipulation on the victim models can notably impact their performance as seen in Figure 8. It is observed in Table 8 that AraBERT and XLM-T have endured the heaviest accuracy reduction by 58% and 39%, respectively. This can be related to the models reliance on understanding the syntactic structure and relationships conveyed by these conjunctions, especially that this attack technique involves altering conjunctions within the original example leading to confuse the models’ understanding of different parts of the sentence. However, this time CaMeLBERT has the best scores when it comes to perturbation rate and similarity scores, showing that it is capable of understanding the text’s overall meaning, even when the Arabic conjunctions are manipulated. Unlike the previously discussed attacks, paraphrasing the sentences involves rephrasing the entire sequence while maintaining the original meaning. This method significantly alters the text’s structure and wording but preserves its semantic content, making it a powerful technique for evaluating model robustness. It is worth noting that, unlike character-level and word-level attacks which generate model-specific adversarial examples tailored to each victim model, the paraphrase attack produces a single shared set of adversarial examples applied uniformly to all models. As a result, the perturbation and similarity metrics are identical across all victim models. The
19
20
Pre-attack Accuracy 94% 99% 95% 87 % 94%
Model
AraBERT MARBERT CaMeLBERT mBERT XLM-T
13% 39% 24.3% 39.2% 44%
Decrease 70% 81% 64% 69% 82%
Perturbation Rate
Table 7: MLM Attacks Evaluation Results
81% 60% 70.7% 47.8% 50%
Post-attack Accuracy
38% 27% 29% 36% 25%
Cosine Similarity
61% 59% 62% 61% 58%
USE Similarity
21
94% 99% 95% 87 % 94%
AraBERT MARBERT CaMeLBERT mBERT XLM-T
36% 97.5% 73.5% 83% 55%
Post-attack Accuracy 58% 1.5% 21.5% 4% 39%
Decrease 29% 25% 24% 26% 53%
Perturbation Rate
Table 8: Conjunctions Attacks Evaluation Results
Pre-attack Accuracy
Model
96% 93% 96% 94% 94%
Cosine Similarity
51% 51% 52% 50% 50%
USE Similarity
Fig. 8: The results obtained from conjunctions manipulation on the victim models.
Model AraBERT MARBERT CaMeLBERT mBERT XLM-T
Pre-attack Accuracy
Post-attack Accuracy
Decrease
94% 99% 95% 87 % 94%
14% 14% 22% 28% 11%
80% 85% 73% 59% 83%
Table 9: Paraphrase Attacks Evaluation Results
perturbation rate for paraphrasing attacks is notably high, reaching 83%, indicating that a large portion of the text is changed during paraphrasing as shown in Figure 9. Despite this high perturbation rate, the paraphrased texts preserve semantic similarity, achieving an average Cosine Similarity of 39.43% and a USE Similarity of 61.48% across the test set, as illustrated in Figure 10. MARBERT is the most affected by an 85% accuracy decrease, followed by XLM-T as shown in Table 9.
Fig. 9: The results of the paraphrase-based attack on the victim models.
In general, the five victim models were successfully attacked by our generated textual adversarial samples. All the models endured notable performance degradation, 22
Fig. 10: Cosine and USE Similarity scores between original samples and their corresponding adversarial examples.
which shows the effectiveness of the attacks techniques. Moreover, the proposed methods have met the constraints of providing grammatically correct and semantically similar perturbed samples. Our experimental findings can be highlighted as follows:
• Among the three character-level attacks, the diacritics attack had the heaviest impact on the models’ performance as well as the lowest perturbation rates and highest similarity scores. • As for word-level attacks, manipulation of Arabic conjunctions maintained a low perturbation distance, more than 90% Cosine Similarity, and a considerable accuracy reduction that goes up to 58%. • The paraphrase-based attack hindered the models’ capability to correctly classify adversarial samples by an average of 76% reduction in accuracy. The above experimental results reveal critical weaknesses in the leading Arabic LMs’ ability to handle punctuation insertions, visual and phonetic substitutions as well as syntactic variability. This underscores the need for developing more robust models that can accurately interpret and process diverse inputs without significant performance degradation.
4.2 Adversarial Training As a defense mechanism, adversarial training [3, 63] was conducted on the victim models. Although adversarial training induces a modest decline (roughly 1-2%) in the clean accuracy of the fortified models, it substantially increased resilience. Another trade-off is the additional computational overhead, which increased the training time by approximately 25–40%, which varies based on model size and attack complexity. Table 10 demonstrates the performance of five Arabic language models—AraBERT, MARBERT, CaMeLBERT, mBERT, and XLM-T on the distinct adversarial attacks after adversarial training. For each attack, the table shows the clean accuracy (pre-attack), the post-attack accuracy after adversarial training, the accuracy decrease, and the robustness improvement.
23
24
99%
95%
87%
94%
Diacritic Phonetic Visual MLM Conjunction Paraphrase Diacritic Phonetic Visual MLM Conjunction Paraphrase Diacritic Phonetic Visual MLM Conjunction Paraphrase Diacritic Phonetic Visual MLM Conjunction Paraphrase
AraBERT
MARBERT
CaMeLBERT
mBERT
XLM-T
73.04% 88.35% 79.13% 78.33% 76.42% 48.40%
55.65% 80.43% 57.39% 71.12% 85.78% 60.60%
30.43% 78.13% 33.91% 86.48% 81.33% 67.30%
95.65% 98.77% 89.57% 83.85% 98.89% 62.40%
22.61% 92.89% 20.87% 87.16% 49.33% 52.30%
Post-attack Accuracy
Table 10: Model performance after adversarial training.
94%
Diacritic Phonetic Visual MLM Conjunction Paraphrase
Pre-attack Accuracy
Attack
Model
20.96% 5.65% 14.87% 15.67% 17.58% 45.60%
31.35% 6.57% 29.61% 15.88% 1.22% 26.40%
64.57% 16.87% 61.09% 8.52% 13.67% 27.70%
3.35% 0.23% 9.43% 15.15% 0.11% 36.60%
71.39% 1.11% 73.13% 6.84% 44.67% 41.70%
Decrease
+8.04% +8.35% +6.13% +28.33% +21.42% +37.40%
+10.65% +15.43% +9.39% +23.32% +2.78% +32.60%
+15.43% +14.13% +16.91% +15.78% +7.83% +45.30%
+1.65% +0.77% +3.57% +23.85% +1.39% +48.40%
+20.61% +1.89% +20.87% +6.16% +13.33% +38.30%
Improvement
Across all models, adversarial training led to consistent improvements in robustness. The notable findings of the adversarial training experiments are summarized below:
• Despite adversarial fine-tuning, Diacritic and Visual attacks remained highly effective in degrading performance. AraBERT dropped to 22.61% (Diacritic) and 20.87% (Visual), despite improvements over its pre-trained state. These results indicate that Diacritic and Visual attacks remain a challenge in most victim models robustness although adversarial training was conducted. This emphasizes the need for more targeted defense strategies. • The highest robustness gains were observed for word-level attacks, particularly MLM and the sentence-level attack Paraphrase. MARBERT and CaMeLBERT achieved improvements of +48.40% and +45.30% on Paraphrase attacks, respectively, which may indicate that their contextual embeddings are particularly effective at capturing underlying semantic meaning, enabling them to generalize better to such paraphrased inputs during adversarial training. • Model-Specific Behavior: – AraBERT exhibited substantial gains from adversarial training, especially on Diacritic, Visual, and Paraphrase attacks (over +20% each). – MARBERT remained the most consistently robust model overall, with minimal decreases across all attack types. – CaMeLBERT demonstrated strong improvements in general, suggesting effective generalization to adversarial data. – mBERT improved moderately but was less robust than Arabic-specific models. – XLM-T performed well against word-level attacks, particularly with +37.40% improvement on Paraphrase, highlighting the benefits of multilingual pre-training. From the findings above, adversarial training enhanced robustness, but challenges persist with character-level noise in morphologically rich languages like Arabic. MARBERT proved most resilient overall, while AraBERT showed the greatest relative gains, emphasizing both the potential and limits of the conventional adversarial training approaches.
5 Conclusion and Future Work Adversarial attacks on Arabic language models (LMs) have ethical implications, particularly in critical applications such as healthcare, finance, and governance, where incorrect predictions can cause harm, bias, or misinformation. To address these concerns, researchers should explore and report model vulnerabilities through a proper adversarial robustness evaluation to achieve reliability. Hence, the objective of this paper was to study and evaluate the adversarial robustness of the most successful Arabic and multilingual NLP models against customized Arabic textual black-box attacks. Victim models were evaluated on a diverse set of adversarial examples at all levels of granularity, including three character-level attacks (diacritics, visual, and phonetic), two word-level attacks (MLM and conjunctions), and one sentence-level attack (paraphrase).
25
The results reveal that AraBERT model is the weakest when it comes to diacrtics, visual, and conjunction attacks. It is followed by XLM-T, which endured the heaviest performance degradation after MLM attacks. In addition, CaMeLBERT showed the poorest post-attack performance when it was evaluated after dialectal manipulation and MARBERT had the most accuracy decrease when it was evaluated on paraphrased examples. Among the attacks, paraphrasing is the most effective in reducing model accuracy, leading to the highest average accuracy decrease. Diacritic manipulations prove most impactful for perturbation distance, while conjunction substitutions cause the most significant drop in similarity scores. Nevertheless, the use of adversarial training improved model robustness, particularly against higher-level word-based perturbations. However, resilience to character-level noise remains an open challenge. MARBERT stands out as the most robust overall, while AraBERT exhibits the largest relative improvements. These findings highlight both the effectiveness of adversarial attacks and the limitations of adversarial training, underscoring the need for further research to enhance models’ ability to capture subtle character-level distortions. This work can be extended through incorporating importance-based character and sentence perturbations to minimize the perturbation rate and increase the attack success and similarity scores. Attacks can also be used to evaluate the recent and popular GPT models similar to [64, 65] approaches. Furthermore, the models’ robustness can be evaluated on other NLP tasks such as dialect identification, named-entity recognition, or question answering. Such cross-task assessments would help identify any task-specific vulnerabilities. To enhance the robustness of Arabic language models (LMs), researchers and developers should strengthen model defenses through integrating customized defense techniques [66], the information bottleneck method [67], and advanced frameworks like Interval Bound Propagation (IBP) [68, 69]. Overall, this study presents the first comprehensive adversarial robustness evaluation targeting Arabic NLP models across multiple perturbation levels. Our findings contribute valuable insights into the practical vulnerabilities and defense limitations of widely used Arabic language models, setting a foundation for future work in secure and trustworthy Arabic NLP.
Funding The authors received no specific funding for this study.
Data availability The dataset [55] that was used to train the models is available for public use on huggingface.
Declarations Conflict of Interest The authors declare that they have no conflict of interest.
26
Appendix A
SHAP Force Plots
Figures A1 and A2 in this appendix show SHAP force plots for four Arabic test sentences, generated using the AraBERT model fine-tuned for sentiment analysis. For each sentence, two subplots are presented: the first highlights token contributions driving the prediction toward the negative class, and the second highlights contributions toward the positive class. Horizontal bars indicate how each token shifts the model’s output from its expected value (base value) toward the selected class. Blue bars represent features that push the base value lower, whereas red bars represent features that push the prediction higher.
Fig. A1: SHAP force plots for the first pair of examples.
27
Fig. A2: SHAP force plots for the second pair of examples.
References [1] Amodei, D., Olah, C., Steinhardt, J., Christiano, P.F., Schulman, J., Mané, D.: Concrete problems in AI safety. CoRR abs/1606.06565 (2016) 1606.06565 [2] Cubuk, E.D., Zoph, B., Schoenholz, S.S., Le, Q.V.: Intriguing properties of adversarial examples. arXiv preprint arXiv:1711.02846 (2017) [3] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014) [4] Wang, X., Wang, H., Yang, D.: Measure and improve robustness in NLP models: A survey. CoRR abs/2112.08313 (2021) 2112.08313
28
[5] Alshemali, B., Kalita, J.: Improving the reliability of deep neural networks in nlp: A review. Knowledge-Based Systems 191, 105210 (2020) https://doi.org/10. 1016/j.knosys.2019.105210 [6] Qiu, S., Liu, Q., Zhou, S., Huang, W.: Adversarial attack and defense technologies in natural language processing: A survey. Neurocomputing 492, 278–307 (2022) https://doi.org/10.1016/j.neucom.2022.04.020 [7] Wiyatno, R.R., Xu, A., Dia, O., De Berker, A.: Adversarial examples in modern machine learning: A review. arXiv preprint arXiv:1911.05268 (2019) [8] Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., Madry, A.: Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152 (2018) [9] Yang, Y., Rashtchian, C., Zhang, H., Salakhutdinov, R., Chaudhuri, K.: Adversarial robustness through local lipschitzness. CoRR abs/2003.02460 (2020) 2003.02460 [10] Eberhard, D.M., Simons, G.F., Fennig, C.D.: Ethnologue: Languages of the World. SIL International. Online version (2025). https://www.ethnologue.com [11] Alajmi, A., Altabaa, H., Abed, S., Ahmad, I.: Arabic question generation using transformers. ACM Transactions on Asian and Low-Resource Language Information Processing 24(3), 1–21 (2025) [12] Habash, N.Y.: Introduction to Arabic Natural Language Processing. Springer (2022) [13] Al-Sulaiti, L., Atwell, E.S.: The design of a corpus of contemporary arabic. International journal of corpus linguistics 11(2), 135–171 (2006) [14] Antoun, W., Baly, F., Hajj, H.M.: Arabert: Transformer-based model for arabic language understanding. CoRR abs/2003.00104 (2020) 2003.00104 [15] Abdul-Mageed, M., Elmadany, A., et al.: Arbert & marbert: Deep bidirectional transformers for arabic. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7088–7105 (2021) [16] Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., Habash, N.: The interplay of variant, size, and task type in arabic pre-trained language models. In: 6th Arabic Natural Language Processing Workshop, WANLP 2021, pp. 92–104 (2021). Association for Computational Linguistics (ACL) [17] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
29
[18] Barbieri, F., Anke, L.E., Camacho-Collados, J.: Xlm-t: Multilingual language models in twitter for sentiment analysis and beyond. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 258–266 (2022) [19] Morris, J.X., Lifland, E., Lanchantin, J., Ji, Y., Qi, Y.: Reevaluating adversarial examples in natural language. arXiv preprint arXiv:2004.14174 (2020) [20] Qiu, S., Liu, Q., Zhou, S., Wu, C.: Review of artificial intelligence adversarial attack and defense technologies. Applied Sciences 9(5), 909 (2019) [21] Chen, Y., Su, J., Wei, W.: Multi-granularity textual adversarial attack with behavior cloning. arXiv preprint arXiv:2109.04367 (2021) [22] Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., John, R.S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al.: Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018) [23] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982–3992. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-1410 . https://aclanthology.org/D19-1410 [24] Shreya, G., Khapra, M.M.: A survey in adversarial defences and robustness in nlp. arXiv preprint arXiv:2203.06414 (2022) [25] Eger, S., Benz, Y.: From hero to zéroe: A benchmark of low-level adversarial attacks. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 786–803. Association for Computational Linguistics, Suzhou, China (2020). https://aclanthology.org/2020.aaclmain.79 [26] Formento, B., Foo, C.S., Tuan, L.A., Ng, S.K.: Using punctuation as an adversarial attack on deep learning-based NLP systems: An empirical study. In: Findings of the Association for Computational Linguistics: EACL 2023, pp. 1–34. Association for Computational Linguistics, Dubrovnik, Croatia (2023). https://aclanthology.org/2023.findings-eacl.1 [27] Chai, Y., Liang, R., Samtani, S., Zhu, H., Wang, M., Liu, Y., Jiang, Y.: Additive feature attribution explainable methods to craft adversarial attacks for text classification and text regression. IEEE Transactions on Knowledge and Data Engineering, 1–14 (2023) https://doi.org/10.1109/TKDE.2023.3270581 [28] Alshemali, B., Kalita, J.: Character-level adversarial examples in arabic. In:
30
2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 9–14 (2021). IEEE [29] Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 4765–4774. Curran Associates, Inc., ??? (2017) [30] Gao, J., Lanchantin, J., Soffa, M.L., Qi, Y.: Black-box generation of adversarial text sequences to evade deep learning classifiers. In: 2018 IEEE Security and Privacy Workshops (SPW), pp. 50–56 (2018). IEEE [31] Li, J., Ji, S., Du, T., Li, B., Wang, T.: Textbugger: Generating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271 (2018) [32] Morris, J.X., Lifland, E., Yoo, J.Y., Grigsby, J., Jin, D., Qi, Y.: Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiv preprint arXiv:2005.05909 (2020) [33] Li, L., Ma, R., Guo, Q., Xue, X., Qiu, X.: BERT-ATTACK: Adversarial attack against BERT using BERT. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6193–6202. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020. emnlp-main.500 . https://aclanthology.org/2020.emnlp-main.500 [34] Garg, S., Ramakrishnan, G.: BAE: BERT-based adversarial examples for text classification. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6174–6181. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020. emnlp-main.498 . https://aclanthology.org/2020.emnlp-main.498 [35] Alshemali, B., Kalita, J.: Adversarial examples in arabic. In: 2019 International Conference on Computational Science and Computational Intelligence (CSCI), pp. 371–376 (2019). IEEE [36] Ekbal, A., et al.: Adversarial sample generation for aspect based sentiment classification. In: Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pp. 478–492 (2022) [37] Alajmi, A., Ahmad, I., Mohammed, A.: Evaluating the adversarial robustness of arabic spam classifiers. Neural Computing and Applications 37(6), 4323–4343 (2025) [38] Alshalan, H., Rekabdar, B.: Attacking a transformer-based models for arabic language as low resources language (lrl) using word-substitution methods. In: 2023 Fifth International Conference on Transdisciplinary AI (TransAI), pp. 95–101 (2023). IEEE
31
[39] Yoo, J.Y., Qi, Y.: Towards improving adversarial training of nlp models. arXiv preprint arXiv:2109.00544 (2021) [40] Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR abs/1910.01108 (2019) 1910.01108 [41] Shen, L., Zhang, X., Ji, S., Pu, Y., Ge, C., Yang, X., Feng, Y.: Textdefense: Adversarial text detection based on word importance entropy. arXiv preprint arXiv:2302.05892 (2023) [42] Huber, L., Kühn, M.A., Mosca, E., Groh, G.: Detecting word-level adversarial text attacks via shapley additive explanations. In: Proceedings of the 7th Workshop on Representation Learning for NLP, pp. 156–166 (2022) [43] Mozes, M., Stenetorp, P., Kleinberg, B., Griffin, L.D.: Frequency-guided word substitutions for detecting textual adversarial examples. arXiv preprint arXiv:2004.05887 (2020) [44] Zhou, Y., Jiang, J.-Y., Chang, K.-W., Wang, W.: Learning to discriminate perturbations for blocking adversarial attacks in text classification. arXiv preprint arXiv:1909.03084 (2019) [45] Zhou, Y., Zheng, X., Hsieh, C.-J., Chang, K.-w., Huang, X.: Defense against adversarial attacks in nlp via dirichlet neighborhood ensemble. arXiv preprint arXiv:2006.11627 (2020) [46] Alshahrani, N., Alshahrani, S., Wali, E., Matthews, J.: Arabic synonym bert-based adversarial examples for text classification. arXiv preprint arXiv:2402.03477 (2024) [47] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997) [48] Kim, Y.: Convolutional neural networks for sentence classification. In: Moschitti, A., Pang, B., Daelemans, W. (eds.) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746– 1751. Association for Computational Linguistics, Doha, Qatar (2014). https: //doi.org/10.3115/v1/D14-1181 . https://aclanthology.org/D14-1181 [49] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017) [50] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018) 1810.04805
32
[51] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) [52] Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019) [53] Ribeiro, M.T., Singh, S., Guestrin, C.: ” why should i trust you?” explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016) [54] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, É., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440–8451 (2020) [55] Alajmi, A.M.: Arabic Sentiment Analysis [dataset], 2023. https://huggingface. co/datasets/Anwaarma/MySentimentAnwarBig/viewer/default/train?p=1 [56] Elmadany, A., Mubarak, H., Magdy, W.: Arsas: An arabic speech-act and sentiment corpus of tweets. OSACT 3, 20 (2018) [57] Barbieri, F., Espinosa Anke, L., Camacho-Collados, J.: XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 258–266. European Language Resources Association, Marseille, France (2022). https://aclanthology.org/2022.lrec-1.27 [58] Abdelali, A., Darwish, K., Durrani, N., Mubarak, H.: Farasa: A fast and furious segmenter for Arabic. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pp. 11–16. Association for Computational Linguistics, San Diego, California (2016). https://doi.org/10.18653/v1/N16-3003 . https://aclanthology.org/N163003 [59] Kudo, T., Richardson, J.: SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66–71. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-2012 . https://aclanthology.org/D18-2012 [60] Alajmi, A., Ahmad, I., Mohammed, A.: Evaluating the adversarial robustness of arabic spam classifiers. Neural Computing and Applications, 1–21 (2024) [61] Bani-Almarjeh, M., Kurdy, M.-B.: Arabic abstractive text summarization using
33
rnn-based and transformer-based architectures. Information Processing & Management 60(2), 103227 (2023) [62] Levenshtein, V.I., et al.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet Physics Doklady, vol. 10, pp. 707–710 (1966). Soviet Union [63] Yuan, X., He, P., Zhu, Q., Li, X.: Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems 30(9), 2805–2824 (2019) [64] Kadaoui, K., Magdy, S., Waheed, A., Khondaker, M.T.I., El-Shangiti, A., AbdulMageed, M., et al.: Tarjamat: Evaluation of bard and chatgpt on machine translation of ten arabic varieties. In: Proceedings of ArabicNLP 2023, pp. 52–75 (2023) [65] Khondaker, M.T.I., Waheed, A., Nagoudi, E.M.B., Abdul-Mageed, M.: Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp. arXiv preprint arXiv:2305.14976 (2023) [66] Bountakas, P., Zarras, A., Lekidis, A., Xenakis, C.: Defense strategies for adversarial machine learning: A survey. Computer Science Review 49, 100573 (2023) [67] Zhang, C., Zhou, X., Wan, Y., Zheng, X., Chang, K.-W., Hsieh, C.-J.: Improving the adversarial robustness of nlp models by information bottleneck. arXiv preprint arXiv:2206.05511 (2022) [68] Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., Kohli, P.: On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715 (2018) [69] Goyal, S., Doddapaneni, S., Khapra, M.M., Ravindran, B.: A survey of adversarial defenses and robustness in nlp. ACM Computing Surveys 55(14s), 1–39 (2023)
34