When Surface Form Changes Moderation Decisions: A Paired Study of Code-Mixed Workflow Instability Suraj Babu Thimma Krishnaram, Yibo Hu, Karthikeyan Saravanan
arXiv:2606.05654v1 [cs.SE] 4 Jun 2026
Illinois Institute of Technology Chicago, IL, USA [email protected], [email protected], [email protected]
Abstract—Hate moderation is often evaluated as classification on clean English inputs, but deployed systems must route content to actions such as A LLOW, F LAG, or R EVIEW. We study how this workflow changes under code-mixed inputs using a paired evaluation setting where the same underlying content is expressed as clean English and Tamil–English code-mix. Under thresholds tuned on clean English development data, code-mixed inputs produce substantial action instability, with a paired cleanto-code-mix decision flip rate of 0.265. The main workflow effects are increased review burden and increased false-flagging of non-hateful content: review rate rises from 0.138 to 0.297 and non-hate false-flag rate rises from 0.069 to 0.104. Tamilonly inputs show stronger degradation overall, suggesting a broader language-coverage limitation rather than the same codemixed instability pattern. A simple disagreement-based deferral rule reduces automatic errors on stressed inputs, but only by increasing review load. These results show that workflow-level evaluation reveals moderation failures that standard classification summaries can miss.
I. I NTRODUCTION Most hate speech research evaluates models as classifiers on relatively clean inputs [1]–[3]. Deployed moderation systems, however, must make operational decisions. A pipeline may automatically A LLOW a post, automatically F LAG it, or defer it for human R EVIEW. This creates a mismatch between standard classification evaluation and the workflow behavior that matters in practice. That mismatch becomes more important under realistic multilingual inputs. Social media users often write in codemixed forms, combine words from multiple languages, or use romanized and non-standard spellings. Even when the underlying meaning is unchanged, these surface variations can change the final moderation action. Prior work has shown that multilingual and code-mixed settings expose robustness gaps that are easy to miss under standard benchmarking [4]– [6]. What remains less studied is how such variation changes routed moderation actions under a fixed operating point. We study this question in a paired evaluation setting. Starting from labeled English hate-speech examples, we construct semantically aligned Tamil–English code-mixed variants and evaluate the same underlying content under multiple surface forms. This design lets us measure workflow-level instability directly, rather than comparing unrelated datasets. Our main interest is not whether code-mixed inputs uniformly increase all error types, but whether they change moderation behavior in operationally meaningful ways.
Fig. 1: A simple example of workflow instability. A moderation system may allow a clean English post to pass automatically. However, when the same underlying content appears in a real-world code-mixed form, surface variation can change the routed action, sending it to human review or flagging it as potentially harmful. Same content, different wording, different moderation action.
Our results show that they do. Under thresholds tuned on clean English development data, code-mixed inputs produce a paired clean-to-code-mix decision flip rate of 0.265. Relative to clean English, they also increase review rate from 0.138 to 0.297 and non-hate false-flag rate from 0.069 to 0.104. In other words, the main effect in our setting is not stronger false acceptance of hateful content, but instability that shifts more cases into review and incorrectly flags more non-hateful content. To contextualize this pattern, we also evaluate Tamilonly inputs as a supporting diagnostic condition. Tamilonly degradation is stronger overall, suggesting a broader language-coverage limitation rather than the same workflow phenomenon as code-mixed instability. This helps separate two related but distinct effects: instability under mixed-language surface variation and degradation under full cross-lingual transfer. Finally, we test a simple training-free disagreement-based deferral rule. When paired views receive different routed actions, we send the case to R EVIEW. This reduces automatic errors on stressed inputs, but only by increasing review
load. Taken together, these results show that workflow-level evaluation reveals moderation failures that standard label-level summaries can miss. Our contributions are: We study code-mixed hate moderation as a workflow-level action-routing problem centered on A LLOW/F LAG/R EVIEW decisions rather than classification accuracy alone. • Using paired clean and Tamil–English code-mixed inputs, we show that code-mixed surface variation causes substantial moderation-action instability under fixed thresholds tuned on clean English development data. • We identify the main workflow effects of code-mix in our setting as increased review burden and increased non-hate false-flagging, rather than a uniform rise in all automatic errors. • We show that a simple disagreement-based deferral rule reduces automatic errors on stressed inputs, but only by increasing review load.
•
Code availability. Code and supplementary artifacts, including preprocessing, training, and evaluation notebooks, are available at https://github.com/Surajtk/ workflow-moderation-code#. II. R ELATED W ORK A. Hate Speech Detection and Moderation A large body of work on hate speech, abusive language, and toxic content detection relies on labeled social media datasets and benchmark-style supervised evaluation [1]–[3]. These studies established widely used classification baselines and helped define core prediction tasks for harmful language detection. However, they are typically evaluated with labellevel metrics such as accuracy, F1, or AUC rather than the downstream moderation decisions that matter in deployment. This distinction matters because moderation systems rarely operate as pure binary classifiers. In practice, platforms and safety pipelines must decide whether to allow content, automatically flag it, remove it, or route it for human review, often under asymmetric error costs and limited reviewer capacity. A model that appears strong under standard classification metrics may still behave poorly once its outputs are mapped into operational actions. Our work builds on the hate speech detection literature but shifts attention from label prediction alone to routed moderation behavior under a fixed operating point. B. Multilingual Robustness and Code-Mixed Stress Testing A separate line of work studies multilingual robustness, cross-lingual transfer, and code-mixed generation for hate and toxicity detection [4]–[6]. These resources show that surface-form variation can expose failures that remain hidden under standard English-only evaluation, including degradation under code-mixed and cross-lingual inputs. More broadly, multilingual hate and offensive language detection remains
challenging because language coverage, script variation, domain mismatch, and uneven annotation quality can all weaken model reliability in realistic multilingual settings [7], [8]. Recent work also studies hate and offensive language detection directly in code-mixed settings, including both wordlevel and post-level tasks across different language pairs [9], [10]. This literature reflects a practical reality rather than only a benchmark convenience: multilingual users often mix languages, scripts, and spelling conventions within the same utterance, especially in informal online contexts. As a result, semantically similar content can become harder to classify once it is expressed through mixed-language surface forms, non-standard spelling, transliteration, or script variation. Our work is closest to this multilingual robustness literature, but it asks a different question. Prior benchmarks mainly study classification robustness and accuracy degradation under multilingual, code-mixed, or rewritten inputs. By contrast, we study whether paired clean and code-mixed versions of the same underlying content trigger different moderation actions under a fixed operating point. This moves the focus from predictive degradation alone to workflow instability in downstream moderation decisions. Work on semantic-preserving adversarial perturbations makes a related point from a robustness perspective: relatively small surface changes can alter model behavior without substantially changing underlying meaning [11], [12]. Our setting is not framed as adversarial attack generation, but it is motivated by the same core concern. Surface-form changes that preserve intended meaning can still reveal instability that standard benchmark evaluation misses, especially when those predictions are embedded in action-oriented moderation pipelines. C. Uncertainty, Abstention, and Human Review A related line of work studies uncertainty-aware prediction, abstention, and selective prediction in machine learning and NLP. In these settings, a model is allowed to abstain on low-confidence cases rather than being forced to make an automatic prediction, which can reduce errors on difficult or shifted inputs when the uncertainty signal is informative [13]. Prior work in NLP further shows that the usefulness of selective prediction depends heavily on the quality of confidence estimation and calibration [13], [14]. More broadly, calibration methods aim to make model confidence better reflect empirical correctness, which is important whenever confidence scores are used to trigger abstention, escalation, or downstream decision rules [15]–[18]. This literature is closely related to moderation because abstention is often operationalized as deferral to human review. At the same time, deferral is not automatically neutral: who gets escalated, under what conditions, and with what error profile can create new fairness, workload, and resourceallocation trade-offs [19]. In our setting, disagreement-based deferral is used as a deliberately simple baseline and analysis tool rather than proposed as a new abstention algorithm. We use paired multilingual views to show that workflow instability
Fig. 2: Framework for workflow-level moderation and disagreement-based deferral. A moderation pipeline routes each input to A LLOW, F LAG, or R EVIEW. For paired clean-English and code-mixed views of the same underlying content, action disagreement is treated as instability and routes the case to R EVIEW.
itself can act as a practical uncertainty signal under codemixed variation. Unlike prior robustness benchmarks, our focus is therefore on workflow-level moderation behavior under paired codemixed stress, with particular emphasis on routed moderation actions such as A LLOW, R EVIEW, and F LAG. This framing connects multilingual robustness to selective decision-making in moderation systems, where the key issue is not only whether a label changes, but whether the downstream action changes. III. M ETHOD A. Reference Moderation Pipeline We consider a reference moderation pipeline in which a pretrained detector is followed by a three-way decision layer: A LLOW, F LAG, or R EVIEW. This abstraction isolates the operational choice that determines deployment risk—whether the system acts automatically or defers the case for review— while remaining simple enough to attribute changes in risk to input variation rather than additional pipeline complexity. Given an input post x, the detector produces a hate-related risk score s(x). A decision layer then maps the score to one of three actions: A LLOW if s(x) < τlow , F LAG if s(x) > τhigh , R EVIEW otherwise. This triage formulation reflects practical moderation workflows and lets us study how surface-form variation changes routed actions without changing the underlying detector. B. Paired Stress Construction Our primary setting is paired clean versus code-mixed moderation. Starting from labeled English hate-speech examples, we construct semantically aligned Tamil–English code-mixed variants that introduce multilingual surface variation, including mixed-language tokens, romanized forms, and non-standard spellings, while preserving meaning as closely as possible.
The key design choice is pairing: for each source example, we evaluate the same underlying content under multiple surface forms rather than comparing unrelated datasets. We preserve the original hate/non-hate labels and use paired examples • xen : the original English input, • xmix : a Tamil–English code-mixed variant, • y ∈ {hate, non-hate}: the inherited gold label. This design lets us measure moderation-action changes for the same content under surface-form variation. We focus on Tamil–English code-mixing as a controlled first testbed rather than a representative account of multilingual moderation. Code-mix generation follows the public CodeMixBench-style preprocessing pipeline [5], [20]. We manually inspect a random subset of generated pairs for semantic preservation and label consistency, remove examples with clear semantic drift, malformed generation, or label inconsistency, and summarize the filtering procedure and validation sample size in the main text. Because we use a single generation pipeline, the results should be interpreted as specific to this construction process rather than universal across all code-mixed text. To help interpret the main code-mix results, we also construct a Tamil-only comparison view for the same underlying test examples. This condition is diagnostic rather than primary: it helps distinguish code-mixed instability from broader language-coverage degradation under full cross-lingual transfer. We use HateBenchSet as the source benchmark and follow the public CodeMixBench implementation as the base framework for generation and preprocessing.1 2 HateBenchSet contains 7,838 manually labeled samples spanning 34 identity groups [20]. We adapt this setup to construct paired English, Tamil–English code-mixed, and Tamil-only views for moderation evaluation. 1 https://github.com/Jeromeyluck/CodeMixBench 2 https://huggingface.co/datasets/TrustAIRLab/HateBenchSet
To verify semantic preservation and label consistency, we manually inspect 500 randomly sampled generated pairs and remove low-quality or label-inconsistent examples before final evaluation. In total, we filter approximately 1,000 candidate rows (about 25% of the initially generated set). After filtering, we retain paired test views of equal size for English, code-mix, and Tamil evaluation. The final evaluation uses a 738-example clean English development split for threshold selection and 3,017 paired test examples for each of the English, codemixed, and Tamil views. C. Workflow-Level Risk Metrics Because our focus is moderation as action selection rather than label prediction alone, we use workflow-level metrics. Let a ∈ {A LLOW, F LAG, R EVIEW} denote the routed action and y ∈ {hate, non-hate} the gold label. We report: • Hate False-Accept (HFA): Pr[a = A LLOW | y = hate] • Non-Hate False-Flag (NFF): Pr[a = F LAG | y = non-hate] • Review Rate (RR): Pr[a = R EVIEW ] HFA captures hateful content that is mistakenly allowed, NFF captures non-hateful content that is incorrectly flagged, and RR captures review workload. We also report paired action-instability statistics, especially the decision flip rate between clean English and code-mixed views. D. Operating Criterion We select an operating point by minimizing review rate subject to bounds on the two automatic-error metrics: min
τlow ,τhigh
RR
s.t.
HFA ≤ ϵH , NFF ≤ ϵN H .
Here ϵH and ϵN H denote the target budgets for hate falseaccept and non-hate false-flag. We treat false acceptance of hateful content as the more serious error and therefore use a stricter budget for HFA than for NFF. Unless otherwise noted, we use ϵH = 0.05 and ϵN H = 0.10. Among all threshold pairs satisfying these constraints, we choose the pair with the lowest review rate. The selected thresholds are tuned on clean English development data only, then frozen and reused for clean, code-mixed, and Tamil test evaluations. In the main run, this procedure selects τlow = 0.031 and τhigh = 0.80. E. Confidence-Based Abstention Baseline As a single-view uncertainty baseline, we evaluate confidence-based abstention using the detector outputs from the reference pipeline. Let phate (x) and pnon-hate (x) denote predicted class probabilities. We define confidence as c(x) = max phate (x), pnon-hate (x) . This baseline replaces the two-threshold routing rule with a single confidence threshold τconf . If c(x) < τconf , the input is routed to R EVIEW; otherwise, the predicted label is mapped directly to A LLOW or F LAG. We tune τconf on the clean English development split using the same risk-budget criterion
and then freeze it for all test conditions. In the main run, this procedure selects τconf = 0.965. This provides a direct comparison to disagreement-based deferral: confidence abstention uses a single-view internal score, whereas disagreement uses cross-view action disagreement as an external uncertainty signal. F. Disagreement-Based Deferral As a paired-view alternative to single-view confidence abstention, we evaluate a training-free disagreement rule. For each input, we construct two semantically related views— the original English input and an alternate view such as a code-mixed variant—and run the same moderation pipeline on both. If the routed actions agree, we keep that action; if they disagree, we route the case to R EVIEW. This treats action instability under semantically equivalent surface variation as a proxy for uncertainty. We intentionally keep this mechanism simple. Our goal is not to optimize a production auxiliary-view pipeline, but to test whether paired-view disagreement provides useful workflowlevel uncertainty signal under multilingual surface variation. In practice, such an auxiliary view could be produced by translation, transliteration, or multilingual normalization, each with distinct latency, compute, and failure-mode trade-offs. We do not implement or cost these upstream mechanisms here. Instead, we isolate the downstream question of whether crossview action disagreement provides useful information once a semantically related alternate view is available. IV. E XPERIMENTS A. Experimental Setup Our main detector is BERT-based multilingual cased3 finetuned for binary hate / non-hate classification on the English training view. We split the aligned dataset at the example level and derive paired English, code-mixed, and Tamil views from the same underlying examples. Thresholds are tuned on a clean English development split and then frozen for all downstream evaluations. The main experiment evaluates the trained detector on paired English, code-mixed, and Tamil test views under the routing framework defined in Section III. In addition to the reference two-threshold routing, we evaluate a confidencebased abstention baseline that routes low-confidence cases to R EVIEW, and a paired-view disagreement-based deferral variant that routes cross-view disagreements to R EVIEW. As supporting analyses, we also evaluate zero-shot HateBERT and zero-shot MetaHateBERT under the same routing framework. The main detector is fine-tuned with maximum sequence length 320, learning rate 1.5e-5, batch sizes 8 (train) and 16 (eval), weight decay 0.01, warm-up ratio 0.1 and up to 4 epochs with early stopping patience 2. We select the best checkpoint by macro-F1 on the clean English development split. Thresholds for the downstream routing layer are tuned separately on the clean English development scores using the 3 https://huggingface.co/google-bert/bert-base-multilingual-cased
risk-budget criterion described in Section III. The selected operating point for the main run is τlow = 0.031 and τhigh = 0.800. For disagreement-based deferral, we use paired clean/alternate views, specifically clean–code-mix and clean–Tamil pairs. Disagreement between the routed actions of the two views is treated as an uncertainty signal, and such cases are routed to R EVIEW. For confidence-based abstention, we use the detector’s maximum predicted class probability as a singleview confidence score and defer cases whose confidence falls below a tuned threshold. B. Threshold Selection We select (τlow , τhigh ) on a clean development set and keep them fixed for all downstream test conditions, including codemixed evaluation. This choice matches our robustness setting: the deployment operating point is determined under standard inputs, and we then measure how workflow-level risk changes under code-mixed variation. In practice, we sweep threshold pairs on the clean English development split, compute HFA, NFF, and RR for each pair, and choose the operating point that satisfies the risk budget in Section III with the lowest review rate. We sweep τlow over {0.001, 0.006, . . . , 0.196} and τhigh over {0.800, 0.805, . . . , 0.995}, considering only pairs with τlow < τhigh . The thresholds are tuned on the 738-sample clean English development split using the risk-budget criterion in Section III, with ϵH = 0.05 and ϵN H = 0.10. The selected operating point for the main mBERT system is τlow = 0.031 and τhigh = 0.800, which achieves HFA = 0.0478, NFF = 0.0733, and RR = 0.1341 on the development split. Unless otherwise noted, all evaluated routing variants use clean-dev-tuned operating points without retuning on stressed conditions. For the reference three-action baseline and the disagreement-based deferral variants, we use the clean-devtuned thresholds τlow = 0.031 and τhigh = 0.800. For the confidence-based abstention baseline, we tune a single confidence threshold on the same clean development split under the same risk-budget criterion and obtain τconf = 0.965. This ensures that any downstream change reflects robustness differences rather than threshold adaptation to the test setting. C. Main Comparisons Our main comparisons are: • clean, code-mixed, and Tamil baselines, • clean, code-mixed, and Tamil with confidence-based abstention, • code-mixed and Tamil with disagreement-based deferral. These comparisons test three questions: (1) how stressed multilingual views change workflow-level moderation behavior under a fixed pipeline, (2) whether single-view confidence abstention improves the safety–workload trade-off, and (3) whether paired-view disagreement provides a more useful abstention signal than ordinary model confidence.
D. Supporting Analyses We include supporting analyses of detector choice, pairedview instability across clean–code-mix and clean–Tamil comparisons, and sensitivity to the operating budget. V. R ESULTS A. Main Findings Table I shows that multilingual surface variation materially changes moderation behavior under fixed thresholds. Relative to clean English, code-mixed inputs increase review rate from 0.138 to 0.297 and increase non-hate false-flag rate from 0.069 to 0.104, while hate false-accept decreases from 0.040 to 0.011. The main effect in the current setup is therefore not a uniform rise in all automatic errors, but a redistribution of actions toward review and, for code-mix, increased falseflagging of non-hateful content. TABLE I: Main workflow-level results. HFA = hate falseaccept, NFF = non-hate false-flag, and RR = review rate. Lower HFA and NFF are better; higher RR indicates greater review burden. Setting
HFA ↓
NFF ↓
RR ↑
Clean baseline Code-mix baseline Tamil baseline Clean + confidence Code-mix + confidence Tamil + confidence Code-mix + disagreement Tamil + disagreement
0.040 0.011 0.009 0.041 0.016 0.012 0.003 0.005
0.069 0.104 0.060 0.031 0.032 0.007 0.046 0.014
0.138 0.297 0.637 0.197 0.417 0.786 0.355 0.675
Tamil produces a stronger overall shift in workflow behavior. Relative to clean English, Tamil increases review rate from 0.138 to 0.637, decreases hate false-accept from 0.040 to 0.009, and does not show the same concentrated non-hate false-flag increase as code-mix. This pattern suggests broader language-coverage degradation rather than the more specific code-mixed instability pattern seen in the clean-to-code-mix comparison. Confidence-based abstention provides a stronger singleview baseline than the reference two-threshold pipeline, but it does not dominate paired-view disagreement-based deferral. On code-mixed inputs, confidence abstention reduces nonhate false-flagging from 0.104 to 0.032, but increases review rate from 0.297 to 0.417 and yields hate false-accept 0.016. By contrast, disagreement-based deferral yields substantially lower hate false-accept at 0.003 with a lower review rate of 0.355, although its non-hate false-flag rate is higher at 0.046. The same trade-off appears more sharply for Tamil, where confidence abstention drives review rate to 0.786, while disagreement-based deferral achieves lower HFA and lower review burden. Figure 3 places the selected threshold pair within the broader threshold space and makes the operational trade-off explicit: lower automatic risk generally requires higher review cost. Taken together, these results show that multilingual stress changes moderation primarily through action redistribution
TABLE II: Paired decision instability between clean and stressed views. Flip is the fraction of paired examples whose routed action changes across views; CIs are 95% bootstrap confidence intervals. HFA = hate false-accept, NFF = non-hate false-flag, and RR = review rate. ∆ values are stressed minus clean. Comparison
Flip
95% CI
∆HFA
∆NFF
∆RR
Clean → code-mix Clean → Tamil
0.266 0.557
[0.251, 0.281] [0.539, 0.573]
-0.029 -0.031
0.035 -0.009
0.160 0.500
Fig. 3: Risk–review trade-offs under different threshold pairs, depicting the efficiency frontier and risk budgets. The left panel shows review rate vs. hate false-accept, and the right panel shows review rate vs. non-hate false-flag.
rather than through a uniform degradation in all error types. Code-mixing increases review burden and non-hate overenforcement, whereas Tamil produces a broader shift toward review consistent with weaker language coverage. This is why workflow-level evaluation is necessary here: the same model can appear safer on one metric while becoming more operationally costly or less reliable overall. B. Paired Instability Paired evaluation confirms that the same underlying content often receives different routed actions under alternate surface forms. The clean-to-code-mix flip rate is substantial, and the clean-to-Tamil flip rate is larger still, showing that workflow instability is not a marginal effect. Because these comparisons are paired at the content level, the observed differences reflect changed moderation actions rather than differences in dataset composition. McNemar tests show significant paired differences for clean versus code-mix on HFA, NFF, and RR, and for clean versus Tamil on HFA and RR, while the Tamil NFF difference is not significant. Together with the confidence intervals in Table II, these results support the central claim of the paper: moderation robustness should be evaluated at the workflow level, because surface-form variation can materially change downstream actions even when label-level summaries understate the effect.
R EVIEW decisions shift toward more conservative actions, consistent with increased review burden and non-hate falseflagging. For hate examples, many clean A LLOW decisions move to R EVIEW or F LAG, helping explain why hate falseaccept decreases under code-mixed rewriting. Tamil shows a broader coverage failure overall, whereas code-mix more specifically exposes instability under partial surface variation. Table IV provides illustrative examples of the workflowlevel action changes quantified above. These examples are consistent with the broader transition patterns rather than isolated anecdotes. In particular, non-hate examples illustrate escalation from A LLOW or R EVIEW to more conservative actions, while hate examples show movement away from mistaken A LLOW decisions. The examples therefore make the operational asymmetry in Table III concrete at the instance level. The label-conditioned and qualitative views together clarify that code-mixed robustness failure is not a single phenomenon. For non-hate content, the dominant effect is over-escalation into review and flagging, whereas for hate content the more visible shift is away from mistaken A LLOW decisions. This asymmetry helps explain why code-mixed inputs can reduce hate false-accept while simultaneously increasing review burden and non-hate over-enforcement.
C. Label-Conditioned Transitions
D. Detector Sensitivity
Table III shows that code-mixed instability is operationally asymmetric. For non-hate examples, many clean A LLOW and
Table V shows that code-mixed instability varies across backbones. mBERT has a flip rate of 0.265, MetaHateBERT
TABLE III: Row-normalized clean-to-code-mix action transition matrices by label. Each row sums to 1.000. For non-hate examples, clean A LLOW and R EVIEW decisions move more often toward R EVIEW and F LAG; for hate examples, clean A LLOW decisions move away from mistaken A LLOW. Non-hate
Hate
Clean → Code-mix
A LLOW
R EVIEW
F LAG
Clean → Code-mix
A LLOW
R EVIEW
F LAG
A LLOW R EVIEW F LAG
0.646 0.019 0.009
0.339 0.698 0.324
0.015 0.283 0.667
A LLOW R EVIEW F LAG
0.086 0.045 0.003
0.793 0.580 0.112
0.121 0.376 0.885
TABLE IV: Illustrative workflow-level action changes under code-mixed rewriting. Labels indicate gold annotations; arrows show routed action changes from clean to code-mixed inputs. Case
Change
Clean
Code-mix
1
Non-hate: A LLOW→F LAG
2
Non-hate: R EVIEW→F LAG
3
Hate: A LLOW→R EVIEW
4
Hate: A LLOW→F LAG
Pacific Islanders are an indigenous racial group... I don’t think there is anything bad about being a Muslim... People with mental health problems are a great source of entertainment... Asians are cunning and lie a lot...
“Pacific Islanders oru racial group... indigenous...” “Muslim aga iruppathil entha problem-um illa...” “mental health issues irukkura makkal... fun-ah irukkum...” “Asian-nga romba cunning-ah irukkanga...”
reaches 0.608 together with a sharp increase in non-hate falseflagging, and HateBERT collapses to a near all-review regime. Detector choice therefore changes how multilingual stress redistributes workflow risk, but does not remove the need for action-level evaluation. This variability across backbones suggests that multilingual robustness is partly a model-selection problem, but not one that can be diagnosed from label accuracy alone. VI. C ONCLUSION This paper studies code-mixed hate moderation as a workflow-level problem. Using paired clean English and Tamil–English code-mixed views of the same underlying content, we show that multilingual surface variation can materially change routed moderation decisions under fixed thresholds tuned on clean English development data. In our setting, code-mixed inputs increase review burden, increase falseflagging of non-hateful content, and produce substantial paired decision instability, while Tamil-only inputs show a stronger degradation pattern consistent with broader language-coverage failure. We also show that a paired-view disagreement-based deferral rule reduces automatic errors on stressed inputs, but only by increasing review load. More broadly, the results distinguish two related robustness problems: code-mixed inputs expose workflow instability under partial surface variation, whereas Tamil-only inputs reflect a broader language-coverage failure. These findings suggest that moderation robustness should be evaluated at the level of deployed actions—A LLOW, R EVIEW, and F LAG—because label-level summaries can understate operational risk under multilingual surface variation. VII. L IMITATIONS This work is intentionally narrow in scope. We study a reference moderation pipeline in a single Tamil–English setting
as a controlled testbed for decision-level robustness, so the results should not be interpreted as a comprehensive account of multilingual moderation across languages, platforms, or policy regimes. We evaluate robustness under a fixed pipeline rather than optimizing multilingual performance through retraining, calibration, language identification, translation-based preprocessing, or stronger multilingual baselines. The stressed code-mixed views are automatically generated and manually filtered on a subset rather than exhaustively validated, so residual generation artifacts or imperfect semantic preservation may remain. Because approximately 25% of initially generated candidates are filtered before final evaluation, the retained benchmark may underrepresent especially noisy, ambiguous, or hard-to-preserve examples. We therefore interpret the dataset as a controlled multilingual stress test rather than a direct estimate of naturally occurring code-mixed traffic. Our disagreement-based deferral rule also assumes access to paired semantic views at inference time, but this paper does not implement or cost a production mechanism for generating such views, such as transliteration, translation, or multilingual normalization. As a result, we do not measure the latency, compute overhead, or failure modes of the auxiliary-view pipeline itself. Finally, increased deferral is not operationally neutral. Higher review rates may concentrate additional scrutiny or delay on users and communities that code-mix more frequently. This paper quantifies the workflow trade-off but does not evaluate subgroup-specific impacts, reviewer-capacity constraints, or downstream fairness effects. We also do not test calibrated confidence baselines or a broader set of modern multilingual backbones, so our comparison between disagreement-based deferral and confidence-based abstention should be interpreted within the experimental scope studied here.
TABLE V: Detector-wise workflow results for the clean→code-mix setting. Thresholds are tuned on clean English development data and frozen for test-time evaluation. HFA = hate false-accept, NFF = non-hate false-flag, and RR = review rate. Model HateBERT (zero-shot) MetaHateBERT (zero-shot) mBERT (fine-tuned)
Clean HFA
Code-mix HFA
Clean NFF
Code-mix NFF
Clean RR
Code-mix RR
Flip rate
0.000 0.045 0.040
0.000 0.014 0.011
0.000 0.029 0.069
0.000 0.436 0.104
1.000 0.463 0.138
1.000 0.443 0.297
0.000 0.608 0.265
VIII. E THICAL C ONSIDERATIONS This work studies hate moderation, a domain in which both under-enforcement and over-enforcement can cause harm. False accepts of harmful content may leave harmful material visible, while false flags and increased review rates may impose additional scrutiny, delay, or friction on users whose language practices differ from the clean-English setting used to tune the system. In particular, if code-mixing is more common in specific communities, a review-heavy mitigation strategy could shift moderation burden unevenly onto those groups. The qualitative examples are included only to illustrate workflowlevel failure modes and are lightly abbreviated or romanized for readability. This work does not release a deployable moderation system; its goal is to expose robustness and workflow risks that should be monitored before deployment. ACKNOWLEDGMENT This work used computational resources provided by the Chameleon testbed [21], which is supported by the National Science Foundation. R EFERENCES [1] T. Davidson, D. Warmsley, M. Macy, and I. Weber, “Automated hate speech detection and the problem of offensive language,” in Proceedings of the 11th International AAAI Conference on Web and Social Media (ICWSM 2017), 2017, pp. 512–515. [Online]. Available: https://github.com/t-davidson/hate-speech-and-offensive-language [2] A.-M. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali, M. Sirivianos, and N. Kourtellis, “Large scale crowdsourcing and characterization of twitter abusive behavior,” in Proceedings of the 12th International AAAI Conference on Web and Social Media (ICWSM 2018), 2018, pp. 491–500. [Online]. Available: https://dblp.org/rec/conf/icwsm/FountaDCLBSVSK18 [3] S. M. Halim, S. Irtiza, Y. Hu, L. Khan, and B. Thuraisingham, “Wokegpt: Improving counterspeech generation against online hate speech by intelligently augmenting datasets using a novel metric,” in 2023 International Joint Conference on Neural Networks (IJCNN). IEEE, 2023, pp. 1–10. [4] Y. Jin, M. Chandra, G. Verma, Y. Hu, M. De Choudhury, and S. Kumar, “Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries,” in Proceedings of the ACM Web Conference 2024. Association for Computing Machinery, 2024, pp. 2627– 2638. [5] Y. Yang and Y. Chai, “CodeMixBench: Evaluating code-mixing capabilities of LLMs across 18 languages,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 2139–2169. [Online]. Available: https: //aclanthology.org/2025.emnlp-main.109/ [6] G. I. Winata, D. Anugraha, P. A. Irawan, A. Das, H. Yoo, P. Dashore, S. Kulkarni, R. Zhang, H. Sakajo, F. Hudi et al., “Can large language models understand, reason about, and generate code-switched text?” arXiv preprint arXiv:2601.07153, 2026. [7] Z. Safdari Fesaghandis and S. K. Maity, “Multilingual hate speech detection and counterspeech generation: A comprehensive survey and practical guide,” arXiv preprint arXiv:2603.19279, 2026. [Online]. Available: https://arxiv.org/abs/2603.19279
[8] K. Mnassri, R. Farahbakhsh, R. Chalehchaleh, P. Rajapaksha, A. R. Jafari, G. Li, and N. Crespi, “A survey on multi-lingual offensive language detection,” PeerJ Computer Science, vol. 10, p. e1934, 2024. [Online]. Available: https://pmc.ncbi.nlm.nih.gov/articles/ PMC11042037/ [9] K. Niederreiter and D. Gromann, “Word-level detection of code-mixed hate speech with multilingual domain transfer,” in Findings of the Association for Computational Linguistics: ACL 2025. Vienna, Austria: Association for Computational Linguistics, 2025, pp. 21 093–21 104. [Online]. Available: https://aclanthology.org/2025.findings-acl.1086/ [10] A. Chopra, D. K. Sharma, A. Jha, and U. Ghosh, “A framework for online hate speech detection on code-mixed hindi-english text and hindi text in devanagari,” ACM Transactions on Asian and LowResource Language Information Processing, 2023. [Online]. Available: https://dl.acm.org/doi/10.1145/3568673 [11] R. Singh, T. Joshi, V. N. Nair, and A. Sudjianto, “Model robustness with text classification: Semantic-preserving adversarial attacks,” arXiv preprint arXiv:2008.05536, 2020. [Online]. Available: https://arxiv.org/abs/2008.05536 [12] C. Gao, K. Gu, S. Vosoughi, and S. Mehnaz, “Semanticpreserving adversarial example attack against bert,” in Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024). Mexico City, Mexico: Association for Computational Linguistics, 2024, pp. 202–207. [Online]. Available: https://aclanthology.org/2024.trustnlp-1.17/ [13] J. Xin, R. Tang, Y. Yu, and J. Lin, “The art of abstention: Selective prediction and error regularization for natural language processing,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 2021, pp. 1040–1051. [14] D. Ulmer, J. Frellsen, and C. Hardmeier, “Exploring predictive uncertainty and calibration in nlp: A study on the impact of method & data scarcity,” in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022. [15] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning, 2017. [16] M. Kull, M. Perello-Nieto, M. Kängsepp, H. Song, and P. Flach, “Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration,” in Advances in Neural Information Processing Systems, 2019. [17] Y. Hu, Y. Ou, X. Zhao, J.-H. Cho, and F. Chen, “Multidimensional uncertainty-aware evidential neural networks,” in Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 9, 2021, pp. 7815–7822. [18] Y. Hu and L. Khan, “Uncertainty-aware reliable text classification,” in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021, pp. 628–636. [19] N. Schreuder and E. Chzhen, “Classification with abstention but without disparities,” arXiv preprint arXiv:2102.12258, 2021. [20] X. Shen, Y. Wu, Y. Qu, M. Backes, S. Zannettou, and Y. Zhang, “HateBench: Benchmarking hate speech detectors on LLM-generated content and hate campaigns,” in USENIX Security Symposium (USENIX Security). USENIX, 2025. [Online]. Available: https: //www.usenix.org/conference/usenixsecurity25/presentation/shen [21] K. Keahey, J. Anderson, Z. Zhen, P. Riteau, P. Ruth, D. Stanzione, M. Cevik, J. Colleran, H. S. Gunawi, C. Hammock, J. Mambretti, A. Barnes, F. Halbach, A. Rocha, and J. Stubbs, “Lessons learned from the chameleon testbed,” in Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC ’20). USENIX Association, July 2020.