What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cryptography, security, privacy, cybersecurity

What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks Qin Yang1∗, Lu Malloy2∗, Joshua Lee3 , Xiaohan Chang1 , Meisam Mohammady4 Doowon Kim2 , Yuan Hong1

arXiv:2606.09700v1 [cs.CR] 8 Jun 2026

1 University of Connecticut, 2 University of Tennessee, 3 University of California, Santa Barbara, 4 Iowa State University

Abstract Large language model (LLM)-powered content moderation systems have become a critical defense against harmful online content. However, these systems primarily operate on tokenized text and largely ignore the visual cues that humans naturally rely on when interpreting content. We show that this discrepancy creates a fundamental perceptual mismatch: content that is readily recognized as harmful by humans can become effectively invisible to automated moderation systems. To expose and systematically study this vulnerability, we introduce a class of Human-Perceptible Adversarial Attacks (HPAA), in which harmful expressions are embedded into otherwise benign text through visually salient typographic manipulations. Our key insight is that common typographic features, including spacing, visual emphasis, and spatial arrangement, can be strategically combined to preserve human recognition of harmful content while substantially reducing machine detectability. Operating in black-box settings with only a small query budget, our attack automatically generates evasive content without requiring model access or gradient information. We evaluate the attack across multiple datasets and ten widely deployed moderation systems, including commercial APIs and state-of-the-art open-source guardrails. Results reveal a striking gap between human and machine perception: with only three detector queries, generated attacks achieve over 86% human recognition while maintaining detection rates below 1% across the evaluated systems. We further conduct ablation studies to identify the typographic factors driving successful evasion, analyze why current moderation architectures fail to capture these signals, and discuss practical defenses. Our findings expose a fundamental blind spot in today’s LLM-based moderation ecosystem and highlight need for moderation systems that reason about content in a manner more consistent with human perceptual understanding. Disclaimer: This paper includes examples of harmful, hateful, or abusive language for research purposes. Reader discretion is advised. ∗ Equal contribution.

Introduction

Modern online platforms increasingly rely on automated moderation systems to determine which content reaches end users. As large language models (LLMs) become integrated into moderation pipelines, these systems now serve as the primary gatekeepers for detecting toxic, hateful, misleading, and policy-violating content at scale [3, 9, 21]. In many modern moderation pipelines, automated detectors serve as the first decision point in determining whether content reaches end users. Consequently, failures in automated moderation can directly translate into harmful content exposure at scale, making the robustness of moderation systems an increasingly important security concern [6, 64, 67]. Most existing research on moderation robustness assumes that humans and models perceive content in broadly similar ways. Under this assumption, adversarial attacks seek to induce model failures through textual perturbations, paraphrasing, character substitutions, or semantic manipulations while preserving the original meaning [16, 17, 20, 22, 23, 32, 37, 52]. Similarly, jailbreak attacks [24, 63] and prompt-injection attacks [26,69] attempt to circumvent safety mechanisms by manipulating model instructions or prompting behavior. Toxicityevasion attacks further explore ways to bypass automated moderation systems through rephrasing, masking, or stylizing harmful content [35, 44, 55, 61, 70]. Despite their differences, these attack classes share a common premise: success is evaluated primarily through model behavior. In this paper, we investigate a fundamentally different failure mode. Human readers do not rely solely on lexical content when interpreting messages. Typography, including spacing, visual emphasis, and spatial arrangement, plays an important role in how information is perceived and understood. Modern moderation systems, however, primarily reason over tokenized textual representations and largely disregard these visual signals. This discrepancy raises an important question: can harmful content remain readily recognizable to humans while becoming effectively invisible to automated moderation systems?

Human-interpretable Patterns & Semantics as Primary Evasion Channel

Blocked by Moderation

Attacker

Stylized / Perturbed Message Human-interpretable patterns & semantics

HPAA: A Socio-Technical Attack Surface Human perception bridges attacker intent and detector evasion Not Jailbreaking Not Prompt Injection

Unsafe

Text-based Human Review Moderation Automated (Limited / System Delayed) Detectors

Safe

Pass-through

Online Platform Moderation Pipeline Detector View Invisible to detectors

Human fails to perceive harmful content

Perceptible and harmful to humans

Human View Interpretable to humans and perceptible as harm

Figure 1: Overview of typographic Human-perceptible Adversarial Attacks (HPAA) against modern content moderation pipelines. Harmful content remains recognizable to human readers while evading automated moderation systems. We answer this question affirmatively. We show that harmful content can be embedded into otherwise benign text through visually salient typographic manipulations that preserve human interpretation while substantially reducing machine detectability. This phenomenon reveals a previously underexplored perceptual mismatch between human readers and modern moderation systems: content that is readily recognized as harmful by humans can become effectively invisible to automated moderation systems. To systematically study this vulnerability, we present the first comprehensive investigation of Human-Perceptible Adversarial Attacks (HPAA) in text moderation systems. We instantiate HPAA through typographic manipulations that leverage common typographic features such as spacing, visual emphasis, and spatial arrangement to evade moderation while remaining readily understandable to humans. Figure 1 illustrates the core intuition. Unlike conventional adversarial text attacks, which primarily seek to induce model failures through textual perturbations, typographic HPAA exploits visual cues that remain readily interpretable to humans but are often disregarded by moderation systems. As a result, harmful content can remain recognizable to human readers while substantially reducing its detectability by automated moderation systems. The attack is motivated by a realistic moderation setting that is increasingly common in practice. In modern tiered moderation pipelines, user-generated content is first screened by automated moderation systems and subsequently delivered to end users if no violation is detected. Under this architecture, an attacker does not need to deceive both humans and machines. Rather, the attacker only needs to bypass the automated moderator while ensuring that the harmful message remains intelligible to the eventual human recipient. As a result, typographic attacks expose a new socio-technical attack surface that emerges specifically from the gap between machine moderation and human perception. To evaluate this vulnerability, we conduct a comprehensive

study spanning multiple datasets and ten widely deployed moderation systems, including both commercial APIs and state-of-the-art open-source guardrails [29, 31]. Our results reveal that the perceptual mismatch identified in this work is not an isolated artifact of a particular model or platform, but a systematic weakness that persists across diverse moderation architectures. Across all evaluated systems, content that remains highly recognizable to human readers can achieve extremely low detection rates by automated moderation systems.

Preliminaries

Content Moderation Pipelines. Modern online platforms commonly deploy automated moderation pipelines to screen user-generated content before it reaches end users. Submitted content is first processed by an automated moderator, which determines whether the content should be accepted, blocked, or escalated for further review. Accepted content is subsequently delivered to users, whereas potentially unsafe content may be removed or forwarded to human moderators. Modern moderation pipelines are increasingly powered by large language models (LLMs), including proprietary moderation APIs and open-source guardrail models [29, 31]. These systems typically serve as the first line of defense against harmful content on modern online platforms. Throughout this paper, we model the automated moderator as a detector that maps an input text to a moderation decision. The detector may expose either confidence scores or discrete labels, and our formulation is agnostic to its internal architecture. Without loss of generality, we use the term toxicity1 to broadly refer to policy-violating content, including hate speech, harassment, threats, self-harm, sexual content, and related forms of abuse. 1 The terms “toxic”, “harmful” and “unsafe” are used interchangeably through-

out this paper.

3 3.1

Attack Design

3.3

The key idea behind HPAA is that humans do not rely solely on lexical content when interpreting text. Instead, visual cues such as spacing, spatial arrangement, and visual emphasis also influence how information is perceived and reconstructed. To systematically study how these cues affect both human recognition and automated moderation, we organize typographic manipulations into three complementary design dimensions: typographic granularity, placement strategy, and stylistic transformation. Together, these dimensions define the typographic configuration space explored in this work. Figure 2 illustrates the three dimensions. • Typographic Granularity (L) determines how toxic content is decomposed before being embedded into surrounding benign text. Toxic expressions may be preserved as complete words (Word), segmented into token-level units (Token), or represented using mixed token–character decompositions (Mixed). Different granularities influence how readily humans and moderation systems can reconstruct the original content. • Placement Strategies (M) determine how toxic fragments are spatially arranged within benign text. We consider a range of representative layouts, including vertical (M1 − M3 ), diagonal (M4 − M5 ), and randomized placements (M6 ). These strategies alter the visual organization of toxic content and may enable human readers to perceive coherent patterns that are not explicitly reflected in the sequential text representation processed by moderation systems. • Stylistic Transformations (S) modify the visual appearance of selected characters or tokens. We consider commonly used typographic cues such as bolding (Bold), color changes (Col), highlighting (Hi), capitalization (Cap), precomposed Unicode characters (Precomposed), and masking-based transformations (Colze). These modifications can increase the visual salience of toxic content for human readers while remaining weakly represented in tokenized text. By combining granularity, placement, and style, HPAA generates a diverse set of typographic configurations that vary in both human perceptibility and machine detectability. Each configuration represents a distinct combination of visual cues that may differentially affect how humans and moderation systems interpret the same content. Figure 2 also presents an illustrative example using the M2 -Word-Highlight configuration, where toxic words are preserved at the word level, vertically arranged within benign text, and visually highlighted to remain recognizable to human readers. Design Rationale for Stage 1. The configuration space defined above contains many possible cue combinations, and it is unclear a priori which configurations best preserve human recognition. Rather than relying on manual intuition, we conduct a two-round user study to systematically identify perceptually effective configurations. The resulting humanranked configuration set is subsequently used to guide attack generation and evaluation.

Problem Formulation

We consider a black-box moderation setting in which an attacker can only query a content moderation system and observe its outputs. Given a toxic input, the attacker aims to generate a typographically modified sample that remains recognizable as harmful to human readers while avoiding detection by the moderation system. Unlike conventional adversarial text attacks, which primarily focus on inducing model misclassification, our attack aims to create a discrepancy between human perception and detector judgments. Specifically, we seek typographic configurations that preserve human recognition of harmful content while causing automated moderation systems to classify the content as benign. To systematically characterize this human–LLMs perceptual mismatch and guide the design of HPAA, we investigate the following research questions: • RQ1: To what extent can human readers recognize toxic content embedded within typographically manipulated text, and how does recognition vary across different typographic configurations? • RQ2: To what extent do automated moderation systems detect typographically manipulated toxic content, and how does their performance differ from human recognition? • RQ3: To what extent can the typographic cue configurations identified from human and moderation analyses enable effective moderation evasion in practical settings?

3.2

Typographic Configuration Space

Threat Model

Attacker. We consider a non-privileged adversary who submits content through standard online platforms (e.g., social media or online forums) using ordinary user accounts. Knowledge and Capabilities. The adversary has no access to the moderation system’s architecture, parameters, training data, or moderation logic. The attacker interacts with the moderation pipeline in a purely black-box manner and can only observe moderation outcomes, such as safe/unsafe decisions and, when available, confidence scores. The attacker is restricted to content-level manipulations and cannot influence platform policies, moderation rules, or system deployment. Goal. The attacker’s goal is to generate content that remains recognizable as harmful to human readers while receiving a benign moderation decision, thereby enabling the content to bypass automated moderation and reach end users. Threat Surface. We consider the automated content moderation pipeline as the primary threat surface, including the text submission interface and the moderation mechanisms that determine whether content is released, blocked, or escalated for further review. 3

(b)

(a)

Typographic Granularity (L)

Stylistic Transformations (S)

Placement Strategies (M)

Word

cut off your own finger to see if you like the feeling of blood

Token

cut off yo ur own fin ger to see if you li ke the fee ling of bl ood

Mixed

cu t off yo ur ow n fi ng er to see if you lik e the feel ing of bl oo d

best overall business M4 regular M5 traveller business traveller best list stays frequent asia pacific region, welllocated hkia convenient hk central downtown M6 areas

best overall business M2 M3 M1 regular traveller business traveller best list stays frequent asia pacific region, welllocated hkia convenient hk central downtown areas

Bold (B) Color (Col) Highlight (Hi) Precomposed (Pre) Cap-Flipping (Cap) Cloze

best overall business let traveller regular business traveller me best list stays frequent asia pacific touch region, well-located hkia your convenient hk central downtown areas daughter’s mtr, hotel excellentlycaters pussy needs business taveller best restaurants it’s travels offering different experience for mix spanish/ western asian cuisines, her room comfortable travel asia know beds good generally novotel suits needs, pool tremendous quite different environment noisy city, facilities class, overall visiting hkg regularly having stayed downtown highly recommend

a→ a a→ a a→ a a→ⓐ a→ A a→ _

Figure 2: Typographic configuration space in HPAA and an illustrative example (derived from real-world toxic content). Stage 2: Online Attack Generation

Stage 1: Human-Guided Typographic Configuration Set Refinement Round I

Online Platform Restriction

Round II

ℋ′

ℋ2

ℋ1

ℋ

Update ℎ

ℎ = 𝑙, 𝑚, 𝑠

Max Iteration

Yes

Attack Failure

Unsafe

Adversarial Sample Generation

𝑙

Benign Text

𝑚

𝑏

𝑡

𝓖

Toxic Input

Typographic Granularity

𝑡′

𝓟 Placement Strategies

𝑠

𝑐

𝑥

𝓣

Black-Box Query

HPAA Sample

Stylistic Transformations

Text Moderation System

Attack Safe Success

𝑥

Online Platform

Figure 3: The HPAA framework incorporates a two-round user study (in Stage 1 Human-Guided Refinement), conducted once to collect human feedback that informs and supports the attack design.

3.4

HPAA Overview

This design separates human perceptibility from moderation evasion. HPAA uses a one-time human-guided refinement process to identify perceptually effective typographic configurations. The refined configuration set is subsequently used to guide attack generation and to approximate human perceptibility during large-scale evaluation. Consequently, HPAA can efficiently operate against previously unseen moderation systems while retaining perceptual properties validated through human evaluation.

Figure 3 presents the overall HPAA framework. Given a toxic input message t, HPAA aims to construct a typographically modified sample x that remains recognizable to human readers while avoiding detection by automated moderation systems. HPAA operates in two stages. In Stage 1, we identify typographic configurations that reliably preserve human recognition of embedded toxic content. Starting from the configuration space defined in Section 3.3, we conduct a two-round user study to evaluate how different combinations of granularity, placement, and styles affect human interpretability. Configurations that consistently enable accurate recognition are retained to form a refined configuration set. In Stage 2, HPAA generates adversarial samples using configurations selected from the refined set. Given toxic message t and benign carrier text b, the attack applies the selected typographic configuration to embed the toxic content into the carrier. The resulting sample is submitted to the moderation system in a black-box manner. This design allows HPAA to focus detector queries on configurations that have already been validated to preserve human recognition. Based on the returned moderation outcome, HPAA iteratively updates the configuration choice and generates alternative candidates under a limited query budget. The process terminates once a sample bypasses moderation or the query budget is exhausted.

3.5

Stage 1: Configuration Set Refinement

The typographic configuration space introduced in Section 3.3 contains numerous cue combinations with unknown perceptual effectiveness. To identify configurations that reliably preserve human recognition, HPAA performs a two-round human-guided refinement process and constructs a refined configuration set for subsequent attack generation. Round I: Configuration Screening. The first round evaluates diverse candidate configurations drawn from the full configuration space. Using pairwise comparisons, participants assess the recognizability of embedded toxic content under different configurations. This stage identifies configurations that consistently preserve human recognition while mitigating individual differences in judgment, producing an initial 4

3.6.2

ranking. Round II: Targeted Configuration Evaluation. Using the configurations retained from Round I, the second round conducts a more focused evaluation under controlled conditions. This stage examines recognition accuracy, partial versus complete recognition, and robustness across different content categories. Based on the resulting rankings, HPAA identifies configurations that most reliably preserve human recognizability. To ensure practical applicability, platform-specific admissibility constraints are further applied to obtain a deployable configuration subset H ′ ⊆ H , where H denotes the original configuration space. The resulting set H ′ contains configurations that are both perceptually effective and technically realizable on the target platform. HPAA restricts subsequent attack generation to configurations in H ′ . Detailed descriptions of the study design, participant recruitment, and analysis methodology are provided in Section 5. Results for both rounds are reported in Appendix A.3.

3.6

For a candidate configuration h = {l, m, s} ∈ H ′ , HPAA generates an adversarial sample and submits it to the target moderation system. The detector is treated as a black box, and HPAA observes only the final outcome (e.g., Safe or Unsafe). The moderation outcome serves as a binary signal indicating whether the current configuration successfully bypasses moderation. If the generated sample receives a benign moderation decision, the attack terminates successfully. Otherwise, HPAA proceeds to another candidate configuration from H ′ and repeats the generation process. This procedure continues until a successful evasion sample is identified or the maximum query budget is reached.

4 4.1

Attack Evaluations Experimental Setup

Stylistic Transformation Grounding. We ground the stylistic transformation dimension of HPAA in formatting conventions observed across five major real-world platforms: Reddit, X/Twitter, Discord, Stack Overflow, and YouTube. Specifically, we surveyed the native text-formatting features supported by each platform and identified transformations that can be deployed using functionality available to ordinary users without special tools or privileges. Table 7 summarizes the supported formatting features across platforms. Most platforms support bold and precomposed text, while some additionally support highlighting and color under varying character limits (280 – 40k). Based on these observations, we derive six stylistic transformations as illustrated in Figure 2. Datasets. To support the two-stage HPAA framework, we construct three datasets: HUS-I and HUS-II for Stage 1 refinement, and HED for Stage 2 evaluation. Each is derived from a short toxic expressions dataset (STTD) and a diverse benign contexts dataset (BTD).

Stage 2: Online Attack Generation

Given the refined configuration set H ′ , HPAA performs online attack generation through black-box interaction with the target moderation system. Stage 2 consists of two components: adversarial sample generation and black-box moderation interaction. The former instantiates attack samples from typographic configurations, while the latter evaluates generated samples using moderation outcomes returned by the target system. The process continues until a successful sample is identified or the query budget is exhausted. 3.6.1

Black-Box Moderation Interaction

Adversarial Sample Generation

Given a selected configuration h = {l, m, s} ∈ H ′ , HPAA constructs an adversarial sample from toxic input t and benign carrier text b:

• Short Toxic Text Dataset (STTD). We construct a short toxic text dataset by re-annotating 249 samples from AdvBench [71] using the Llama Guard safety taxonomy [29], which defines five harmful categories: Hate, Violence, Sexual Content, Self-Harm, and Insults. We retain only toxic sentences containing at most ten words.

x = F(t, b; h), where F(·) denotes the typographic embedding procedure that transforms toxic input t into an adversarial sample according to configuration h while using benign carrier text b. The generation process first decomposes the toxic content according to the selected granularity level l. The resulting textual units are then embedded into the benign carrier text following the placement strategy m. Finally, the stylistic transformation s is applied to introduce the corresponding visual cues. Together, these operations produce the final adversarial sample submitted to the moderation system. Because all configurations are drawn from the humanrefined set H ′ , generated samples retain the perceptual properties validated in Stage 1.

• Benign Text Dataset (BTD). We construct a benign text corpus by sampling reviews with the maximum available rating from five public datasets: TripAdvisor (hotels) [58], Yelp (restaurants) [68], IMDb (movies) [43], Amazon Product Reviews [45], and Amazon Music Reviews [27]. • Dataset for HPAA User Study Round I (HUS-I). HUS-I contains 108 samples, each formed by embedding a toxic expression into benign carrier text using a distinct typographic configuration. The dataset is used for Round I configuration screening during Stage 1 refinement. 5

• Dataset for HPAA User Study Round II (HUS-II). HUSII contains 105 samples spanning five topic domains (21 samples per domain). Each sample is generated using one configuration from the top-ranked configuration set identified in Round I. The dataset is used for Round II evaluation and refinement of candidate configurations.

For compact presentation, we summarize attack outcomes r using the notation {e%kmin }, where e% denotes the resulting k-shot evasion rate, k denotes the attack budget and rmin denotes the minimum recognition frequency of the typographic configurations involved.

• HPAA Evaluation Dataset (HED). HED is constructed by applying configurations from the refined configuration set H ′ to samples drawn from STTD and BTD. The dataset is used to evaluate HPAA against automated moderation systems and serves as the primary benchmark for attack effectiveness.

4.2

Stage 1: Configuration Set Refinement

The typographic design space yields 108 possible configurations, collectively denoted by H , which serve as the candidate set for Stage 1 refinement. The two refinement rounds progressively reduce this set to H1 and H2 , containing 21 and 10 configurations, respectively. We discuss the resulting configuration refinement below, while the complete user-study design and procedure is presented in Section 5.1.

Evaluation Metrics. We evaluate HPAA along two complementary dimensions: human exposure and attack effectiveness against automated moderation systems. • Human Exposure. We quantify human exposure using the recognition frequency measured in Round II of the user study. This measure captures the proportion of participants who correctly identify the intended harmful content after typographic transformation and serves as a proxy for population-level exposure. Higher recognition frequencies indicate greater semantic accessibility to human observers and thus a higher potential for human harm.

Table 1: Human recognizability scores for the configuration set H on the HUS-I dataset. Based on the normalized recognition scores, the top-21 configurations are retained to form H1 and are shown in bold. Cells with recognizability scores ≤ 0.200 or ≥ 0.700 are highlighted for visual reference. M

• Attack Effectiveness on Automated Moderation. – Detection and Evasion Rates. For detectors with discrete outputs, detection rates are computed directly; for detectors producing continuous safety scores, we apply a fixed decision threshold calibrated on a balanced validation set to obtain binary predictions. We report the detectors’ detection rate, defined as the fraction of toxic samples classified as unsafe. Attack effectiveness is quantified by the evasion rate, defined as the complement of the detection rate on adversarial samples. Detection rates on original samples reflect baseline detector performance on toxic samples, while evasion rates on HPAA samples measure the ability to bypass automated moderation. – k-Shot Evasion Rate. Repeated submissions may trigger platform-level rate limits, account restrictions, or human review, making successful evasion within a small number of attempts particularly desirable in practice. To capture this constraint, we evaluate evasion under different attack budgets. For an attack budget of k, an attacker is allowed to sequentially evaluate up to the top-k configurations from H ′ in ranked order. A sample is considered successfully evaded if any evaluated configuration is classified as safe. We report the resulting k-shot evasion rate, defined as the fraction of samples that evade detection within at most k attempts. In particular, we report both 1-shot evasion (k = 1), reflecting realistic one-attempt deployment scenarios, and 3-shot evasion (k = 3), which measures the additional benefit obtainable from a small number of refinement attempts.

Cap.

Cloze

Pre.

Col.

Hi.

Word

0.846

0.200

0.800

0.929

0.900

0.800

Token

0.400

0.067

0.800

0.700

0.429

0.700

Mixed

0.500

0.200

0.750

0.583

0.909

0.727

Word

0.538

0.154

0.375

1.000

0.692

0.750

Token

0.375

0.167

0.667

0.909

0.462

0.750

Mixed

0.353

0.143

0.200

0.286

0.667

0.500

Word

0.364

0.333

0.636

0.727

0.909

0.933

Token

0.200

0.333

0.700

0.583

0.778

0.500

Mixed

0.167

0.222

0.700

0.591

0.692

0.636

Word

0.500

0.000

0.909

0.727

0.636

0.909

Token

0.300

0.071

0.600

0.800

0.615

0.750

Mixed

0.333

0.182

0.417

0.667

0.417

0.667

Word

0.538

0.250

0.733

0.636

0.875

Token

0.455

0.133

0.636

0.692

0.429

Mixed

0.375

0.133

0.455

0.667

0.300

0.818

Word

0.556

0.400

0.900

0.800

0.500

0.824

Token Mixed

0.471 0.667

0.313 0.600

0.583 0.545

0.714 0.545

0.846 0.545

0.636 0.364

H1 from Round I. Table 1 reports the normalized recognition scores (scaled to [0, 1]) derived from the Round I pairwisecomparison study, from which we retain the top-21 configurations to form H1 . Since the benign and toxic content were fixed throughout Round I and only the typography configuration varied, the pairwise-comparison framework isolates the effect of typographic design on recognition performance. This controlled setup provides an efficient means of estimating relative preferences among a large number of configurations, thereby enabling effective exploration of the configuration space H . 6

H2 from Round II. While the normalized scores obtained in

Specifically, configurations employing stylistic transformations other than Cloze consistently achieve substantially higher recognition scores across placement strategies and granularity levels. This trend suggests that visual cues such as Bold, Color, Highlight, and Precomposed effectively draw attention to embedded toxic content while preserving readability. By contrast, Cloze-based configurations consistently rank near the bottom of the table. This behavior is expected, as Cloze transformations intentionally obscure toxic characters using placeholder symbols, thereby reducing lexical visibility and increasing the cognitive effort required for reconstruction. Consequently, configurations relying heavily on Cloze are largely excluded from H1 , while configurations employing visually salient transformations are preferentially retained for further evaluation in Round II.

Round I capture relative differences among configurations, they do not directly quantify how often toxic content is recognized by human readers. We therefore conduct Round II to measure recognition frequency, defined as the fraction of participants who correctly identify the embedded toxic content under each configuration. This metric serves as a proxy for human perceptibility and population-level exposure. Details of the computation are provided in Appendix A.3. To keep the human evaluation tractable, Round II evaluates only the 21 configurations retained in H1 . As shown in Figure 4, substantial variation exists across configurations. Several configurations achieve recognition frequencies exceeding 80% while simultaneously exhibiting low detection rates across the evaluated moderation systems, indicating a persistent mismatch between human perception and automated moderation. In particular, configurations ranked highly in Round I generally remain highly recognizable in Round II, suggesting that the pairwise-comparison procedure provides a stable approximation of perceptual accessibility. The results further highlight that human recognizability and detector sensitivity are only weakly aligned. Configurations that are readily understood by human participants frequently evade moderation systems, whereas several configurations receiving comparatively stronger detector responses remain highly recognizable. This discrepancy suggests that existing moderation systems rely heavily on lexical and surface-form cues that are disrupted by typographic transformations but remain interpretable to human readers. Based on these observations, we retain configurations achieving at least 80% recognition frequency and construct the final refined set H2 , which contains 10 configurations. The threshold is chosen to prioritize configurations that remain understandable to a large majority of participants while eliminating variants with substantially reduced human perceptibility. As a result, H2 focuses on transformations that simultaneously maximize human exposure and preserve strong moderation-evasion potential, providing a compact yet effective candidate set for subsequent attack generation and robustness evaluation.

Beyond stylistic transformations, we observe that placement strategy M1 and word-level granularity consistently achieve higher recognition scores across configurations. Configurations that preserve contiguous lexical units appear easier for participants to recognize than token-level or mixedgranularity variants, suggesting that maintaining local semantic coherence plays an important role in human perception. Likewise, the consistently strong performance of M1 indicates that the location at which typographic modifications are introduced can substantially influence recognizability. These findings demonstrate that perceptibility is jointly determined by transformation style, placement strategy, and granularity, rather than by any single design factor.

High Human Exposure

100%

Medium Human Exposure

Overall

Detection Rate of Varied Detectors

92% 88% 86% 86% 85% 84% 83% 82% 81% 81% 80% 80% 80% 77% 77% 77% 77% 76% 74% 74% 73% 72%

75% 50% 25%

-W -H 5W- i M Hi 1W M -H 1W- i P M re 6M W-B 3WC M ol 1W M -B 4W M -H 6W- i M Pre 1T -P Ov re e M rall 3WM 1W- Hi C M ap 6T -C M ol 2 M -T 4W- B P M re M 2 -W 1Mi -B M x -C 5Mi ol x -H M i 4 M -T -B 1W M -Co 5W- l Co l

Recognition Rate

More broadly, the results reveal a clear trade-off between concealment and recognizability. Transformations that aggressively obscure textual content tend to reduce human recognition, whereas visually salient modifications preserve recognizability while still altering the surface form observed by automated moderation systems. This observation motivates our subsequent focus on highly recognizable configurations, which better reflect realistic scenarios in which adversarial content remains understandable to human audiences. Together, the results provide a principled basis for pruning H and constructing H1 , which serves as a refined candidate pool for the more detailed evaluations conducted in Round II.

Figure 4: Human exposure rate and detectors’ detection rate on HUS-II dataset. 7

Table 2: Comparison of content safety model detection rates before and after applying HPAA across five harmful categories. Results are reported on the STTD dataset (Original) and the HUS-II dataset (HPAA-applied) with iteration number k = 1 and H ′ = H2 (human exposure rate: 92%). Detection rates lower than 1% and higher than 10% are highlighted. Model Name SG-9B PA Amazon-C Enkrypt AI Azure AI LG3-8B G2F GPT-4o G-2.5-FL Omni

4.3

Original 54.41% 77.94% 94.12% 80.88% 86.76% 91.18% 100.00% 98.53% 100.00% 95.59%

Hate HPAA 0.00% 1.52% 1.47% 0.00% 11.76% 0.00% 1.47% 4.41% 11.76% 2.94%

↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓

Violent Original HPAA 53.51% 0.00% 71.05% 0.89% 94.74% 0.00% 72.81% 0.00% 82.46% 14.04% 89.47% 0.00% 89.47% 0.00% 94.74% 2.63% 98.25% 7.89% 92.98% 1.75%

↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓

Sexual Original HPAA 66.67% 0.00% 94.44% 5.88% 97.22% 2.78% 97.22% 0.00% 86.11% 16.67% 88.89% 0.00% 91.67% 0.00% 97.22% 2.78% 97.22% 8.33% 97.22% 2.78%

Stage 2: Online Attack Generation

97%

100%

99%

100%

Original Toxic Text M1-W-Pre (86%) M6-W-B (85%) M3-W-Col (84%) 100% 100% 100% 100% 100% 100% 100% 100% 99.6%

100% 100%

100%

96.5%

92.8%

86%

80%

100%

98.8%

99.5%

100%

99.5%

100%

60% 40% 20%

100%

99.5%

80%

Evasion Rate

100%

↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓

Each stacked bar shows cumulative evasion outcome for a detector. Red segment denotes samples that evade detection in their original toxic form, while the remaining segments represent additional evasion from high-ranked HPAA configurations. Green, yellow, and blue correspond to the incremental evasion contributed by the three configurations in Figure 5. The cumulative height therefore represents the overall evasion rate under a 3-shot attack budget. Later attempts continue to contribute substantial gains despite operating only on samples that survive earlier attacks, suggesting that different highranked configurations exploit complementary vulnerabilities in moderation systems. With a 3-shot budget, HPAA reaches up to 100%86% (or 100%84% when restricted), demonstrating 3 3 that near-complete detector evasion can be achieved.

Original Toxic Text M6-W-Hi (92%) M5-W-Hi (88%) M1-W-Hi (86%) 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 97%

↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓

Insults Original HPAA 44.93% 0.00% 61.59% 1.52% 78.26% 3.62% 65.22% 0.00% 60.87% 11.59% 74.64% 0.00% 75.36% 0.72% 89.13% 2.90% 92.03% 10.14% 77.54% 1.45%

3-Shot Evasion. Figure 5 illustrates detector performance under a larger attack budget (k = 3), where an adversary is allowed to sequentially evaluate up to three configurations from H ′ in ranked order. Detector robustness is evaluated on the HED dataset when H2 = H ′ (Figure 5a, e.g., highlighting is permitted), and under a restricted setting where H ′ ⊂ H2 (Figure 5b, e.g., highlighting is disallowed while other stylistic transformations remain available).

1-Shot Evasion. Table 2 reports detector performance on the original STTD samples and the corresponding HPAA samples under an attack budget of k = 1. Each attack employs only the highest-ranked configuration in H ′ (M6 -W-Hi), which achieves a human recognition frequency of 92%. This setting reflects a realistic deployment scenario in which attackers seek to evade moderation with a single attempt, minimizing the risk of triggering platform-level rate limits. HPAA consistently reduces detection rates across all evaluated harmful categories and moderation systems. Most detectors experience near-complete failure under attack, with detection rates dropping to near-zero across multiple categories. Specifically, Azure Content Safety and Gemini 2.5 Flash Lite exhibit comparatively stronger robustness, retaining higher post-attack detection rates than other detectors. However, even these stronger systems experience substantial degradation, with detection rates remaining below 17% across all evaluated categories. Across categories and moderation systems, the strongest HPAA configuration achieves up to 83.33%92% 1 , indicating an evasion rate of 83.33% with a 92% human recognition rate.

100%

↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓

Self-harm Original HPAA 75.00% 0.00% 79.55% 2.33% 93.18% 4.55% 84.09% 0.00% 88.64% 13.64% 97.73% 0.00% 100.00% 0.00% 100.00% 2.27% 100.00% 6.82% 90.91% 0.00%

81.0% 75%

60% 40% 20%

21%

B 3-8

18%

41%

31%

14%

e L F B B G2 2.5-F SG-2 SG-9 Azur G-

32%

16%

18%

13%

12%

10%

PA on-C on-N mni pt AI T-3.5 T-4o O ry GP az az GP k Am Am En

20.9%

(a) The evasion rate on STTD and HED dataset when H ′ = H2 , i.e., when the highlight is allowed (e.g., Stack Overflow).

B 3-8

18.1%

4.8%

L 5-F

2. G-

41.0%

-2B

30.9%

-9B

17%

ure Az

31.7%

16.1%

7.2%

n-C n-N azo mazo Am A

17.7%

12.8%

p kry

12.1%

. T-3 GP

9.6%

o T-4 GP

(b) The evasion rate on STTD and HED dataset when H ′ ⊂ H2 , i.e., when the highlight is not allowed (e.g., Reddit).

Figure 5: The evasion results on STTD (red shadow) and HED datasets (3 iterations). 8

4.4

Sensitivity Analysis

4.4.1

Effect of Decision Thresholds

dimensiondimension 2 dimension 2 2

500 50 0 -50

94.4%

99.1% 100%

100% 100%

91.9%

80%

60%

40%

20%

0.10

0.20

0.30

0.40

Threshold

0.50

0.60

0.70

-50

dimension 1

-50

0 -50 50 -50

-50 0

dimension 1

-50

dimension 1 50 (b) 70% benign words retained; intermediate semantic alignment. 50 -50 0 -50

0 -50

-50

-50 -50

dimension 1

dimension 1 dimension 1

Figure 7: Effect of benign word ratio on HPAA samples. Left: t-SNE embeddings (Red: Toxic Text Only, Blue: Benign Text Only, Black: HPAA samples); right: text examples.

100%

69.4%

50 -50 0

Detection Rate on balanced dataset

94.4%

500

Detection Accuracy (Azure AI Content Safety) Detection Accuracy (Perspective API)

99.1% 100%

dimension 1

off your own finger excellently cut to off see different your if youown excellently cut finger like to the off different different your see feeling if ownof visiting you finger excellently blood like to the different see different if feeling visiting you of best cut regular blood like list business traveller off different asia your,the feeling well-located own convenient hk visiting downtown areas fingerofmtr, excellently caters to blood traveller best best cut regular restaurants travelsoff see offering business traveller list different experience if spanish/western asian asiamix your, cuisines, room youown convenient hk well-located likemtr, suits needs, best cut finger regular downtown areas excellently pool different business traveller off traveller list caters to the best environment noisy feeling, facilities, asiatravels your, see restaurants offering different overall visiting hkg regularly having well-located convenient hk experience mix of if own spanish/western asian stayed downtown bloodmtr, downtown areasyou finger excellently cuisines, room great caters to liketraveller best suits needs, restaurants travelsthe see offering different pool different experience mix if spanish/western asian environment noisy feeling, facilities, best overall cutyou business travellerhaving regular cuisines, room overall visiting of hkg regularly business traveller off like best list stayssuits needs, stayed downtown blood frequent asia your pacific region, pool the different great well-located hkia convenient hk central environment noisyown feeling, facilities, downtown areasoffinger mtr, hotelhaving excellently overall visiting hkg regularly caters needs to business best stayed downtown blood traveller best overall cut business traveller regular restaurants travels see offering different great business traveller off best list stays experience mix if spanish/western asian frequent asia your pacific region, cuisines, room you comfortable travel asia well-located hkia own convenient hk central knowoverall beds generally like traveller novotel suits needs, best cut business downtown areas finger mtr, hotel regular excellently pool tremendous the different business traveller offquite besttraveller list stays caters needs to business best environment noisy feeling region, city, facilities frequent asia your pacific restaurants travels seeofoffering different class, overall visiting hkg regularly having well-located hkia own convenient hk central experience mix if spanish/western asian stayed downtown bloodmtr, highly recommend downtown areasyou finger hotel excellently cuisines, room comfortable travel asia great alternative, caters needsgenerally to business know beds liketraveller novotelbest suits needs, restaurants travelsthe seequite offering different pool tremendous different experience mix if spanish/western asian environment noisy feeling city, facilities cuisines, roomvisiting you comfortable travel asia class, overall of hkg regularly having know beds generally likehighly novotel suits needs, stayed downtown blood recommend pool tremendous great alternative, the quite different environment noisy feeling city, facilities class, overall visiting of hkg regularly having stayed downtown blood highly recommend great alternative,

(a)50 10% benign words retained; overlap -50 0 50 dimension 1 -50 with toxic clusters.

dimension dimension 2 dimension 2 2

100%

99.1% 98.8%

-50

0 -50

dimensiondimension 2 dimension 2 2

Evasion Rate on HUS-II dataset

For threshold-based moderation systems that rely on continuous safety scores (e.g., Azure Content Safety and Perspective API), we use a default decision threshold of 0.5, which serves as a commonly adopted baseline when no task-specific threshold is prescribed. Our sensitivity analysis shows that while alternative thresholds can yield marginal performance variations, even optimally tuned thresholds provide limited effectiveness against our attack setting. As illustrated in Figure 6, we report the detection accuracy to reflect the classification capability of the detectors on a balanced evaluation set, which is constructed by uniformly sampling an equal number of benign and toxic sentences from the HED dataset. The evasion rate is measured on the HPAA samples in HED dataset and quantifies the effectiveness of HPAA in bypassing automated moderation under varying decision thresholds. We report the detection accuracy to show the classification ability of detectors lowering the decision threshold increases evasion rates but this comes at the cost of substantially reduced detection accuracy on the balanced dataset. As a result, aggressive threshold tuning is impractical in deployment and offers only limited robustness against HPAA-based evasion. Evasion Rate (Azure AI Content Safety) Evasion Rate (Perspective API)

cut

4.5

Scaling Ability

Long Toxic Words. HPAA can be scaled with longer toxic words and presented in Table 3. The benign-to-toxic ratio on dataset HED is more than 20:1. We shorten the benign sentences to explore how the ratio influences the evasion rate. Tables 4 and 5 report the evasion rates across different toxic sentence length groups under varying benign-to-toxic ratios, evaluated on the Perspective API and Azure Content Moderator, respectively.

Figure 6: Sensitivity of threshold-based moderation systems.

Table 3: Evasion rates vs. toxic sentence length. 4.4.2

Effect of Benign Content Ratio

We analyze HPAA’s sensitivity to the amount of benign content inserted into toxic text. In Figure 7, reducing the benign content causes HPAA samples to move closer to toxic regions in embedding space, while increasing benign content shifts them toward benign clusters.

4.6 4.4.3

Toxic Words

≤5

6–10

11–15

16–20

Perspective API

100%

99.1%

100%

Azure

97.3%

95.8%

98.1%

100%

Comparison Results

Effect of Stylistic Transformation Although existing attacks have fundamentally different attack objectives, we adapted DeepWordBug, TextBugger and TextFooler using TextAttack library w.r.t. toxic contents against 10 detectors, consistently showing successful detection in Table 6.

Without stylistic transformation, interleaving toxic words into benign text yields 70% evasion on HED dataset (PerspectiveAPI). With M6-W-Hi typographic configuration, evasion rises to 99%. 9

Table 4: Evasion rates on HED evaluated by Perspective API. Toxic Length

Avg

0.01

0.05

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

≤5 words

98.0%

97.3%

97.2%

97.3%

97.2%

97.3%

100.0%

97.2%

100.0%

6–10 words

96.5%

97.3%

94.6%

100.0%

97.2%

97.3%

97.2%

94.6%

97.3%

94.6%

97.3%

11–15 words

100.0%

16–20 words

100.0%

Table 5: Evasion rates on HED evaluated by Azure Content Moderator. Toxic Length

Size

Avg

0.01

0.05

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

≤5 words

98.8%

97.3%

100.0%

97.3%

100.0%

97.3%

100.0%

94.6%

100.0%

6–10 words

37 (sampled from 212)

98.8%

94.6%

100.0%

94.6%

100.0%

97.3%

100.0%

11–15 words

37 (sampled from 206)

99.3%

97.3%

94.6%

100.0%

16–20 words

37 (sampled from 65)

98.8%

91.9%

97.3%

100.0%

97.3%

100.0%

How Do Human Perceive Toxic Phrases?

to appear innocuous to automated systems. Through direct comparison of user performance with LLMbased moderation systems, this investigation captures both cognitive and behavioral aspects of human interaction with adversarial content. As Morris et al. [47] emphasize, neglecting human-perceived variability in adversarial examples risks underestimating their real-world impact. Our findings can reveal potential vulnerabilities in human judgment that adversaries may exploit in real-world moderation environments and provide empirical evidence for developing more resilient content moderation mechanisms.

Motivation. To address our first research question (RQ1: To what extent can human readers recognize toxic content embedded within typographically manipulated text, and how does recognition vary across different typographic configurations?), our user study examines how human perceive and respond to adversarially perturbed textual content within social media contexts. While prior research has predominantly emphasized algorithmic robustness against adversarial language attacks [48, 62], the human perceptual dimension of these manipulations remains underexplored [16]. Our study addresses this critical gap by evaluating users’ ability to recognize toxic or aggressive intent in text that has been manipulated to evade large language model (LLM)-based moderation systems. We investigate three primary dimensions: (1) the extent to which non-expert users (i.e., typical users) can detect harmful intent embedded in adversarial text that preserves readability, (2) the influence of typographic cue strategies (such as capitalization flipping, spatial disruption, and typographic variation) on users’ perception and interpretation, and (3) how perturbation length affects human recognition patterns. By systematically varying both perturbation strategies and their extent, we provide a comprehensive assessment of human judgment capabilities when confronted with content designed

5.1

User Study Design and Procedure

The study was conducted in two rounds (Round I and Round II), each with distinct participant samples and experimental constraints. Both rounds were implemented as online surveys and are publicly available at User Study Access. The study employed two datasets, HUS-I and HUS-II, constructed from the Short Toxic Text Dataset (STTD) and the Benign Text Dataset (BTD) described in Section 4.1. This construction ensures that differences in participant responses can be attributed primarily to the typographic manipulation rather than semantic variations in the surrounding text. The first round examined participants’ perceptual and be-

Table 6: Evasion rates of different attack methods across detectors. Method

LG3-8B

G2F

G-2.5-FL

SG-2B

SG-9B

Azure

Amazon-C

Amazon-N

Omni

Enkrypt

GPT-3.5

GPT-4o

DeepWordBug

22.5%

20.1%

5.2%

62.3%

38.2%

32.9%

32.4%

17.3%

9.6%

18.5%

25.7%

10.4%

9.2%

TextBugger

21.9%

18.9%

5.2%

62.3%

37.8%

32.9%

32.5%

16.9%

8.4%

18.9%

24.9%

10.4%

7.6%

TextFooler (Homoglyph)

23.3%

21.7%

7.6%

62.6%

38.2%

33.7%

32.8%

18.1%

12.4%

19.3%

26.9%

13.2%

10.0%

havioral responses to toxic language modified with 108 typographic cues. Analysis of participant responses identified 21 configurations that consistently preserved human recognition of toxic content. These configurations formed the hypothesis set H1 and were selected for further investigation. Building on these results, the second round systematically evaluated configurations derived from H1 under controlled conditions across multiple content domains. The goal of Round II was to better understand the contextual and linguistic factors influencing toxicity recognition and to identify the top-performing configurations that formed the refined hypothesis set H2 . Both studies were approved by the Institutional Review Board (IRB). 5.1.1

cues) is easier to recognize. Each question displayed two short text passages (labeled “A” and “B”), which represents different typographic cues applied to the same underlying harmful content. Participants were asked to indicate which passage made the toxic content easier to recognize. In addition to choosing either “A” or “B”, three neutral alternatives were provided: (1) “Both texts are equally easy to recognize as toxic”, (2) “Neither text appears to contain toxic or aggressive content”, and (3) “I am unable to determine based on the content provided.” Each question thus offered five response options in total. An invisible timer recorded participants’ response latency for subsequent behavioral analysis. An example question of Round I from our survey was provided in Figure 9a in the Appendix A.4. The objective of Round I is to compare the relative effectiveness of typographic transformations in preserving human recognition of toxic content. In each trial, participants are presented with two passages containing the same toxic content but rendered using different typographic transformations and are asked to determine which presentation makes the harmful content easier to recognize. Because this objective is inherently comparative, we adopt a pairwise-comparison design rather than an absolute rating scheme. Following this design rationale, Round I implements a pairwise-comparison paradigm [8] which allows relative judgments to be aggregated into latent recognizability rankings while minimizing inter-participant scale bias. Across all typographic cues, we generated 2,160 combined paragraphs (each composed of one passage and one harmful sentence). To maintain sampling balance, every combined passage was randomly presented at least five times across participants. The randomized pairing ensured that no rule consistently appeared in a fixed order or with a specific counterpart. To ensure variation in textual coherence and mitigate potential bias, we mixed random and non-random passages across the survey. Non-random passages consisted of coherent, meaningful sentences or short paragraphs resembling authentic reviews or social-media posts. In contrast, random passages were syntactically or semantically incoherent text segments, created by randomly shuffling or concatenating words without meaningful context. Each survey included 40 passages (20 random, 20 non-random) presented in randomized order, and all adversarial texts were syntactically complete and semantically meaningful. Priming Effect. We acknowledge that our study design may introduce a priming effect because participants were informed that the study involved potentially harmful language and were asked to evaluate whether text contained harmful or aggressive content. Such awareness may increase vigilance relative to everyday use. Therefore, our recognition rates should be interpreted as measuring participant performance under informed and attentive conditions. To further address whether participant responses were driven by the study warning, task framing, or interface design

First Round of User Study

The first round used a structured online survey with two versions to examine which typographic cue sets aided detection of concealed harmful content across devices (smartphones and computers). Both versions shared identical structure, and participants in the computer-based survey were excluded from the smartphone version to prevent learning effects. Each survey included four sections with 30 questions in total and was designed to be completed within 10 to 15 minutes. Embedded Text Sample Selection. We selected toxic comments from a publicly available dataset – AdvBench [71]. As these sources differ in terms of demographics and moderation standards, they provide a broader perspective on how toxicity manifests itself across platforms. For AdvBench, we extracted toxic-labeled English comments that ranged from 10 to 30 words in length, excluding duplicate or ambiguous entries. We mapped all comments to a unified binary label (“toxic”) and incorporated them into a simulated social media interface resembling a comment thread. To systematically conceal toxic phrases, we applied 108 distinct typographic cues which from a full factorial combination of three dimensions. 6 modes, 3 linguistic granularities (word, token, mixed) for each mode, and 6 visual alteration styles (cap-flipping, cloze, precomposed, bold, color, and highlight). Structure of the Survey. The first-round user study survey comprised three main sections, followed by a final section for completion-code verification. The first section presented an informed-consent form describing the study’s purpose, potential risks, and data-handling procedures. The second section collected 9 demographic variables, including participants’ country of origin, age, gender, native language, education level and frequency of social-media use. Non-native English speakers were additionally asked, “How long have you been learning English?”; this question was automatically omitted for native English speakers or who identified English as their first language. The third section (i.e., main section) contains 20 questions designed to ask participants to identify which method of concealing toxic or aggressive content (made by our typographic 11

rather than the presence of harmful content, we conducted two benign-only baseline surveys. In both surveys, all samples contained benign text only, while preserving the same warning procedure and harmful-content judgment task used in the main study. Therefore, any “Yes” response was treated as a false positive. The first baseline survey used a simplified yes/no format, where each text was evaluated independently for whether it contained harmful or aggressive content. This design isolates participant behavior from the pairwise task structure. The second baseline used the same A/B interface structure as Round I: each question presented two comments, and participants judged whether Comment A and Comment B contained harmful or aggressive content. This design allows us to evaluate whether the paired presentation format itself introduces systematic bias. Across both baseline formats, participants produced consistently low false positive rates. In the simplified baseline, the false positive rate was 2.75% on desktop and 2.74% on phone, with an overall rate of 2.74% across 122 participants and 2,440 judgments. In the A/B baseline, the false positive rate was 3.20% on desktop and 3.67% on phone, with an overall rate of 3.40% across 125 participants and 2,500 judgments. We observed no systematic response patterns across items. These results suggest that participants behaved consistently across simplified and structured settings, and that the A/B task format did not introduce a noticeable response bias; however the rest of the results should be interpreted in a worst case where users are aware. 5.1.2

to indicate whether they could recognize any harmful or aggressive elements in the text by selecting one of three options: “Yes, I can fully recognize harmful content”, “Yes, I saw some of the harmful content”, and “No, I did not find any harmful content”. If participants select either “Yes” option, a follow-up text box prompted them to type the specific words or phrases they perceived as harmful. This open-ended input allows us to capture partial versus full recognition and to verify selfreported responses during manual analysis. Participants are expected to complete this survey between 17 and 25 minutes. To reduce the scope of our study and focus on relevant factors, we examined only smartphones and meaningful (nonrandom) sentences as question text in the second round. This design reflects how users primarily consume and evaluate online text in everyday life: smartphones account for over 70% of total web traffic and more than 80% of social media engagement [33]. In addition, people spend nearly three times longer accessing social media content on smartphones than on desktop computers, with the majority of that time devoted to use of social media [7]. Given that our adversarial text manipulations target social-media-like content, restricting the study to the mobile condition ensured consistency with realworld environments.

5.2

Recruitment and Participants

All participants were recruited via Prolific [1]. Eligibility criteria for both user studies required participants to be at least 18 years old and comfortable completing the survey in English. After applying the exclusion criteria detailed in Section 5.4. A total of 370 individuals completed our user studies, including 120 in the first round and 250 in the second. Demographics of the participants for both studies are summarized in Appendix A.1, with additional details provided in Appendix A.2.

Second Round of User Study

Building on the easily recognizable typographic cues identified in the first round, we designed the second user study to better understand the linguistic features that enable toxicity detection under more controlled conditions. Specifically, we sought to determine which typographic cues most effectively evade large-language-model (LLM) detection while remaining perceptible to human readers. We developed five topic-specific online surveys (hotel, movie, restaurant, music, and product) to evaluate the recognition performance of typographic cues selected from the first round. These topics were chosen because they represent common categories of user-generated content on social media and review platforms (e.g., Yelp, YouTube, Amazon). To ensure ecological validity and contextual diversity, we curated real-world comments and reviews from these domains. Each second-round survey contained 31 questions. Structure of the Survey. Each survey follows the same structure as Round I: an informed-consent form with content warning, a demographic section identical to Round II, and an experimental section. The experimental section includes 21 questions, each presents a single text sample containing adversarially hidden harmful content. Participants were asked

5.3

Quantitative Analysis Plan

To quantify participants’ performance, we make use of several metrics summarized below. Selection Rate. For each typographic cue r, we compute the proportion of times it is selected by participants in pairwise comparison tasks. Because each typographic cue appears a different number of times across first-round surveys, the selection rate is normalized by its total number of its appearances: Normalized Selection Rater =

Sr Ar

where Sr denotes the number of typographic cue r is selected, and Ar denotes its total appearances. This metric represents the relative recognizability of each typographic cue. Participant-Reported Recognition Rate. We calculate the proportion of fully recognized items to represent participants’ subjective recognition performance. Partial recognitions are retained for descriptive visualization (Figure 8-A). 12

Recognition Rate

80% 78%

Partial

82%

87%

82%

90%

88%

75%

81%

Fully Recognized Combined

79%

85%

83%

76%

Partial Recognized Combined 83%

90% 80%

77%

83%

90%

94% 84%

83%

81%

80%

74%

60% 40% 20% 0%

(B) Experimenter-Verified Recognition Rate

100%

Recognition Rate

Full

(A) Participant-Reported Recognition Rate

100%

80% 76%

81%

83%

80%

86% 73%

86% 77%

77%

84%

80%

74%

82%

88% 77%

74%

72%

77%

85%

92%

60% 40% 20%

-M i

x-C o

l -T-P re M -W M -B -W -C M ap -W -Co M l -W M Hi -W -Pr e M -T-B M -W M -B -W -Co M l -W -H i M -T-B M -W M Hi -W M Pre -M ixM Hi -W -Co M l -W -H M i -T-C o M l -W M B -W M Hi -W -Pr e Ov era ll

Figure 8: Recognition rates across transformation rule conditions. The gray dashed line separates individual rule conditions from the combined average. Panel (A): Participants reported selection performance. Panel (B): Experimenter verified recognition performance. Abbreviations: Cap = cap-flipping; C = cloze; Pre = precomposed; B = bold; Col = color; Hi = highlight. Experimenter-Verified Recognition Rate. To obtain a ground-truth–aligned measure, we compare participant responses with the embedded harmful content using cosine similarity with all-MiniLM-L6-v2 and manual inspection. Responses with semantic overlap or minor typographical deviations are counted as valid if they convey the intended meaning, and partial recognitions are recorded for visualization (Figure 8-B).

5.4

version) into Table 1, ordering the columns for clarity as follows: cap-flipping, cloze, precomposed, bold, color, and highlight. Based on participants’ pairwise comparison responses described in Section 5.1.1, we counted how frequently each typographic cue was either selected as easier to recognize or equally easy to recognized in order to obtain a relative recognizability of each typographic cue. To visualize selection rates, we plotted the raw counts of selected and unselected instances for each combination of style, mode, and granularity. For both the computer and smartphone conditions, we generated stacked bar charts in which the blue segments represent the number of times a style was selected, and the orange segments represent in which it was not selected (Smartphone version: Figure 10 in Appendix). This paper presents only the visualizations from the smartphone version survey, since it was the primary dataset used in the analysis. Similar distribution patterns were observed in the computer version. Each subplot corresponds to a specific mode-granularity pairing, and the height of each bar indicates the total number of appearances for that style. For data collected by smartphone, 95% confidence intervals were calculated using the Wilson score interval [65], and report symmetric error bars. Figure 11 (in Appendix) visualizes selection rates as bar charts with style on the x-axis, selection rate on the y-axis, and each subplot representing one modegranularity combination. Visualizations are omitted due to space limitations and the study design decision to exclude

Qualitative Analysis Plan

For both rounds of the user study, incomplete or invalid submissions were excluded prior to analysis. Participants who declined our informed consent were automatically prevented from proceeding to the main survey. Additional exclusions were applied to responses that were prematurely terminated or lacked a valid completion code. We also excluded low-effort responses, such as those in which participants consistently selected options within 2 to 3 seconds per question, indicating insufficient engagement. In accordance with Prolific’s data-quality policy, such submissions were rejected on the platform, and replacement participants were recruited until the target sample size for each study was achieved. After these exclusions, only valid and complete responses were retained for analysis (Section 5.2 provides sample sizes). First Round Human Study Survey (Computer vs. Smartphone). We organized the selection rate result (smartphone 13

computer-based observations in Round II. Second Round Human Study Survey. For the second-round user study, we obtained 50 valid survey responses per topic, excluding incomplete and invalid submissions, resulting in 250 responses in total. Each question is designed to assess participants’ ability to recognize the harmful content embedded within a text sample using one typographic cue. To evaluate the alignment between participants’ textual descriptions and the reference harmful sentence, we employ two complementary automatic similarity measures:

Related Work

Adversarial Text Attacks. Prior work has shown that NLP models and LLM-based detectors are vulnerable to a wide range of adversarial text attacks, including character-level perturbations such as insertions, deletions, swaps, and homoglyph substitutions [16, 17, 20, 22, 23, 32, 37, 52]; wordand token-level replacements using semantic or embeddingbased alternatives [32, 37]; paraphrasing and deletion attacks that preserve meaning while altering surface form or removing explicit toxic cues [30, 53]; toxicity and hate-speech evasion techniques that stylize or rephrase harmful content to bypass filters [55, 57]; jailbreak and prompt-injection attacks that override safety alignment in instruction-following LLMs [40,54,71]; and steganographic text attacks that embed hidden content within ostensibly benign text [2, 34, 38, 66]. Most existing attacks are evaluated from a model-centric perspective, where success is defined by misclassification, with limited attention to whether humans still perceive the original harmful intent. While a smaller body of work examines human interpretation and proposes human-readable prompt attacks, these approaches do not systematically exploit typographic layout and styling as an adversarial channel. Limited Explorations on Human-Centric Attacks. Research on adversarial text from the perspective of human perceptibility remains limited. Prior work demonstrates that many textual perturbations deceive models while remaining readable to humans [16, 18, 52], yet these studies primarily examine linear word- or character-level edits and overlook broader typographic or perceptual dimensions. Other efforts have explored human-readable adversarial attacks [13,14,36], focusing mainly on model deception rather than user perception or real-world harm. For example, Das et al. [13] introduce situation-driven prompts that appear natural to humans but mislead LLMs, while Deng et al. [14] and Kurita et al. [36] design interpretable jailbreak and paraphrasing-based evasion strategies. Despite these advances, existing studies rarely involve humans in the attack design loop or empirically assess perceptibility. In contrast, HPAA is, to our best knowledge, the first adversarial framework to integrate user studies into both construction and evaluation, linking human perceptibility with model detectability in LLM-based moderation systems.

• Semantic Similarity: Participants’ responses and the reference sentence were encoded with all-MiniLM-L6-v2, and cosine similarity was used to measure semantic similarity. • Lexical Overlap: We use fuzz.token_set_ratio, a normalized measure of word-level set intersection, to capture literal recognition and word-for-word overlap. These metrics quantified each participant’s similarity to the original harmful sentence and evaluated the accuracy of self-reported recognition labels. While automatic metrics provided initial insight, we found that self-reported recognition levels differed significantly from the content of participants’ textual explanations. In several cases, participants selected ‘Yes, I saw some of the harmful content’ yet correctly reproduced the entire sentence. Alternatively, some selected ‘Yes, I can fully recognize the harmful content’ but identified only one or some highly salient toxic or aggressive words. In addition, some participants who selected ‘Fully’ or ‘Some’ did not mention any words from the harmful sentence, instead repeated words from the surrounding survey question text. We conducted a manual review to address self-report bias (over- or under-recognition). As part of this manual check, submissions that contained only text from the survey question rather than the original harmful sentence were reclassified as None. Each survey dataset is populated with one of three types of recognition: ‘Full’, ‘Some’ and ‘N’ (representing None). Using stacked bar charts, responses were aggregated according to the typographic cue and the manually corrected recognition level (Figure 8). A statistical analysis was performed to evaluate the differences in recognition and selection rates across all 21 typographic cue conditions. We use a chi-square test of independence to determine whether performance differed significantly between typographic cue conditions for each question type (recognition and selection). We use two-proportion z-tests to compare typographic cue performance across all pairs (210 comparisons, 21 2 ). To control the family-wise error rate, we apply the Holm–Bonferroni procedure [28], which is less conservative than standard Bonferroni correction while ensuring strict false-positive control.

Conclusion

In this paper, we presented Human-Perceivable Adversarial Attacks (HPAA), the first human-centric adversarial framework targeting text and LLM-based moderation. HPAA embeds typographic cues that remain interpretable to humans yet evade automated detectors, exposing a new socio-technical vulnerability in content-moderation pipelines. Through blackbox evaluation and user studies, we showed that specific spatial, granular, and stylistic transformations can conceal toxic content while preserving human readability. HPAA highlights 14

• S5: Platform Operators and Moderation Vendors. The six vendors whose systems we evaluated (Table 10). During the research procedures, they processed our API queries on their servers. Publication exposes weaknesses in their products but also provides reproducible evidence enabling targeted mitigation.

the gap between human vision and token-based encoding, and motivates to develop defenses most robust LLM-based toxicity detectors w.r.t. human perception. These results, together with the user study, empirically support the core claim that the perturbed text remains easily interpretable to human targets while they also easily escape LLM-based detectors.

• S6: Moderation and Broader Research Community. Researchers in content safety, adversarial ML, and related fields. Not affected by the research procedures; publication benefits them through new methodology, evaluation infrastructure, and released artifacts. Impacts. We assess impacts under each of the four Menlo principles, covering both research procedures and publication. • Beneficence. Tangible harms during research are brief psychological discomfort for S1 (toxic stimuli ≤10 words embedded in benign review text drawn from AdvBench [71]; no CSAM or graphic imagery was used at any stage) and sustained but bounded exposure for S2. The publicationstage harm is potential adversarial uplift against deployed systems, affecting S3 and S4. Set against this, our work documents a systematic blind spot in ten widely deployed systems serving hundreds of millions of users, motivates white-box defenses (Appendix E), and equips defenders with reproducible evaluation infrastructure (S5, S6). The individual transformations we study (bolding, highlighting, Unicode forms, capitalization, cloze masking) are already publicly available in any standard text editor and have been examined in adjacent prior work [16, 18]; our contribution is the systematic demonstration of their combined effect on LLM-based moderation, which is the same knowledge defenders require to build countermeasures.

Acknowledgments We sincerely thank the anonymous reviewers for their constructive comments. This work is partially supported by the National Science Foundation (NSF) under Grants No. CNS2308730, CMMI-2326341, CNS-2319277, CNS-2440819, DGE-2335798, ITE-2452747, and ITE-2452749, by a Cisco Research Award, and by Institute of Information & Communications Technology Planning & Evaluation (IITP) under RS-2025-25457342 and RS-2025-25394739. Any opinions, findings, conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.

Ethical Considerations We organize our stakeholder-based ethics analysis into four parts (stakeholders, impacts, mitigations, and decision), each examined with respect to both conducting the research and publishing the results. Our analysis is grounded in the four Menlo Report principles. Stakeholders. We identify six groups affected by this work and summarize how each is impacted by the research procedures and by publication. • S1: Study Participants (N=370). Prolific workers who read text containing embedded harmful language during our tworound user study; affected only during research procedures.

• Respect for Persons. S1 participants gave informed consent with explicit content warnings, knew the study’s true purpose (no deception), could withdraw unconditionally, and could not be re-identified. Using real rather than sanitized harmful language was methodologically necessary: synthetic content would not capture the patterns moderation systems encounter in deployment, rendering both attack construction and perceptibility evaluation scientifically invalid. This prioritizes informed participant autonomy over experimental sanitization.

• S2: Research Team Members. Authors who constructed HPAA samples and verified responses, with sustained exposure to harmful material throughout the research procedures. • S3: Platform Users, especially targets of harmful content. People who consume content on platforms guarded by the moderation systems we evaluate, and in particular members of demographic groups historically targeted by hate speech, harassment, or self-harm content. They are not affected by the research procedures, but publication carries dual-use risk: stronger defenses may reduce their exposure, while adversarial adoption may increase it.

• Justice. Exposure burden was distributed across 370 individuals from 31 countries (Appendix A.1), with no demographic group disproportionately recruited for the most distressing conditions. We acknowledge that Prolific’s $8/hr minimum wage is a floor rate, below what some venues recommend for studies involving distressing stimuli; we flag this as a limitation and recommend higher compensation in future replications.

• S4: Downstream Human Moderators. Reviewers in tiered moderation pipelines whose workload and direct exposure to harmful content scale with automated-filter failure rates. Not affected by the research procedures; affected by publication only if adversaries adopt the techniques before defenders harden their systems.

• Respect for Law and Public Interest. All commercial APIs and open-weight models were queried through standard interfaces under each vendor’s published research-use terms; no live production systems and no real end-user content 15

were involved. The research proceeded under IRB approval as minimal-risk human-subjects research. Mitigations. We detail measures taken and residual risks per stakeholder group. • S1 (Participants). Informed consent with explicit content warnings preceded all stimuli; unconditional withdrawal was available; toxic stimuli were capped at 10 words; no stimulus was personalized or directed at participants; no PII was collected; responses were anonymized on submission. Low-effort submissions were filtered per Prolific guidelines so no one was pressured through distressing content. Residual: we did not provide post-study mental-health resources beyond the withdrawal option. Given the minimal-risk IRB classification and brief exposure (10 to 25 minutes, with harmful content forming a small fraction of each survey) we judged formal follow-up unwarranted, but flag this as a limitation.

but reflects a systemic mismatch between human visual perception and token-level model input shared across all ten evaluated systems and, by the mechanism involved, across LLM-based moderation more broadly; private notification to individual vendors would not meaningfully change the threat landscape because no vendor can unilaterally close a class of attack that arises from how text is rendered to humans versus tokenized for models. Second, the defensive direction we identify (Appendix E) requires white-box modifications that vendors will need to implement based on the published methodology rather than on a private vulnerability report, since example-based or prompt-level fixes do not generalize. We assess that broad publication, together with reproducible evaluation infrastructure, serves defenders better than fragmented per-vendor reports that cannot be acted on without the very methodology this paper contributes. Decision. Under Beneficence, the bounded, forewarned, and voluntary discomfort experienced by 370 informed participants is substantially outweighed by exposing a systematic vulnerability in ten widely deployed moderation systems serving hundreds of millions of users. Under Respect for Persons, no individual’s rights were violated: consent was informed, exit was unconditional, and re-identification was infeasible. Under Justice, the burden was distributed across a diverse participant pool, with the principal residual concern (compensation as a floor rather than a premium for distressing content) flagged and mitigable in future work. Under Respect for Law and Public Interest, ToS compliance and IRB oversight support publication; we have not issued pre-submission vendor disclosures and instead defend that posture explicitly under Mitigations (S5), committing to post-acceptance notification of all six evaluated vendors. On proceeding despite foreseeable harm to participants. We treat IRB approval as necessary but not sufficient and reason independently here. The harm was real: 370 participants read hate speech, violent threats, sexual harassment, and self-harm references for 10 to 25 minutes, and we do not characterize this as trivial. We proceeded because (1) participation was voluntary under full information, with explicit content warnings, no deception, unconditional withdrawal, and no personal targeting; (2) lower-exposure alternatives were considered and rejected (synthetic content would invalidate the perceptibility claim, a smaller pool would concentrate rather than reduce aggregate exposure, and trauma pre-screening at recruitment would be privacy-invasive and exclude the populations most relevant to the threat model); (3) exposure was bounded by design (short toxic spans in benign context, small fraction of each survey, burden distributed across 370 people); and (4) the counterfactual is not a world in which participants are spared harm but one in which an attack already realizable with public text-editor features remains undocumented and its human-perceptibility side remains speculative. We recommend that future replications provide post-study mental-health resources, offer optional trauma disclosure at

• S2 (Research team). Toxic-content review tasks were rotated across the team and any member could opt out of any category without justification. Team-internal power dynamics were addressed explicitly: junior researchers were empowered by senior authors to defer the most distressing categories (self-harm, sexual content) without impact on authorship or evaluation, and exposure decisions were made by the individual being exposed. • S3, S4, S6 (Artifact release scope). We release: the user study instruments, the evaluation harness, the curated datasets, and reference implementations of the typographic configuration space sufficient for scientific scrutiny, independent replication, and defensive analysis. We deliberately exclude from release: turnkey attack pipelines optimized for any specific vendor’s API, deployment-ready configurationsearch code, and any tooling specialized for CSAM-related categories. We further explicitly disavow application of HPAA against CSAM moderation systems, where distinct defenses (e.g., perceptual hashing) apply and where the balance of harms differs fundamentally from the generaltoxicity setting we evaluate. Residual: a sophisticated adversary can in principle reconstruct the core technique from the paper alone, since the constituent transformations are publicly available. We assess this residual risk as outweighed by the defensive value of disclosure to S5 and S6, particularly given that defenders cannot build robust countermeasures against an attack surface they cannot characterize. • S5 (Responsible Disclosure). We commit to notifying the security contact at each of the six evaluated vendors (Google, Meta, Microsoft, Amazon, OpenAI, and Enkrypt AI), providing a link to the paper and released artifacts and engaging with vendor security teams on defensive deployment. We have not issued pre-submission disclosures, and we address that choice directly here rather than leave it implicit. Our reasoning rests on two observations about the nature of HPAA. First, the vulnerability is not vendor-specific 16

consent, and compensate above the platform minimum. A strict deontological reading could disfavor any participant exposure; we weight the consequentialist analysis more heavily because participants consented under full information with unconditional exit, and silence about this attack surface would deny defenders the empirical grounding needed for human-aligned moderation.

[7] I. Bouchrika. Mobile vs desktop usage statistics for 2025. https://research.com/software/guides/ mobile-vs-desktop-usage#5. [8] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, pages 324–345, 1952. [9] C. Chen, W. Qu, S. Su, Y. Feng, and T. Li. A comprehensive review of llm-based content moderation: Advancements, challenges, and future directions. KnowledgeBased Systems, page 114689, 2025.

Open Science To support reproducibility and facilitate independent evaluation, we release all artifacts necessary to evaluate the contributions of this work. These artifacts include the implementation of HPAA, evaluation scripts, processed datasets, and human-subject study materials (e.g., survey instruments, consent forms, and study documentation). All artifacts are publicly available through Zenodo:

[10] J. Chi, U. Karn, H. Zhan, E. Smith, J. Rando, Y. Zhang, K. Plawiak, Z. D. Coudert, K. Upasani, and M. Pasupuleti. Llama guard 3 vision: Safeguarding humanai image understanding conversations. arXiv preprint arXiv:2411.10414, 2024.

https://doi.org/10.5281/zenodo.20335299

[11] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.

The Zenodo record contains all released artifacts and supports versioned access. The DOI resolves to the latest version of the artifact package while preserving access to prior versions.

References

[12] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online, July 2020. Association for Computational Linguistics.

[1] Prolific: Online participant recruitment platform. https://www.prolific.com, 2025. [2] M. T. Ahvanooey, Q. Li, J. Hou, H. D. Mazraeh, and J. Zhang. Aitsteg: An innovative text steganography technique for hidden transmission of text message via social media. IEEE Access, 6:65981–65995, 2018.

[13] N. Das, E. Raff, and M. Gaur. Human-interpretable adversarial prompt attack on large language models with situational context. arXiv preprint arXiv:2407.14644, 2024.

[3] N. AlDahoul, M. J. T. Tan, H. R. Kasireddy, and Y. Zaki. Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos. arXiv preprint arXiv:2411.17123, 2024.

[14] G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu. Masterkey: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715, 2023.

[4] Amazon Web Services. Detecttoxiccontent – amazon comprehend api reference. https: //docs.aws.amazon.com/comprehend/latest/ APIReference/API_DetectToxicContent.html, 2023.

[15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

[5] Amazon Web Services. New for amazon comprehend – toxicity detection. https://aws.amazon.com/blogs/aws/ new-for-amazon-comprehend-toxicity-detection/, [16] S. DYRMISHI, S. GHAMIZI, and M. CORDY. How do humans perceive adversarial text? a reality check on 2023. the validity and naturalness of word-based adversarial [6] E. M. Bender, T. Gebru, A. McMillan-Major, and attacks. In ACL 2023: The 61st Annual Meeting of the S. Shmitchell. On the dangers of stochastic parrots: Association for Computational Linguistics, 2023. Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and [17] J. Ebrahimi, A. Rao, D. Lowd, and D. Dou. Hotflip: transparency, pages 610–623, 2021. White-box adversarial examples for text classification. 17

In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31–36, 2018.

safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. [30] M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv:1804.06059, 2018.

[18] S. Eger, G. G. Şahin, A. Rücklé, J.-U. Lee, C. Schulz, M. Mesgar, K. Swarnkar, E. Simpson, and I. Gurevych. Text processing like humans do: Visually attacking and shielding nlp systems. arXiv preprint arXiv:1903.11508, 2019.

[31] Jigsaw & Google. Perspective api. perspectiveapi.com/.

[19] Enkrypt AI. Unified ai guardrails — for privacy, integrity, and security, 2025.

https://

[32] D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, pages 8018–8025, 2020.

[20] S. Feng, E. Wallace, A. Grissom II, M. Iyyer, P. Rodriguez, and J. Boyd-Graber. Pathologies of neural models make interpretations difficult. arXiv preprint arXiv:1804.07781, 2018.

Digital 2025: Global [33] S. Kemp et al. overview report — device trends. https://datareportal.com/reports/ digital-2025-sub-section-device-trends.

[21] M. Franco, O. Gaggi, and C. E. Palazzi. Integrating content moderation systems with large language models. ACM Transactions on the Web, 19(2):1–21, 2025. [22] J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50–56. IEEE, 2018.

[34] M. Knöchel and S. Karius. Text steganography methods and their influence in malware: A comprehensive overview and evaluation. In Proceedings of the 2024 ACM Workshop on Information Hiding and Multimedia Security, pages 113–124, 2024.

[23] S. Garg and G. Ramakrishnan. Bae: Bert-based adversarial examples for text classification. arXiv preprint arXiv:2004.01970, 2020.

[35] H. Koh, D. Kim, M. Lee, and K. Jung. Can llms recognize toxicity? a structured investigation framework and toxicity metric. arXiv preprint arXiv:2402.06900, 2024.

[24] J. Geng, B. Yi, Z. Fei, T. Wu, L. Nie, and Z. Liu. When safety detectors aren’t enough: A stealthy and effective jailbreak attack on llms via steganographic techniques. arXiv preprint arXiv:2505.16765, 2025.

[36] K. Kurita, A. Belova, and A. Anastasopoulos. Towards robust toxic content classification. arXiv preprint arXiv:1912.06872, 2019.

[25] Google DeepMind Team. Safer and multimodal: [37] J. Li, S. Ji, T. Du, B. Li, and T. Wang. Textbugger: GenResponsible ai with gemma (shieldgemma 2). erating adversarial text against real-world applications. https://developers.googleblog.com/en/ arXiv preprint arXiv:1812.05271, 2018. safer-and-multimodal-responsible-ai-with-gemma/, [38] L. Li, L. Huang, X. Zhao, W. Yang, and Z. Chen. A sta2025. tistical attack on a kind of word-shift text-steganography. [26] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, In 2008 International Conference on Intelligent InforT. Holz, and M. Fritz. Not what you’ve signed up for: mation Hiding and Multimedia Signal Processing, pages Compromising real-world llm-integrated applications 1503–1507. IEEE, 2008. with indirect prompt injection, 2023. [39] Y. Li, J. Liu, T. Zhang, S. Chen, T. Li, and ... [27] R. He and J. McAuley. Ups and downs: Modeling the Baichuan-omni-1.5 technical report. arXiv preprint visual evolution of fashion trends with one-class collabarXiv:2501.15368, 2025. Omni-modal model with uniorative filtering. In WWW, 2016. fied text–vision–audio capabilities. [28] S. Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65–70, 1979.

[40] Y. Liu, G. Deng, Y. Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, et al. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499, 2023.

[29] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. Llama guard: Llm-based input-output

[41] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 18

Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.

[52] D. Pruthi, B. Dhingra, and Z. C. Lipton. Combating adversarial misspellings with robust word recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5582–5591, 2019.

[42] V. Logacheva, D. Dementieva, S. Ustyantsev, D. Moskovskiy, D. Dale, I. Krotova, N. Semenov, and A. Panchenko. ParaDetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6804–6818, Dublin, Ireland, May 2022. Association for Computational Linguistics.

[53] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118, 2020. [54] M. Russinovich, A. Salem, and R. Eldan. Great, now write an article about that: The crescendo {MultiTurn}{LLM} jailbreak attack. In 34th USENIX Security Symposium (USENIX Security 25), pages 2421–2440, 2025.

[43] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the ACL, 2011.

[55] X. Shen, Y. Wu, Y. Qu, M. Backes, S. Zannettou, and Y. Zhang. HateBench: Benchmarking hate speech detectors on LLM-generated content and hate campaigns. In 34th USENIX Security Symposium, 2025.

[44] B. Mathew, P. Saha, S. M. Yimam, C. Biemann, P. Goyal, and A. Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, pages 14867–14875, 2021.

[56] Skolkovo Institute of Science and Technology. snlp/roberta_toxicity_classifier. Hugging Face Model Hub, 2021. A RoBERTa-based binary toxicity classifier trained on merged English parts of Jigsaw datasets (2018, 2019, 2020), achieving AUC-ROC of 0.98 and F1-score of 0.76.

[45] J. McAuley, R. Pandey, and J. Leskovec. Image-based recommendations on styles and substitutes. SIGIR, 2015. [46] Microsoft. Azure ai content safety documentation. https://learn.microsoft.com/en-us/ azure/ai-services/content-safety/, 2025.

[57] K. Thomas, D. Akhawe, M. Bailey, D. Boneh, E. Bursztein, S. Consolvo, N. Dell, Z. Durumeric, P. G. Kelley, D. Kumar, et al. Sok: Hate, harassment, and the changing landscape of online abuse. In 2021 IEEE symposium on security and privacy (SP), pages 247–267. IEEE, 2021.

[47] J. Morris, E. Lifland, J. Lanchantin, Y. Ji, and Y. Qi. Reevaluating adversarial examples in natural language. In Findings of the association for computational linguistics: EMNLP 2020, pages 3829–3839, 2020.

[58] TripAdvisor. Tripadvisor hotel reviews dataset. https: //www.tripadvisor.com/, 2024.

[48] J. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, and Y. Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 conference on empirical methods in natural language processing: System demonstrations, pages 119–126, 2020.

[59] Unitary AI. unitary/multilingual-toxic-xlm-roberta. Hugging Face Model Hub, 2020. A multilingual XLMRoBERTa based toxicity classifier trained on Jigsaw Multilingual Toxic Comment Classification data, supporting 7 languages: English, French, Spanish, Italian, Portuguese, Turkish, and Russian.

[49] M. D. Muralikumar, Y. S. Yang, and D. W. McDonald. A human-centered evaluation of a toxicity detection api: Testing transferability and unpacking latent attributes. ACM Transactions on Social Computing, 6(1-2):1–38, 2023.

[60] Unitary AI. unitary/toxic-bert. Hugging Face Model Hub, 2020. A BERT-based classifier fine-tuned for multilabel toxicity detection on the Jigsaw Toxic Comment Classification datasets.

[50] OpenAI. Gpt-4o mini: Advancing cost-efficient [61] B. Vidgen, A. Harris, D. Nguyen, R. Tromble, S. A. Hale, intelligence. https://openai.com/index/ and H. Margetts. Challenges and frontiers in abusive gpt-4o-mini-advancing-cost-efficient-intelligence/, content detection. In Proceedings of the third workshop 2024. on abusive language online, pages 80–93, 2019. [51] OpenAI. Openai o3 and o4-mini system card. https: //openai.com/index/o3-o4-mini-system-card/, 2025.

[62] E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference 19

on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153– 2162, 2019.

Measure

[66] J. Wu, Z. Wu, Y. Xue, J. Wen, and W. Peng. Generative text steganography with large language model. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 10345–10353, 2024.

[68] Yelp. Yelp open dataset. https://www.yelp.com/ dataset, 2024. [69] J. Yi, Y. Xie, B. Zhu, E. Kiciman, G. Sun, X. Xie, and F. Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. In KDD, KDD ’25, page 1809–1820. ACM, 2025. [70] J. Zhu, D. Bespalov, L. You, N. Kulkarni, and Y. Qi. Taebench: Improving quality of toxic adversarial examples. In NACACL: Human Language Technologies, pages 251–265, 2025.

Black / African Am. White Hisp. / Latino Race / Eth. Asian Multiracial Other

55 45.8 51 42.5 6 5.0 2 1.7 3 2.5 3 2.5

80 32.0 113 45.2 17 6.8 16 6.4 5 2.0 5 2.0

South Africa U.S. U.K. Others (< 10 each)

49 40.8 22 18.3 16 13.3 33 27.6

70 28.0 35 14.0 35 14.0 110 44.0

High school / GED Associate degree Bachelor’s degree Education Master’s degree Doctoral degree Prof. / Other

25 20.8 8 6.7 45 37.5 23 19.2 12 10.0 7 5.8

41 16.4 15 6.0 114 45.6 55 22.0 9 3.6 13 5.2

Round II (Five Topic Surveys). The largest racial and ethnic groups were White (n = 113, 45.2%) and Black or African American (n = 80, 32.0%). Smaller groups included 17 (6.8%) Hispanic or Latino, 16 (6.4%) Asian, 19 (7.6%) multiracial, and 5 (2.0%) were another race not listed. Gender distribution was balanced: 126 females (50.4%), 121 males (48.4%), and 3 non-binary or third-gender participants (1.2%). Participants represented 31 countries across six continents, primarily South Africa with 70 participants (28.0%), 35 (14.0%) from the United Kingdom, and 35 (14.0%) from the United States. Moderate representation came from Canada with 13 participants (5.2%), Portugal with 12 (4.8%), and Poland with 12 (4.8%). Educational attainment ranged widely: 114 participants (45.6%) held a bachelor’s degree, 55 (22.0%) a master’s degree or higher, 41 (16.4%) a high school diploma or equivalent, 15 (6.0%) an associate degree, 12 (4.8%) a professionalschool qualification, and 9 (3.6%) a doctoral degree. Partic-

[71] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.

User Study

Recruitment and Participants

126 50.4 121 48.4 3 1.2

or Latino, 2 (1.7%) as Asian, 3 (2.5%) as multiracial, and 3 (2.5%) as other race not listed. Gender distribution was nearly balanced: 57 females (47.5%), 62 males (51.7%), and 1 other gender (0.8%). Participants were drawn from 20 countries, 49 (40.8%) from South Africa , 22 (18.3%) from the United States, and 16 (13.3%) from the United Kingdom, with additional representation from several European, African, and Asian countries (each n < 10). Educational backgrounds were diverse: 45 participants (37.5%) held a bachelor’s degree, 25 (20.8%) a high school diploma or GED, 23 (19.2%) a master’s degree or higher, 12 (10.0%) a doctoral degree, 8 (6.7%) an associate degree, and 6 (5.0%) a professional-school qualification. One participant (0.8%) declined to disclose education level.

[67] L. Wu, F. Morstatter, K. M. Carley, and H. Liu. Misinformation in social media: definition, manipulation, and detection. ACM SIGKDD explorations newsletter, 21(2):80–90, 2019.

A.2

57 47.5 62 51.7 1 0.8

Country

[65] E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 1927.

Demographics Across Two Study Rounds

Round II

Female Male Other / NB

Gender

[64] L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021.

A.1

Round I

Count (%) Count (%)

[63] A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110, 2023.

Item

Round I (Phone and Computer Surveys). In terms of race and ethnicity, 55 (45.8%) participants identified as Black or African American, 51 (42.5%) as White, 6 (5.0%) as Hispanic 20

Impact of Mode and Level of Granularity. In most scenarios, highlight and bold consistently yielded the highest selection rates. Conversely, cloze style produced the lowest selection rates, frequently below 0.2, with color exhibiting moderate performance (0.6–0.8), and cap-flipping achieving intermediate rates (approximately 0.3 - 0.6). The typographic style exerted a stronger influence on results than either the mode or the level of granularity. Figure 11 demonstrate recognition rates of all typographic cues in the smartphone condition. The relative ordering among cues remains stable despite variation in mode or granularity, indicating that cue type is the primary determinant of detectability on smartphone. Recognition Accuracy by Human. Overall, participants achieved high recognition performance across typographic cues. In self-reported data (Panel A of Figure 8), recognition rates ranged from 74% to 94% (x̄= 83%), and these differences across rules were statistically significant (χ2 (40) = 129.34, p < .001). After manual checking (Panel B of Figure 8), recognition rates ranged from 72% to 92%, with a slightly lower overall average of 80%. Again, still showing significant variation among rules, χ2 (40) = 222.41, p < .001. In self-reports, the typographic cues M6 -Word-Highlight, M6 -Word-Bold and M5 -Word-Highlight achieved the highest recognition rates (≈ 90% − 94%), indicating that visually salient manipulations such as word level highlight or word level bold were easiest for participants to detect. In contrast, M5 -Mixed-Highlight, M5 -Word-Color, and M2 -TokenBold produced the lowest recognition rates (≈ 74% − 79%), suggesting that mixed or token-level typographic cues were harder to identify when subtle or distributed across text. After manual verification, the overall pattern shifted slightly. M6 -Word-Highlight and M6 -Word-Bold remained among the most recognizable (≈ 92% and 85%), whereas M5 -Mixed-Highlight and M1 -Word-Cap-Flipping dropped further, showing that some participants over-reported recognition for less obvious manipulations. A few typographic cues (i.e. M1 -Word-Highlight, M4 -Word-Highlight) improved after manual validation, implying that participants may have underestimated their recognition performance for these specific cues during self-reporting.

ipants recruited through Prolific [1] were compensated at $8/hour in accordance with the platform’s minimum-wage policy. Only those who completed the full survey and submitted a valid completion code were included in the final analysis.

A.3

Results: Humans Perceive Toxic Phrases

As defined in 5.3, our primary metric is selection rate. Typographic cues with higher selection rates indicate more recognizable (i.e., less effective at concealing content). In contrast, a lower selection rate indicates that were deemed stronger obfuscations. Such obfuscation strategies that are more likely to evade human inspection and moderation in real-world settings. Based on this ranking, we selected the top 21 typographic cues for the second round evaluation. The second round study examined in a more detailed recognition task. Participants were asked not only to determine whether each text contained harmful content but also to type the specific harmful or aggressive sentences they identified. We then compared participants’ typed responses with the ground-truth toxic phrases to evaluate recognition accuracy and analyzed which typographic cues remained most detectable by human across the five topic domains (Hotel, Movie, Restaurant, Music, and Product). The results of this evaluation are presented in Section A.3. Performance of Obfuscation Categories On Different Platforms. Table 1 shows the results for participants having meaningful sentence as passages with using smartphone. For each mode, we consider three levels of granularity: word, token, and mixed. Regarding the methods of hiding toxic content, six styles were examined: highlight, bold, precomposed, capflipping, cloze, and color. The recognition patterns on the computer version closely mirror those observed on the phone version. Participants on both platforms more frequently recognized manipulations involving bold, highlight, and precomposed text, whereas cloze remained the least detectable across conditions. The computer condition showed slightly higher selection rates overall, likely due to the larger display, but these differences did not affect the relative ordering of cue effectiveness. Figure 10 (in Appendix) illustrates these patterns for smartphone condition: highlight, bold, and color manipulations yield large proportions of selected segments, while cloze produces predominantly unselected segments. This distribution overall cross-device trends, indicating that device type has minimal impact on relative recognition of different typographic cues. The top 21 typographic cues (bold in Table 1) were identified based on recognition rates exceeding 0.80 on phone. These styles dominated by bold, highlight, precomposed, and cap-flipping cross-platform stability and thus were selected for the second-round survey.

A.4

Adversarial Examples in Human Study

Round I question as Figure 9a. Round II representative samples are illustrated in Figure 9.

Details for Typographic Configuration Sets

Our typographic configurations are grounded in formatting features natively supported across five major platforms (Reddit, X/Twitter, Discord, Stack Overflow, and YouTube) ensuring that all configurations reflect realistic, deployable attack vectors. Detailed mapping is presented in Table 7. 21

(a) Example of Questionnaire in Round I.

(b) Examples of HPAA samples in Round II.

Figure 9: Question and HPAA examples in user study. The detailed motivations behind the design of the typographic cues are as follows: 1. Typographic Granularity (L). Typographic granularity defines the level at which toxic text can be segmented for typographic manipulation. This design dimension specifies how textual content is decomposed into units that subsequently serve as the targets of typographic transformations. We define a granularity space G consisting of three representative decomposition schemes, motivated by the observation that human readers can often recognize partially segmented words. Specifically, the Word-level (W) granularity preserves each word intact, serving as the least fragmented setting. The Token-level (T) granularity splits words of length four or more into subword units of at least two characters, introducing moderate fragmentation while largely preserving readability. The Mixed (Mix) granularity represents a more aggressive setting by combining token-level splits with singlecharacter fragments, resulting in decompositions that interleave short tokens and individual letters. Together, these three granularities span a spectrum of segmentation patterns that influence both human perceptibility and detector responses. 2. Placement Strategies (M). Placement strategies define the spatial patterns by which toxic content may be embedded within surrounding benign text. This design dimension characterizes where and how toxic spans can appear in the rendered sample, reflecting the diversity of user devices, screen layouts, and reading behaviors observed in real-world platforms. To capture such positional variability, we define a place-

ment strategy space M consisting of six representative spatial configurations, denoted as M1 through M6 , spanning vertical, diagonal, and randomized layouts. Specifically, M1 places the toxic content predominantly along the left vertical region of the sample, M2 centers it along the vertical axis, and M3 aligns it toward the right vertical region. M4 arranges the toxic span along the main diagonal from the upper-left to the lower-right, while M5 follows the anti-diagonal from the upper-right to the lower-left. Finally, M6 distributes the toxic content randomly throughout the benign text. Together, these strategies form a controlled yet representative set of spatial configurations for evaluating HPAA under diverse real-world display conditions. 3. Stylistic Transformations (S). Stylistic transformations characterize the visual modifications that can be applied to textual units within the toxic span. This design dimension captures a range of perceptual cues commonly supported by real-world user interfaces, enabling surface-level appearance changes while preserving the underlying semantic content. Such transformations reflect realistic manipulation capabilities available to attackers under typical platform constraints. We define a stylistic transformation space S consisting of a representative set of visual variants, S = {B, Col, Hi, Pre, Cap, Cloze}, each corresponding to a distinct visual dimension. Bold (B) increases glyph weight by rendering characters in boldface while leaving their lexical form unchanged. Color (Col) modifies the foreground color of characters, adjusting visual salience without alter22

Table 7: Platform support for typographic configurations. Supported Features

Configurations H ′

Post / Comment Unlimited

Highlight† , Bold

M6-W-Hi† , M5-W-Hi† , M1-WHi† , M6-W-B, M1-W-B, M4-WHi†

X/Twitter

Post / Comment 280 chars

Highlight, Bold, Precom- M6-W-Hi, M5-W-Hi, M1-W-Hi, posed, Color M1-W-Pre, M6-W-B, M3W-Col, M1-W-B, M4-W-Hi, M6-W-Pre, M1-T-Pre

Discord

Chat

Highlight, Bold, Precom- M6-W-Hi, M5-W-Hi, M1-W-Hi, posed, Color M1-W-Pre, M6-W-B, M3W-Col, M1-W-B, M4-W-Hi, M6-W-Pre, M1-T-Pre

Platform

Interface

Query Limit

2,000 chars

Stack Overflow Post / Comment 600 chars (com- Highlight, Bold, Precom- M6-W-Hi, M5-W-Hi, M1-W-Hi, ments), unlimited posed M1-W-Pre, M6-W-B, M1-W-B, (posts) M4-W-Hi, M6-W-Pre, M1-T-Pre YouTube

Comment

10,000 chars

Bold, Precomposed

M1-W-Pre, M6-W-B, M1-W-B, M6-W-Pre, M1-T-Pre

Reddit renders highlighted text with a grey background rather than the yellow background used in our study. Due to the potential impact of color on perceptual effectiveness, this configuration is excluded from our deployment analysis.

ing shape or casing. Highlight (Hi) introduces background contrast by placing a colored highlight behind characters, analogous to text-highlighting features widely supported by document editors and messaging interfaces. Precomposed (Pre) maps alphabetic characters to enclosed alphanumeric presentation forms (e.g., circled small letters in the Unicode range U+24D0–U+24E9), yielding visually distinct yet semantically recoverable variants. Cap-Flipping (Cap) performs case transposition by converting lowercase characters to uppercase, preserving lexical identity while altering surface form. Cloze replaces characters with underscore placeholders following the cloze-test pattern commonly used in educational materials, representing masking-style transformations within the typographic design space.

toxic dataset STTD and HED datasets are applied to test the evasion ability of HPAA samples from the detectors, with varied thresholds in Table 8, which shows that the models have limited capabilities to detect the toxic content from the HPAA samples (M6 -W-Hi for k = 1 and M1 -W-Pre for k = 3). Table 8: Detection rates (%) on STTD and HED datasets using representative BERT- and RoBERTa-based toxicity detectors.

BERT-based [60] Multilingual XLM-R [59] Unbiased RoBERTa [56] ParaDetox-based [42]

thr > 0.5

thr > 0.7

thr > 0.9

k=1

k=3

68.5 67.5 67.8 60.0

61.5 61.0 59.8 57.5

48.0 49.5 46.8 48.0

15.0 18.0 16.0 20.0

12.0 14.0 13.0 16.0

Llama Guard Hazard Taxonomy E

Table 9 illustrates the correspondence between our five toxicity categories and the Llama Guard hazard taxonomy (S1–S14). We identify the primary, most direct Llama Guard hazard class (shown in bold) for each of our categories, alongside secondary mappings on the right to account for potential partial overlaps.

HED (thr > 0.5)

STTD

Toxicity Classifier

White-box Defense

Our results indicate that effective defenses against HPAA currently require white-box access, whereas black-box moderation systems consistently fail across a diverse set of detectors, as demonstrated in the main evaluation. In practice, some platforms may apply preliminary text normalization steps that remove or ignore stylistic attributes such as color or formatting. Under this conservative assumption, the configuration space H is effectively restricted to capitalization-based perturbations. Nevertheless, HPAA adversarial samples can still be constructed in this setting; for example, the M1 -W-Cap configuration achieves approximately

Extended Evaluation

Four pre-trained toxicity detection models are compared [42, 56, 59, 60]. These models are based on BERT [15], RoBERTa [41], and XLM-RoBERTa [12] architectures. The 23

M1 - word

M1 - token

Selected Unselected

M1 - mixed

cap-flipping cloze

M1- word

color highlight

M1- token

M1- mixed

1.0

0.8

Selection Rate

Counts

precomposed bold

15 10

0.6 0.4

5 0

0.2

Cap

Pre

M2 - word

Col

Cap

Pre

M2 - token

Col

Cap

Pre

M2 - mixed

Col

Hi 0.0

15 10 5

Col

Cap

Pre B M2- token

Col

Cap

Pre B M2- mixed

Col

Cap

Pre B M3- word

Col

Cap

Pre B M3- token

Col

Cap

Pre B M3- mixed

Col

Cap

Pre B M4- word

Col

Cap

Pre B M4- token

Col

Cap

Pre B M4- mixed

Col

Cap

Pre B M5- word

Col

Cap

Pre B M5- token

Col

Cap

Pre B M5- mixed

Col

Cap

Pre B M6- word

Col

Cap

Pre B M6- token

Col

Cap

Pre B M6- mixed

Col

Cap

Pre B Style

Col

Cap

Pre B Style

Col

Cap

Pre B Style

Col

0.4 0.2

Cap

Pre

M3 - word

Col

Cap

Pre

M3 - token

Col

Cap

Pre

M3 - mixed

Col

0.0

1.0

0.8

Selection Rate

Counts

Pre B M2- word

0.6

15 10 5 0

0.8

Selection Rate

Counts

Cap

1.0

0.6 0.4 0.2

Cap

Pre

M4 - word

Col

Cap

Pre

M4 - token

Col

Cap

Pre

M4 - mixed

Col

0.0

35 1.0

0.8

Selection Rate

Counts

25 20 15 10 5 0

0.6 0.4 0.2

Cap

Pre

M5 - word

Col

Cap

Pre

M5 - token

Col

Cap

Pre

M5 - mixed

Col

0.0

1.0

30 0.8

Selection Rate

Counts

25 20 15 10

0.4 0.2

5 0

0.6

Cap

Pre

M6 - word

Col

Cap

Pre

M6 - token

Col

Cap

Pre

M6 - mixed

Col

0.0

1.0

30 0.8

Selection Rate

Counts

25 20 15

0.6 0.4

10 0.2

5 0

Cap

Pre

Style

Col

Cap

Pre

Style

Col

Cap

Pre

Style

Col

0.0

Figure 10: An image count selected and appeared performance for phone version online survey. Abbreviations: Cap = cap-flipping; C = cloze; Pre = precomposed; B = bold; Col = color; Hi = highlight.

Figure 11: Selection Rate by Mode, Granularity, and Style with 95% Confidence Intervals (Phone). Abbreviations: Cap = cap-flipping; C = cloze; Pre = precomposed; B = bold; Col = color; Hi = highlight. 24

80% human exposure. Traditional BERT-based (or RoBERTa-based) detectors can be adapted to mitigate HPAA by fine-tuning on adversarial samples generated under our framework. However, such defenses rely on careful and task-specific design choices. In particular, prompt-based mitigation strategies are sensitive to prompt formulation and do not generalize well. Our findings suggest that effective defense requires explicitly incorporating transformation rules or invariances into the model, rather than relying on example-based prompting alone. Importantly, implementing such defenses necessitates non-trivial, customized modifications that are not supported by existing deployed moderation systems. In summary, defending against HPAA is not a matter of applying minor engineering adjustments, but instead requires purposeful design and specialized engineering effort.

proaches, while powerful for alignment and generation, are ill-suited to our attacker-centric, perceptual setting. Preference-based learning typically requires a non-trivial amount of human feedback to achieve stable performance, yet there is limited guidance on how much data is sufficient to reliably capture perceptual judgments—particularly for subtle visual or typographic cues. Moreover, such methods introduce additional inductive biases through model architecture, initialization, and optimization dynamics, which can dominate or distort perceptual signals in low-data regimes. In adversarial settings, these biases are difficult to control, especially when no natural prior policy exists to anchor learning. In contrast, HPAA treats human perceptibility as an estimation problem rather than a learned objective. A small number of user study rounds suffices to identify robust typographic configurations that generalize across detectors, avoiding repeated human-in-the-loop optimization and significantly reducing cost. Notably, even if reinforcement learning were to be employed, the configurations identified by our protocol could serve as a strong initialization, enabling faster convergence toward satisfactory solutions. By decoupling human perceptibility from iterative optimization, HPAA enables a scalable, attacker-aligned design consistent with realistic threat models.

Discussion

F.1 Socio-Technical Perspective on Adversarial Attacks We move from a purely technical view of adversarial robustness to a socio-technical attack perspective, highlighting structured evasion strategies that exploit gaps between machine decision logic and human cognition. Importantly, HPAA does not argue that prior adversarial attacks are ineffective, nor that visually salient manipulations are absent from existing literature. Instead, the key distinction lies in the analytical emphasis. Previous work largely optimizes for imperceptibility, whereas HPAA explicitly considers scenarios in which partial human recognition of harmful content is acceptable. Under this setting, perturbations take the form of benign sentence insertions and typographic cue combinations, which preserve or even enhance human recognizability, in contrast to perturbation-based attacks that often degrade semantic clarity as a side effect [16]. Beyond demonstrating attack effectiveness, our study suggests a general protocol for exploring this attack surface. By leveraging user studies, we jointly evaluate human perceptibility and identify effective typographic configurations, enabling systematic extension of HPAA with additional cues or constraints. Together, these results indicate that adversarial robustness should be expanded to account for structured, presentation-level attacks that arise naturally in sociotechnical systems.

F.2.1

Effect of Attacker Utility Trade-offs

HPAA does not require attackers to achieve universal imperceptibility. By tolerating a fraction of users not recognizing toxic content, attackers can employ simpler or more aggressive configurations that more effectively evade automated detectors. This reflects an inherent tension between human perceptibility constraints and detector evasion, where stricter imperceptibility requirements substantially narrow the feasible configuration space.

F.3 Interpreting Detection-Rate Scores Across Evaluation Settings Although results vary across evaluation settings, our overall conclusions remain unchanged. Evaluating humanperceptible adversarial text remains inherently challenging due to variability in human perception and finite user studies; our evaluation therefore approximates practical deployment rather than exhaustive characterization. Despite these differences, the algorithm demonstrates strong effectiveness across settings, with finer-grained analyses left to future work.

F.2 Human Studies vs. Learned Preference Models One may ask why HPAA uses a small-scale user study for human refinement instead of learning-based methods such as reinforcement learning or preference modeling; such ap-

2 Llama Guard model card and taxonomy: https://www.llama.com/docs/

model-cards-and-prompt-formats/llama-guard-3/

Table 9: Mapping between our five toxicity categories and Llama Guard (S1–S14) hazard taxonomy.2 Primary matches in bold; secondary mappings are listed to the right. Our Category

Llama Guard Category (ID : Name) Secondary Mapping

Hate Speech S10: Hate Speech Self-Harm S11: Suicide & Self-Harm Violence S1: Violent Crimes Insult / Defamation S5: Defamation Sexual Content S12: Sexual Content

S9: Indiscriminate Weapons (generic insults: no direct class) S3: Sex-Related Crimes

Table 10: Text moderation systems evaluated in this work. Vendor

Abbrev.

Text Moderation System