arXiv CS open access

HarmTrace: Anchor-Calibrated Decoupled Optimization for Fine-Grained Target Identification in Harmful Memes

2026 · arxiv_cs

arXiv CS · Papers · 2026

artificial-intelligence knowledge-representation reasoning

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

HarmTrace: Anchor-Calibrated Decoupled Optimization for Fine-Grained Target Identification in Harmful Memes Yujia Li1 , Yiqun Zhang2 , Zihan Cheng1 , Yijie Huang1 , Tenglong Ye1 , Zihan Wang1 , Xiaocui Yang1 , Shi Feng1∗ , Yifei Zhang1 , Daling Wang1 1

School of Computer Science and Engineering, Northeastern University Shenyang 110819, China 2 Apple Inc., Beijing 100006, China liyujia@mails.neu.edu.cn, fengshi@cse.neu.edu.cn

Multimodal harmful meme detection is typically formulated as image–text harmfulness classification. A model may correctly predict harmfulness while misidentifying the attacked target or its supporting evidence. We therefore extend harmful meme detection with fine-grained target identification, asking what type of target is attacked, who is targeted, and where the target appears in the meme. The model predicts harmfulness for every meme and, for harmful memes, outputs the target category, target entity, textual mention, and visual region. To support this task, we introduce Meme3W, which unifies multiple public harmful meme datasets and provides human-verified annotations for harmful instances. We further introduce Joint Record Accuracy (JRA), a strict record-level metric requiring the harmfulness label and all target-identification fields to be jointly correct. Experiments with representative multimodal large language models reveal a substantial gap between harmfulness accuracy and JRA. To narrow this gap, we propose HarmTrace, an anchor-calibrated decoupled optimization framework. HarmTrace strengthens target-entity supervision through entity-aware supervised fine-tuning. It then applies Conditional Target-identification Policy Optimization (CTPO) to decouple harmfulness and target-identification advantages, restricting target-identification optimization to label-correct responses for harmful examples. CTPO uses a Virtual Positive Anchor (VPA) as a fully correct reference for target-identification advantage normalization. HarmTrace improves both JRA and harmfulness accuracy across the evaluated backbones, with JRA on the Qwen3-VL-8B backbone increasing from 17.58% to 52.51%. Our code is publicly available at https://github.com/llly1234/HarmTrace-for-HarmfulMemes.

Introduction Most existing methods formulate multimodal harmful meme detection primarily as image–text harmfulness classification (Kmainasi et al. 2026; Cheng et al. 2026; Wang et al. 2026; Hou et al. 2026). However, a correct harmfulness label alone does not establish whether the attacked target has been correctly identified (Mia and Fahim 2025). It also does not indicate which textual mention and visual region support that target identification. We therefore extend harmful meme detection with fine-grained target identification, predicting ∗

Corresponding author.

80 Score (%)

arXiv:2608.16622v1 [cs.CV] 17 Aug 2026

Abstract

74.88 47.95

Label: Target Category: Target Entity: Text Mention: Visual Region:

Harmful health condition people with physical disabilities “man” [388,53,908,995]

Label: Target Category: Target Entity: Text Mention: Visual Region:

Harmful gender woman null [49,247,480,992]

73.38 44.75

78.27

JRA Score: 1

72.53

71.97 Label Acc.

38.81

38.13

JRA 17.58

20 0

75.82

JRA Score: 0

6.62 Gemini3-Flash

GLM-5v

Kimi k2.5

Qwen3.5-397B Qwen3VL-8B InternVL3-8B

Figure 1: JRA requires the harmfulness label and all targetidentification fields to be jointly correct. Under this criterion, representative MLLMs show a clear gap between harmfulness accuracy and fine-grained target identification.

harmfulness for every meme and, for harmful memes, additionally identifying the target category, target entity, textual mention, and visual region. For harmful memes, these fields form a traceable record linking the harmfulness judgment to the attacked target and the textual and visual evidence supporting its identification. This record allows reviewers to assess whether the target and evidence are consistent with the final judgment. The EU Digital Services Act emphasizes that moderation decisions should be reasoned and open to review and contestation by affected users (Chiarella 2022). Fine-grained target identification provides the structured information needed for this examination by making the attacked target and its multimodal grounding explicit. The reliability of this record depends on the joint correctness of the harmfulness label and all target-identification fields. Existing harmful meme datasets provide annotations for only subsets of these fields (Hee, Chong, and Lee 2023; Shah et al. 2024; Bui, von der Wense, and Lauscher 2025). They therefore cannot support record-level evaluation of their joint correctness. To address this gap, we introduce Meme3W, a dataset for fine-grained target identification in harmful memes. Meme3W integrates multiple public datasets under a unified schema and provides human-verified

annotations for harmful instances. We further introduce Joint Record Accuracy (JRA), a strict record-level metric requiring the harmfulness label and all target-identification fields to be jointly correct. Evaluation of representative MLLMs on Meme3W reveals a clear gap between harmfulness prediction and jointly correct fine-grained target identification. As shown in Figure 1, harmfulness accuracy consistently exceeds JRA across model scales. The best JRA among generalpurpose MLLMs is 47.95%, while smaller MLLMs generally remain below 25%. To narrow this gap, we propose HarmTrace, an anchorcalibrated decoupled optimization framework for finegrained target identification in harmful memes. HarmTrace first performs entity-aware supervised initialization and then applies conditional policy optimization. Unlike standard SFT with uniform token weighting, Entity-aware Supervised Fine-Tuning (E-SFT) upweights target-entity tokens, strengthening supervision for the short field that connects the target category to textual and visual grounding. Rather than using a single aggregate reward, Conditional Targetidentification Policy Optimization (CTPO) separately normalizes harmfulness and target-identification advantages. It also restricts target-identification updates to label-correct harmful responses. Within CTPO, a Virtual Positive Anchor (VPA) adds a virtual reward representing complete correctness during target-identification advantage normalization. Experiments on two MLLM backbones show substantial improvements. HarmTrace raises JRA from 17.58% to 52.51% on Qwen3-VL-8B and from 6.62% to 49.09% on InternVL38B. It also improves all evaluated target-identification fields on both backbones. Our contributions are as follows: • We extend harmful meme detection with fine-grained target identification and construct Meme3W. It augments target-category annotations with human-verified target entities and available textual and visual grounding. We further introduce Joint Record Accuracy (JRA) for strict record-level evaluation. • We propose HarmTrace, an anchor-calibrated decoupled optimization framework. It strengthens targetentity supervision, decouples harmfulness and targetidentification credit assignment, and calibrates targetidentification advantages with VPA. • We evaluate representative MLLMs on Meme3W, reveal a clear gap between harmfulness prediction and finegrained target identification, and show that HarmTrace improves JRA across the evaluated backbones.

Related Work Multimodal Hateful Meme Detection Harmful meme understanding is moving beyond binary classification toward target prediction. Hateful Memes focuses on hateful/non-hateful classification (Kiela et al. 2020), while MAMI adds misogyny-type prediction (Fersini et al. 2022). Harm-C and PrideMM provide harmfulness levels or coarse target annotations (Pramanick et al. 2021; Shah et al. 2024); MemeMind and MemeIntel add chain-of-thought or

explanation-oriented supervision (Gu et al. 2025; Kmainasi et al. 2025); and MemeLens unifies multilingual and multitask meme resources (Shahraur, Bayan et al. 2026). However, these datasets do not provide unified annotations for the concrete target entity and its supporting textual and visual evidence, nor formulate these fields jointly with harmfulness prediction. Recent work improves harmful meme detection through multimodal representation learning (Burbi et al. 2023) and retrieval- or knowledge-augmented adaptation (Mei et al. 2024; Tzelepi and Mezaris 2025; Mei et al. 2025). Other methods introduce multimodal debate or generated explanations (Lin et al. 2024; Hee and Lee 2025; Mei et al. 2026), while DR-HM adopts reasoning-enhanced training (Cheng et al. 2026). These methods primarily optimize binary harmfulness classification. Fine-grained target identification is not included in their prediction objectives.

Reinforcement Learning for MLLMs Reinforcement learning has become an important posttraining paradigm for improving the reasoning and alignment of LLMs and MLLMs. PPO stabilizes policy updates through clipping and a learned value function (Schulman et al. 2017), DPO directly optimizes preference pairs (Rafailov et al. 2023), and GRPO estimates advantages within sampled groups without a critic (Shao et al. 2024). Recent variants further refine group-based optimization through dynamic sampling, process-level rewards, and negative-enhanced signals for all-negative groups (Yu et al. 2025; Tan et al. 2026; Nan et al. 2025). These methods are mainly studied in mathematical, coding, and general reasoning tasks. In harmful meme detection, HarmTrace applies group-based optimization to fine-grained target identification.

Meme3W: A Fine-Grained Target Identification Dataset for Harmful Memes Task Definition and Output Schema We extend multimodal harmful meme detection with finegrained target identification. The task predicts harmfulness for every meme and, for harmful memes, additionally identifies the target category, target entity, textual mention, and visual region. This extension enables evaluation beyond label correctness by making the attacked target and its available textual and visual evidence explicit. Given a dataset D = {(xi , a⋆i )}N i=1 , each input xi = (Ii , Ti ) consists of a meme image Ii and its associated text Ti . The structured annotation is defined as a⋆i = (li⋆ , c⋆i , e⋆i , s⋆i , Bi⋆ ),

(1)

where li⋆ ∈ {harmful, non-harmful} denotes the harmfulness label, c⋆i denotes the category of the attacked target, e⋆i denotes the specific target entity, s⋆i denotes the textual target mention, and Bi⋆ denotes a list of visual target regions represented by bounding boxes. For harmful samples, the category of the attacked target and the target entity are annotated, while the textual and visual grounding fields are populated only when the corresponding evidence is present. For non-harmful samples, there is no attacked target to identify. Therefore, all non-box

Input

OCR Text Original black friday sale

The image depicts a historical slave-sale scene, showing Black people gathered in the foreground and

Analyze

1 white traders nearby, with ships in the background. The text reads "original black friday sale."

Decode

2 ring a holiday. This is a direct mockery of the historical trauma, dehumanization, and racial persecu-

Distinguish

3 The white traders and ships are contextual or perpetrating elements, not the attacked target. The

Determine

4 The target is present in both modalities. The text explicitly mentions "black", and the image explic-

Classify

5 The meme is harmful as it mocks historical persecution and dehumanizes a protected group based on

The meme equates the forced capture and enslavement of Black people with a commercial 'sale' dution of Black people. It trivializes slavery by framing it as a 'deal'.

attacked target is Black people, who are represented as the people being sold. itly depicts Black people in the slave-sale context.

Where does the target appear?

race/ethnicity. This falls under 'group background'. Who is targeted?

What is the target category?

Output

Religion Caste

"Label": Harmful, "Target Category":"Group Background", "Target Entity": "Black people", "Text Mention": "black", "Visual Region":[110, 334, 439, 868]

Group Background Gender Original black friday sale

black people

Health Condition

Figure 2: A representative Meme3W annotation, showing the analysis process and a final structured output for fine-grained target identification, answering where the target appears, who is targeted, and which target category applies. target fields are set to null, while the visual-region field is set to [].

malformed fields are removed. The model identities are hidden, and the three candidates are randomly ordered for each sample to reduce model-specific anchoring.

Dataset Construction We construct Meme3W by curating samples from four public datasets: PrideMM (Shah et al. 2024), MAMI (Fersini et al. 2022), Hateful Memes (Kiela et al. 2020), and Harm-C (Pramanick et al. 2021). Their original annotations do not jointly identify the attacked target and its textual and visual grounding. We therefore re-annotate the selected samples under the unified schema in Eq. 1. Before annotation, we manually screened the collected samples and removed those whose visual content was unclear or could not be reliably recognized. The resulting benchmark contains 10,662 samples, including 4,418 harmful samples. Harmful samples are organized into four broad target categories: group background, gender, religion and caste, and health condition. Before splitting, we removed exact duplicates and assigned verified perceptual near-duplicate groups to the same split. Meme3W is partitioned into training (85%), validation (5%), and test (10%) sets.

Annotation Pipeline We construct the structured annotations through a threestage MLLM-assisted human annotation pipeline. Figure 2 presents a representative Meme3W annotation, including the analysis process and the resulting fine-grained targetidentification output. Stage 1: MLLM Candidate Initialization. We employ Gemini-3-Flash (Google 2025), GPT-5.2 (OpenAI 2025), and Qwen3.5-397B (Qwen Team 2026a) to independently generate one candidate annotation for each sample. All three models follow the same annotation prompt, which specifies five analysis steps and requires a final structured annotation. The outputs are normalized to the predefined format, and

Stage 2: Human Expert Annotation. Five trained graduate annotators with strong English proficiency and NLP backgrounds participate in this stage. They are trained on the task definition, output schema, and annotation guidelines. Each sample is independently reviewed by two annotators. The MLLM outputs serve only as editable references, and annotators may revise any field based on the sample evidence. Before adjudication, the field-specific agreement scores are Cohen’s κ = 0.908 for target category, normalized agreement of 0.813 for target entity, token-F1 of 0.824 for textual mention, and IoU@0.75 of 0.818 for visual regions. For target entities, semantically equivalent mentions are mapped to a shared canonical form before agreement computation. Stage 3: Human Expert Adjudication. Disagreements are independently reviewed by a third annotator and resolved through discussion among all three annotators. We further conduct a candidate-blind audit on 200 sampled cases, in which annotators receive only the raw samples and annotation guideline. Their annotations achieve an unweighted mean of 0.827 across the four field-specific agreement scores against the final gold annotations.

Method Framework Overview To improve the joint correctness of harmfulness and targetidentification predictions, we propose HarmTrace, an anchor-calibrated decoupled optimization framework. Figure 3 provides an overview. HarmTrace consists of Entityaware Supervised Fine-Tuning (E-SFT) and Conditional Target-identification Policy Optimization (CTPO). E-SFT

[Harmful]

Label

Rollout Responses

Think->Answer Filter

Think Answer

OCR Text

["Think"]

the pimp hand strong enough for man ...

["Label"]

Required Output Fields

["Target Entity"]

[:]

[woman]

["Text Mention"]

[:]

[...]

Label Target Category Target Entity Text Mention Visual Region

["Answer"] [:]

[...]

rtag = 0.35

Target Branch

{

λent 1

Entity token X λent Other tokens × 1

rtag = 0.75

NormG

rtag

VPA

1) Unequal Incomplete

Virtual Positive Anchor

Think y4 Answer rlab = 0

rlab = 1

rtag = 0.55

rlab =1

[...]

...

y1 y2 y3

Think y3 Answer

rlab = 1

All Label Rewards

Label Branch

Entity-aware Token Weighting

...

Think y2 Answer

rlab = 1

E-SFT Policy

Filtered Training Trajectory [...]

Conditional Target-identification Policy Optimization (CTPO)

Entity-aware SFT (E-SFT)

Teacher MLLM

Label-correct Harmful Subset C

Label Rollout Response

Rollout Response Input

[:]

�� = 0.35

�� = 1

rtag = N/A

Ai = Alab+Atag

Alab

Output NormC

Atag

2) Tied Incomplete

w/o VPA 0.35 0.55 0.75

w/o VPA 0.55 0.55 0.55

+VPA

~ μ

0.35

~ σ

0.55 0.75 1

Best partial Atag

0.55 0.55 0.55 1 Signal restored

~ σ =0 Atag = 0 ~ σ ≠ 0 Atag ≠ 0

Label Figure 3: Overview of HarmTrace. The left panel shows an input and its required output schema. E-SFT upweights target-entity tokens, while CTPO normalizes label rewards over all rollouts and target-identification rewards over the label-correct harmful X λent subset C. The bottom panel shows VPA for unequal and tied rewards among incomplete responses.

upweights target-entity tokens to strengthen entity supervision. CTPO decouples harmfulness and target-identification advantages and applies target-identification optimization only to label-correct harmful responses. Within CTPO, a Virtual Positive Anchor (VPA) augments target-identification advantage normalization with a virtual fully correct response. For training, HarmTrace adopts an explain-then-answer sequence: yi = {think : zi , answer : ai }, (2) where the answer field follows Eq. (1), and the think field serves as an intermediate reasoning scaffold for generating the structured answer.

(4)

where wt =

λent , t ∈ Ient , 1, otherwise,

(5)

Ient denotes the target-entity value positions and λent > 1 controls their supervision weight. The resulting policy initializes CTPO.

Conditional Target-identification Policy Optimization (CTPO)

Entity-aware Supervised Fine-Tuning (E-SFT) Before policy optimization, we initialize the policy through cold-start supervised fine-tuning. We use Gemini-3-Flash to generate one explain-then-answer trajectory for each training meme xi . Both trajectory generation and filtering use only the training split. We retain only trajectories whose structured answers are valid and consistent with the verified gold annotations: Dcold = {(xi , yi ) | Match(yi , a⋆i ) = 1} ,

(xi , yi ) ∈ Dcold , we optimize PT wt log pθ (yi,t | xi , yi,<t ) LE-SFT = − t=1 , PT t=1 wt

(3)

where Match(·) extracts the structured answer from yi and checks its format and consistency with a⋆i . The filtered trajectories are used for cold-start training, whereas CTPO uses the full training set. Standard SFT and E-SFT share the same trajectories and differ only in token weighting. The target entity identifies who is attacked and links the target category to textual and visual evidence, yet its short value contributes little to the sequence loss under uniform weighting. E-SFT therefore upweights the target_entity value tokens. For

For each input meme x, the old policy πθold samples a group of responses {yi }G i=1 . Responses violating the output format or label-conditioned schema receive zero rewards. For valid responses, rilab ∈ {0, 1} measures harmfulness-label correctness, while ritag ∈ [0, 1] combines applicable field scores with schema-applicability consistency. The latter penalizes missing required grounding and hallucinated non-applicable grounding, and is computed only for label-correct harmful responses, with ritag = 1 indicating complete correctness. Detailed reward weights and matching functions are provided in Appendix B.3. Combining the two rewards would mix label and targetidentification credit. CTPO therefore computes their advantages separately. The label advantage is normalized over the full rollout group: Alab = i

rilab − µlab , σlab + ϵnorm

(6)

where µlab and σlab are computed from {rjlab }G j=1 , and ϵnorm > 0 is a small constant for numerical stability. Target-identification learning is restricted to C = {j | rjlab = 1, l⋆ = harmful},

(7)

and responses outside C receive zero target-identification advantage. Normalizing target-identification rewards only within C has two limitations when all responses in C are incomplete. With unequal rewards, the best partial response is normalized only against other incomplete responses and can therefore receive a strong positive advantage. With tied rewards, the reward variance becomes zero and all target-identification advantages vanish. We therefore introduce a Virtual Positive Anchor (VPA) by adding a virtual fully correct score rmax = 1 to the normalization multiset: e tag = {rtag | j ∈ C} ⊎ {rmax }. R j C

(8)

When C is not empty, VPA is included only in the normalization statistics and contributes no policy-loss term. Let m = |C|. The resulting statistics are P tag + rmax j∈C rj tag µ eC = , m+1 v (9) u 1 X 2 σ eCtag = u r−µ etag . t C m+1 tag e r∈R C

The target-identification advantage is  tag  etag  ri − µ C , tag Atag = σ e + ϵ i norm C   0,

Metrics. We report three groups of metrics. 1) Harmfulness detection. We evaluate binary harmfulness classification using accuracy and F1 over all test samples. 2) Joint record correctness. Joint Record Accuracy (JRA) is computed over all gold-harmful samples. A record is correct only when the meme is predicted as harmful and every targetidentification field satisfies its respective matching criterion. For a field that is absent in the gold record, the prediction must also indicate absence. Let H = {i | li⋆ = harmful} denote the set of gold-harmful samples and F = {c, e, s, B} the target-identification fields. Here, ˆli and li⋆ denote the predicted and gold harmfulness labels. For each field f , fˆi , fi⋆ , Sf , and τf denote its predicted value, gold value, matching score, and correctness threshold, respectively. We define iY h i 1 X hˆ JRA = 1 li = li⋆ 1 Sf (fˆi , fi⋆ ) ≥ τf , |H| i∈H

i ∈ C,

(10)

i∈ / C.

If C is empty, we set Atag = 0 for all responses. The final i advantage combines the separately normalized terms: Ai = Alab + Atag i i .

Qwen3.5/3.6 (Qwen Team 2026a,b), InternVL3/3.5 (Chen et al. 2023; InternVL Team 2025), Gemma (Gemma Team 2024), and Kimi-K2.5 (Kimi Team 2026). Table 1 lists all models. 2) Harmful meme detection methods. We adapt EXPO-HM (Mei et al. 2026) from our standard SFT checkpoint and apply its RL optimization method to the finegrained target-identification task. 3) RL-based optimization methods. Under the same E-SFT initialization, we compare HarmTrace with PPO (Schulman et al. 2017), GRPO (Shao et al. 2024), DAPO (Yu et al. 2025), and PAPO (Tan et al. 2026).

(11)

CTPO uses Ai in the standard clipped GRPO objective, with the policy obtained after E-SFT fixed as πref . Let ρi,t (θ) denote the token-level probability ratio between πθ and πθold . The optimization objective is |yi | G 1 X 1 X JCTPO (θ) = Ex,{yi }∼πθold G i=1 |yi | t=1 min ρi,t (θ)Ai , clip ρi,t (θ), 1 − ϵclip , 1 + ϵclip Ai − βDKL πθ ∥ πref . (12)

Experiments Experimental Setup Baselines. We evaluate HarmTrace on Meme3W against three baseline groups. 1) General-purpose MLLMs, including Gemini 3 Flash (Google 2025), GPT-5.2 (OpenAI 2025), GLM-5V-Turbo (GLM-V Team 2026), Qwen3-VL models at multiple parameter scales (Qwen Team 2025),

f ∈F

(13) where (Sc , Se , Ss , SB ) = (EM, F1rel , F1tok , IoU) and (τc , τe , τs , τB ) = (1, 0.7, 0.7, 0.5). Semantically equivalent target-entity mentions are canonicalized before computing F1rel . Following prior token-overlap-based relaxed evaluation (Heo et al. 2025), we set τe = τs = 0.7 to allow minor lexical differences in target entities and boundary variations in textual mentions. For visual regions, IoU is computed between the minimum enclosing rectangles of all predicted and gold boxes (Yu et al. 2016). 3) Field-level diagnosis. On gold-harmful samples, we report EM for target category, F1rel for target entity, EM and token-F1 for textual mention, and set accuracy for visual regions at IoU thresholds of 0.5 and 0.75. Settings. All trainable methods were run for 3 epochs on 2×NVIDIA H200 GPUs using LoRA (Hu et al. 2021) with rank = 64 and α = 128. RL training uses a rollout group size of 8. Additional SFT and RL hyperparameters are provided in Appendix B.1.

Main Results Table 1 shows that current MLLMs achieve higher harmfulness accuracy than JRA, revealing a clear gap between harmfulness detection and jointly correct fine-grained target identification. Among the general-purpose MLLMs, the best JRA is 47.95%, while most smaller general-purpose MLLMs remain below 25% despite substantially higher harmfulness accuracy. Strong performance on individual fields also does not necessarily translate into a jointly correct record. The

Label

Model

Acc.

JRA

Category

Entity

Text

Visual

F1rel

63.93 68.72 65.30 70.55 72.60

61.42 65.98 61.19 66.21 68.72

54.57 58.09 58.68 55.02 57.99 61.70 59.82 56.39 55.02 59.10 60.27 57.08 57.76 61.12 63.93 60.73 60.73 64.93 66.67 62.56

F1 IoU50 IoU75

Gain (%)

Score (%)

100

JRA

Acc.

Table 4: Performance comparison of different RL methods under the same E-SFT-initialized Qwen3-VL-8B backbone. Method

Table 3: RL-stage ablation of HarmTrace under different starting points. w/o SFT denotes initializing RL from the base model; +Dec. adds decoupled reward optimization; and +VPA further adds VPA to +Dec.

Label

Method

Positive advantage of best incomplete responses

1.0 0.8 0.6 0.4 0.2 0.0

210

410

610

810

Step Dec. + VPA

Figure 5: Effect of VPA under E-SFT initialization. The left panel shows the zero-advantage group ratio, while the right tracks positive advantage of the best incomplete responses.

43.15%, while all E-SFT variants provide further gains. Among the tested weights, λent = 10 provides the most balanced performance, with a JRA of 45.89% and strong results on harmfulness and visual-region metrics. Although the other weights perform better on a few individual fields, their lower JRA suggests less consistent performance across the full output. Overall, E-SFT consistently improves over standard SFT across the tested weights, with λent = 10 showing a favorable balance between JRA, harmfulness detection, and visual-region performance. RL-Stage Components. To examine whether decoupled optimization and VPA remain effective under different su-

Table 5: Performance comparison of RL methods on targetidentification fields under the same E-SFT initialization.

pervised initializations, Table 3 evaluates their contributions under three starting points. Under E-SFT initialization, decoupled optimization raises JRA from 49.09% to 50.91%, and adding VPA further raises it to 52.51%. The same stepwise improvement is observed under the other two initializations. Figure 4 further shows that, under E-SFT initialization, both components improve all evaluated target-identification fields, indicating that the JRA improvement is not driven by a single field. To further examine the role of VPA, Figure 5 analyzes its effect on credit assignment. VPA reduces zero-advantage incomplete groups and lowers the positive advantage of the best incomplete responses. Overall, decoupled optimization and VPA consistently improve JRA across initialization settings, and their gains under E-SFT extend across all evaluated target-identification fields.

Comparison with RL Baselines To determine whether the gains arise from the optimization design of HarmTrace rather than from applying RL alone, we compare HarmTrace with PPO, GRPO, DAPO, and PAPO under the same E-SFT initialization. As shown in Tables 4 and 5, the generic RL methods show different strengths across metrics, while HarmTrace achieves the highest scores on all reported harmfulness and target-identification metrics. Among the generic RL methods, GRPO obtains the highest JRA of 49.09%, whereas HarmTrace reaches 52.51%. Overall, the optimization design of HarmTrace yields additional gains beyond those obtained by generic RL under the same E-SFT initialization.

Conclusion We study fine-grained target identification in harmful memes, where models jointly predict harmfulness, target category,

target entity, textual mention, and visual region. Together, these outputs form a structured record that can support moderation review. We introduce Meme3W with unified, humanverified annotations and Joint Record Accuracy (JRA) for strict record-level evaluation. We further propose HarmTrace, an anchor-calibrated decoupled optimization framework that combines entity-aware supervision, decoupled credit assignment, and a Virtual Positive Anchor. Experimental results show that HarmTrace improves JRA and all reported fine-grained target-identification fields. Future work will evaluate HarmTrace on larger MLLMs.

References Bui, M. D.; von der Wense, K.; and Lauscher, A. 2025. Multi3 Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision–Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 9714–9731. Albuquerque, New Mexico: Association for Computational Linguistics. Burbi, G.; Baldrati, A.; Agnolucci, L.; Bertini, M.; and Del Bimbo, A. 2023. Mapping memes to words for multimodal hateful meme classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2832–2836. Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; Li, B.; Luo, P.; Lu, T.; Qiao, Y.; and Dai, J. 2023. InternVL: Scaling up Vision Foundation Models and Aligning for Generic VisualLinguistic Tasks. Cheng, Z.; Ma, J.; Yang, X.; Wang, P.; Zhang, W.; Feng, S.; Wang, D.; Zhang, Y.; and Zhang, M. 2026. DR-HM: Distill-then-Reinforce Training with Cognition-Aware Data Synthesis for Harmful Meme Detection. In Findings of the Association for Computational Linguistics: ACL 2026, 42975–42993. San Diego, California, United States: Association for Computational Linguistics. Chiarella, M. L. 2022. Digital Markets Act (DMA) and Digital Services Act (DSA): New Rules for the EU Digital Environment. Athens Journal of Law. Fersini, E.; Gasparini, F.; Rizzi, G.; Saibene, A.; Chulvi, B.; Rosso, P.; Lees, A.; and Sorensen, J. 2022. SemEval-2022 task 5: Multimedia automatic misogyny identification. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), 533–549. Gemma Team. 2024. Gemma: Open models based on gemini research and technology. arXiv:2403.08295. GLM-V Team. 2026. GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents. arXiv:2604.26752. Google. 2025. Gemini 3 Flash: frontier intelligence built for speed. https://blog.google/products/gemini/gemini-3-flash/. Accessed: 2026-07-27. Gu, H.; Yu, Q.; Liu, Y.; Li, Z.; Hou, S.; Zhao, J.; and He, Z. 2025. MemeMind: A Large-Scale Multimodal Dataset with

Chain-of-Thought Reasoning for Harmful Meme Detection. arXiv:2506.18919. Hee, M. S.; Chong, W.-H.; and Lee, R. K.-W. 2023. Decoding the Underlying Meaning of Multimodal Hateful Memes. arXiv:2305.17678. Hee, M. S.; and Lee, R. K.-W. 2025. Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions. arXiv:2502.11073. Heo, R.; Seo, Y.; Lee, J.; and Lee, D. 2025. Can Large Language Models be Effective Online Opinion Miners? In Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 23097–23136. Suzhou, China: Association for Computational Linguistics. ISBN 979-8-89176-332-6. Hou, W.; Tu, H.; Wang, Y.; Zhang, Y.; Liu, Y.; Zhu, D.; Gao, L.; and Zhou, B. 2026. Beyond Single-View Detection: A Dual-Space Reasoning Framework for Interpretable Harmful Meme Understanding. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 10526–10544. San Diego, California, United States: Association for Computational Linguistics. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. InternVL Team. 2025. InternVL3.5: Advancing OpenSource Multimodal Models in Versatility, Reasoning, and Efficiency. arXiv:2508.18265. Kiela, D.; Firooz, H.; Mohan, A.; Goswami, V.; Singh, A.; Ringshia, P.; and Testuggine, D. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33: 2611– 2624. Kimi Team. 2026. Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276. Kmainasi, M. B.; Hasnat, A.; Hasan, M. A.; Shahroor, A. E.; and Alam, F. 2025. MemeIntel: Explainable Detection of Propagandistic and Hateful Memes. arXiv:2502.16612. Kmainasi, M. B.; Kutlu, M.; Shahroor, A. E.; Hasnat, A.; and Alam, F. 2026. Can Thinking Models Think to Detect Hateful Memes? arXiv:2603.01225. Lin, H.; Luo, Z.; Gao, W.; Ma, J.; Wang, B.; and Yang, R. 2024. Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models. In Proceedings of the ACM Web Conference 2024. Mei, J.; Chen, J.; Lin, W.; Byrne, B.; and Tomalin, M. 2024. Improving Hateful Meme Detection through RetrievalGuided Contrastive Learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5333–5347. Mei, J.; Chen, J.; Yang, G.; Lin, W.; and Byrne, B. 2025. Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection. In Conference on Empirical Methods in Natural Language Processing.

Mei, J.; Sun, M.; Chen, J.; Qin, P.; Li, Y.; Chen, D.; and Byrne, B. 2026. ExPO-HM: Learning to Explain-thenDetect for Hateful Meme Detection. arXiv:2510.08630. Mia, M. A.; and Fahim, M. 2025. BanHateME: Understanding Hate in Bangla Memes thorough Detection, Categorization, and Target Profiling. In Proceedings of the Second Workshop on Bangla Language Processing (BLP2025), 180–195. Mumbai, India: Association for Computational Linguistics. Nan, G.; Chen, S.; Huang, J.; Lu, M.; Wang, D.; Xie, C.; Xiong, W.; Zeng, X.; Zhou, Q.; Li, Y.; et al. 2025. Ngrpo: Negative-enhanced group relative policy optimization. arXiv:2509.18851. OpenAI. 2025. Update to GPT-5 System Card: GPT-5.2. https://cdn.openai.com/pdf/3a4153c8-c748-4b718e31-aecbde944f8d/oai_5_2_system-card.pdf. Accessed: 2026-07-27. Pramanick, S.; Sharma, S.; Dimitrov, D.; Akhtar, M. S.; Nakov, P.; and Chakraborty, T. 2021. MOMENTA: A multimodal framework for detecting harmful memes and their targets. In Findings of the association for computational linguistics: EMNLP 2021, 4439–4455. Qwen Team. 2025. Qwen3-VL Technical Report. arXiv:2511.21631. Qwen Team. 2026a. Qwen3.5: Towards Native Multimodal Agents. https://qwen.ai/blog?id=qwen3.5. Accessed: 202607-27. Qwen Team. 2026b. Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model. https://qwen.ai/blog?id=qwen3.627b. Accessed: 2026-07-27. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems, volume 36. Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347. Shah, S. B.; Shiwakoti, S.; Chaudhary, M.; and Wang, H. 2024. Memeclip: Leveraging clip representations for multimodal meme classification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 17320–17332. Shahraur, A.; Bayan, M.; et al. 2026. MemeLens: A Multimodal, Multilingual Benchmark for Meme Understanding. arXiv:2601.12539. Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. Tan, Z.; Yu, Z.; Lin, B.; Geng, Z.; Geng, H.; Zhang, Y.; Zhang, M.; Chen, Y.; Hu, S.; Yin, Z.; Zhang, C.; and Bai, L. 2026. PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization. arXiv:2603.26535. Tzelepi, M.; and Mezaris, V. 2025. Improving Multimodal Hateful Meme Detection Exploiting LMM-Generated

Knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 202–211. Wang, X.; Su, Y.; Li, W.; Wang, X.; Li, Z.; and Liu, A. 2026. SGoT-R1: Social Graph of Thought Reasoning-Enhanced Multimodal Large Language Model for Harmful Meme Detection. In AAAI Conference on Artificial Intelligence. Yu, J.; Jiang, Y.; Wang, Z.; Cao, Z.; and Huang, T. 2016. Unitbox: An advanced object detection network. In Proceedings of the 24th ACM international conference on Multimedia, 516–520. Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; et al. 2025. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.

A Meme3W Construction and Validation This section details the construction and validation of Meme3W, including data collection and filtering, the annotation pipeline, annotation quality assessment, and dataset release. Disclaimer. This paper contains harmful content, which has the potential to be offensive and may disturb readers.

A.1 Data Curation, Unified Schema, and Annotation Prompt Data sources and screening. Meme3W is curated from four existing multimodal meme datasets, namely PrideMM (Shah et al. 2024), MAMI (Fersini et al. 2022), Hateful Memes (FHM) (Kiela et al. 2020), and Harm-C (Pramanick et al. 2021). These datasets provide harmful and non-harmful examples, but their original annotations cover only subsets of the target-identification fields and do not jointly identify the attacked target and its textual and visual grounding. FHM covers all target categories considered in Meme3W, although racial, ethnic, and religious targets dominate its harmful subset. To broaden and balance target coverage, we supplement it with harmful memes from MAMI, PrideMM, and Harm-C, which focus on misogyny, LGBTQ-related harm, and COVID-19-related harm, respectively. After removing samples with unclear or unreliable visual content, we re-annotate the retained harmful memes under the unified Meme3W schema. Table 6 compares the annotation fields provided by the source datasets with those defined in Meme3W. FHM category distribution before augmentation. Table 7 reports the target-category distribution of the retained FHM harmful subset, which contains 3,548 memes after applying the same filtering and deduplication criteria used for Meme3W. Harmfulness definition. A meme is considered harmful when it directs harmful behavior toward a clear target. Harmful behavior includes targeted satire, insults, mockery, degradation, stereotyping, dehumanization, threats, exclusion, segregation, discrimination, assertions of inferiority, comparisons to animals or objects, mockery of hate crimes or historical suffering, and gender- or sexuality-based harm.

Original annotations Dataset

Topic

#Total

#Harm.

Harm.

Tgt. cat.

Tgt. ent.

Text

Box

PrideMM (Shah et al. 2024) MAMI (Fersini et al. 2022) Hateful Memes (Kiela et al. 2020) Harm-C (Pramanick et al. 2021)

LGBTQ+ Misogyny General hate COVID-19

5,063 11,000 10,000 3,544

2,482 5,500 5,000 1,249

✓ ✓ ✓ ✓

✓ ✓ – ✓

– – – –

Meme3W

Mixed

10,662

4,418

✓

Table 6: Annotation fields in the source datasets and Meme3W. Harm. denotes harmfulness labels. Tgt. cat. denotes sourcespecific target-category annotations defined under different taxonomies. Tgt. ent., Text, and Box denote target entities, textual mentions, and visual bounding boxes, respectively. A check mark indicates field availability.

Attack target

Count

Share (%)

Group background Religion and caste Gender Health condition

1,463 1,090 677 318

41.23 30.72 19.08 8.96

Total

3,548

100.00

Table 7: Attack-target distribution of the retained FHM harmful subset before augmentation with MAMI, PrideMM, and Harm-C. Counts use the same filtering criteria as the final Meme3W dataset. Ordinary humor, non-targeted profanity, general criticism, and attacks on criminals, terrorists, or criminal activities are considered non-harmful. Unified schema and taxonomy. Each meme is annotated with a harmfulness label, target category, target entity, textual mention, and visual region. Because non-harmful memes contain no attacked target, all target-identification fields other than the visual-region field are set to null, while the visualregion field is set to an empty list []. For harmful memes, the target category and target entity are always annotated. The textual mention and visual region are included only when corresponding evidence is available. Otherwise, the textual mention is set to null, and the visual region is set to []. A prediction is considered correct for an absent field only when it likewise indicates the field’s absence. The unified target-category taxonomy contains four categories: • group background includes ethnicity, race, nationality, regional or ethnic origin, minority background, and immigration or migrant status. • religion and caste includes religion, religious belief, religious identity, followers or believers of a religion, religious people, religious groups, and caste. • gender includes biological sex, gender identity, and sexual orientation. • health condition includes disability, disease, vaccination-related targets, infection, bodily condition, medical treatment, public-health measures, and health-related policy framing. Unified output schema. Each complete model response operationalizes the explain-then-answer sequence in Eq. (2)

of the main paper as a single flat JSON object. The think field contains auxiliary reasoning. The remaining five fields are label, target_category, target_entity, text_mention, and visual_region. Together, these five fields form the structured prediction ai and follow the structured annotation defined in Eq. (1) of the main paper. In Eq. (2), answer : ai denotes these five fields collectively rather than a literal answer key. Only these five fields are used for semantic matching and target-reward computation, while think is not semantically scored. Before scoring, the response parser extracts them as the input to the verifier. Field applicability and absence values follow the unified schema defined above. Shared prompting protocol. The three MLLMs used for candidate generation and the MLLMs evaluated in our experiments follow the same task definition, five-step analysis procedure, and structured output schema, with modelspecific conversation templates applied when necessary. The operational prompt instructions are provided below. 1. Analyze the visual and textual content. First enumerate all important visible elements in the image, rather than considering only the most salient one. These may include every visible person or group, object, symbol, animal, scene, gesture, item of clothing, flag, religious marker, medical marker, and visible text region. Do not omit an element solely because it may not be the attacked target. Identify the element first and determine its role afterward. Also identify the relevant cues in the accompanying text. 2. Decode the meme semantics. Analyze relevant linguistic and contextual cues, including puns, double meanings, metaphors, euphemisms, coded expressions, offensive terms, symbolic associations, stereotypes, and historical, cultural, or social context. Consider whether the meme conveys insults, discrimination, dehumanization, exclusion, inferiority, or mockery of hate crimes, historical persecution, violence, disability, disease, or suffering. If no such semantic cue is relevant, state this briefly. 3. Distinguish the attacked target from contextual elements. Identify visually salient figures and symbols, and determine whether each is itself being attacked or instead functions as a perpetrator, criminal, terrorist, background object, metaphor, symbol, or visual vehicle used to attack another target. Identify the actual attacked target based on the complete meaning of the meme.

4. Determine the target evidence. For a harmful meme, record the exact textual mention only when the attacked target is explicitly referred to in the accompanying text, and identify the corresponding visual region or regions only when the attacked target is explicitly depicted in the image. Otherwise, assign the predefined absence value to the corresponding field. 5. Classify the meme and target category. Explicitly determine whether harmful behavior is directed toward a clear target and identify the behavior type that is present or absent. Relevant behavior types include targeted satire, insult, mockery, degradation, stereotyping, dehumanization, threats, exclusion, segregation, discrimination, and assertions of inferiority. If no harmful behavior is directed toward a clear target, classify the meme as non-harmful. For a harmful meme, determine the target category and target entity. Finally, return the result using the unified output schema. Output format and normalization. Candidate models receive the meme image, its associated text, the task instructions, and the unified output schema. Each complete response contains intermediate analysis in think together with the five prediction fields in a single flat JSON object. The text_mention field contains the exact text span referring to the attacked target, while the visual_region field contains a list of target bounding boxes, each represented as [x1 , y1 , x2 , y2 ]. All bounding-box coordinates are normalized to the range [0, 1000], where (x1 , y1 ) and (x2 , y2 ) denote the top-left and bottom-right corners, respectively. The parser extracts the five prediction fields, and the verifier validates their values. The resulting predictions are presented to annotators as editable references. Field absence follows the unified schema defined above.

A.2 Human Annotation and Quality Control Annotation guidelines. Annotators follow the requirements below: 1. Determine harmfulness and target category. Annotators first determine whether the meme is harmful. For each harmful meme, the attacked target is assigned to exactly one category from the taxonomy defined in Appendix A.1. Samples whose target category cannot be determined reliably, or that independently attack multiple categories, are excluded rather than forced into a single category. 2. Interpret the meme from the author’s perspective. Harmfulness and target identity are determined from the intended meaning conveyed by the meme author, based only on the image–text content. Annotators consider satire, metaphor, insinuation, double meanings, stereotypes, and relevant historical, cultural, or social context rather than relying only on the literal wording. 3. Identify the actual attacked entity. The target entity must precisely describe the person or group being attacked at an appropriate level of granularity. Visually salient perpetrators, symbols, background figures, and contextual objects must not be annotated as targets unless they are themselves being attacked. When the target

Figure 6: Candidate-assisted human-annotation interface. Annotators inspect the source text and three anonymized MLLM candidates, revise all structured fields, and draw or adjust image-target boxes before saving the annotation.

is conveyed indirectly, the entity is inferred from the complete meaning of the meme. 4. Copy textual mentions exactly. When the attacked target is explicitly referred to in the associated text, annotators copy the exact text span without paraphrasing, expanding, or correcting it. If no explicit textual mention is present, the field is set to null. 5. Mark complete visual target regions. When the attacked target is visually depicted, each bounding box should tightly cover the complete visible extent of the target rather than only its face or another salient part. Multiple boxes are used when multiple visual instances of the same target are present. If the target is not explicitly depicted, the visual-region field is set to []. 6. Ensure accurate multimodal grounding. Textual mentions and visual regions must accurately correspond to the attacked target in the respective modality. Independent annotation and adjudication. Each meme is independently annotated by two annotators using editable MLLM candidates as references. The identity of the model producing each candidate is hidden, and candidate positions are randomly ordered in the annotation interface. Disagreements are independently reviewed by a third annotator and then resolved through discussion to produce the final gold annotation. Figure 6 shows the interfaces used for the initial independent annotation and the third-annotator review. Entity canonicalization. Before computing target-entity agreement and evaluation metrics, we apply a deterministic canonicalization procedure to normalize surface-form variation. We normalize casing, punctuation, and whitespace, and map referentially equivalent spelling, plurality, abbreviation, and referring-expression variants to a shared canonical form. Broader groups and their subgroups, as well as modifiers that alter the target identity, remain distinct. The canonicalization lexicon was manually reviewed, finalized prior to model evaluation, and kept fixed across all experiments. The canonicalized mentions are subsequently compared using F 1rel . For JRA, the target-entity field is considered correct when F 1rel ≥ 0.7.

Entity A

Entity B

Same Form

Field

Metric

women Muslim people Jewish people LGBTQ people immigrants Asian people Black women

woman Muslims Jews LGBTQ community refugees Chinese people women

Yes Yes Yes Yes No No No

Target category Target entity Textual mention Visual region Unweighted mean

Category agreement Normalized F1 Token-F1 IoU –

Table 8: Examples of target-entity canonicalization. Yes indicates that two mentions map to the same canonical form; No indicates that they remain distinct. Final target-entity correctness is evaluated using F 1rel .

Blind vs. Gold 0.900 0.797 0.808 0.803 0.827

Table 9: Candidate-blind audit against the final gold annotations. Group background

Religion and caste

Gender

Health condition

text-only

Group background

18%

34%

23% 25%

662

Gender

478

Religion and caste

523

Health condition 0

(a) Attacked Target Distribution

image-only

text+image 398

241 303

446

316 270

320

249

212

200

400 600 800 1000 1200 1400 1600 (b) Target Modality Composition

Figure 8: Attack-target distribution and target-modality composition among the 4,418 harmful memes in Meme3W. ified near-duplicate groups are assigned to the same split. Recurring meme templates with different textual or visual content are retained as distinct samples.

Figure 7: Candidate-blind annotation interface without access to MLLM candidates. Candidate-blind audit. We conduct a candidate-blind audit on 200 cases sampled to cover all four target categories and different textual and visual grounding conditions. Annotators receive only the raw meme and annotation guidelines, without access to the MLLM candidate annotations. The blind annotations are compared with the final gold annotations using the same field-specific metrics as in the main annotation process. Figure 7 shows the candidate-blind annotation interface. On this subset, re-evaluation against the blind annotations preserved the relative model ordering obtained with the final gold annotations, suggesting that candidate assistance did not alter the main comparative conclusion.

A.3 Dataset Statistics and Split Integrity Dataset statistics. Meme3W contains 10,662 memes, including 4,418 harmful and 6,244 non-harmful examples. Table 10 reports the exact numbers of examples in each split. Figure 8 shows the overall attack-target distribution and target-modality composition of the harmful subset. Split integrity. We verify that no exact image duplicates occur across the training, validation, and test splits using pixel-level hashing. Near-duplicate candidates identified through perceptual hashing are manually reviewed, and ver-

Data release and responsible use. We will release the structured annotations, fixed data splits, annotation guidelines, shared prompt, and evaluation code. Raw images or source identifiers will be distributed in accordance with the licenses of the corresponding source datasets. Meme3W contains offensive and discriminatory content and is intended for research on harmful-content understanding and moderation.

B Experimental Details and Implementation B.1 Implementation and Evaluation Protocol Shared evaluation prompt. All evaluated MLLMs use the shared task instructions, five-step analysis procedure, and structured output schema described in Appendix A. Each response follows the explain-then-answer format. The think field contains intermediate analysis, while only the five prediction fields are used for evaluation. Model and decoding settings. For API-based baselines, we use the official model endpoints available at the time of evaluation. For open-source baselines, we use the publicly released instruction-tuned versions of the models reported in the main-results table. All models follow the same task definition, output schema, and parsing rules. Malformed structured outputs are treated as invalid predictions. ExPO-HM follows an SFT-to-RL pipeline. Because its task-specific SFT data are not publicly available, we initialize it from our standard SFT checkpoint trained on Meme3W and subsequently apply the ExPO-HM optimization procedure. Training hyperparameters. We use ms-swift for supervised fine-tuning and verl for reinforcement learning. All

Split

Harmful

Non-harmful

Total

Parameter

Value

Training Validation Test

3,759 221 438

5,307 312 625

9,066 533 1,063

Total

4,418

6,244

10,662

LoRA rank / alpha Epochs Train batch size PPO mini-batch size Rollout group size Clip ratio KL coefficient Rollout temperature

64 / 128 3 32 32 8 0.2 0.02 0.8

Table 10: Numbers of harmful and non-harmful memes in each Meme3W split. Parameter

Value

LoRA rank / alpha Epochs Learning rate Max sequence length Per-device train batch size Gradient accumulation steps Effective batch size

64 / 128 3 1 × 10−4 4096 8 2 32

Table 11: SFT hyperparameter settings. trainable methods use LoRA (Hu et al. 2021) with rank 64 and α = 128. The key hyperparameters are summarized in Tables 11 and 12. All hyperparameters and checkpointselection criteria for the reported systems were determined exclusively on the validation split. Test-set results were computed only after the corresponding configurations had been fixed, and no test result was used for hyperparameter or checkpoint selection.

B.2 Teacher-Trajectory Generation and Filtering We use Gemini 3 Flash to generate one explain-then-answer trajectory for each of the 9,066 training samples. Each trajectory first provides intermediate analysis in the think field and then produces the five prediction fields in the same flat JSON object. After automatic filtering, 6,005 trajectories are retained. A trajectory is retained only when its response is parseable and all five fields satisfy the corresponding goldmatching requirements. The filtered trajectories are used only for cold-start SFT and E-SFT. Standard SFT and E-SFT use the same filtered trajectory set and differ only in token weighting. The trajectory filtering does not remove samples from subsequent policy optimization. The subsequent policy-optimization stage uses the full training set.

B.3 Reward Components and Active Target-Identification Fields Format and label reward. Following Eq. (1) of the main paper, only the five prediction fields in ai enter semantic reward computation. The response parser extracts these fields from the flat response object. The verifier then checks the resulting five-field JSON representation against the unified output schema defined in Appendix A.1 and validates the prediction-field value types and bounding-box coordinates. Invalid JSON, missing or additional prediction fields, invalid field values, and invalid bounding boxes fail the format check.

Table 12: RL hyperparameter settings. The binary label reward is rilab = 1[format valid]1[ˆli = li⋆ ].

(14)

Responses that fail the format check receive zero label reward. For a non-harmful prediction, target_category, target_entity, and text_mention must be null, while visual_region must be []. Target-identification rewards are computed only for format-valid, label-correct responses to gold-harmful samples. Applicable-field reward. The target-identification reward provides graded credit for partially correct predictions while preserving complete-record consistency. For a gold-harmful sample, let c, e, s, B, and q denote target category, target entity, textual mention, visual region, and schema consistency, respectively. The active component set is Ai = {c, e, q} ∪ {s : s⋆i ̸= null} ∪ {B : Bi⋆ ̸= ∅}. (15) Category, entity, and schema consistency are always active. Textual and visual components are included only when the corresponding gold evidence is present. The active scores are combined as P f ∈A wf ri,f tag , (16) ri = P i f ∈Ai wf where the fixed weights are 0.25 for category, 0.25 for entity, 0.20 for textual mention, 0.20 for visual region, and 0.10 for schema consistency. These weights sum to 1. When textual or visual evidence is absent, the denominator renormalizes the remaining active weights. Therefore, ritag ∈ [0, 1] and reaches 1 when all active components receive a score of 1. The category component uses exact match. The entity component is ri,e = max(Srel , λ F1ent + (1 − λ) EMent ) ,

(17)

where λ = 0.7, and Srel follows the entity canonicalization and relation matching described in Appendix A.2. The textual component is ri,s = max(EMtok , λ F1tok + (1 − λ) EMtok ) .

(18)

The schema-consistency component equals 1 only when the predicted category is valid and the predicted presence or absence of both textual and visual evidence matches the gold annotation. Otherwise, it equals 0.

49.09 52.51 +3.42

47.49 52.05 +4.57

46.35 50.46 +4.11

46.35 50.23 +3.88

44.52 50.00 +5.48

Table 13: Sensitivity of JRA to the minimum-enclosingrectangle IoU threshold with τe = τs = 0.7. Values are percentages on the 438 gold-harmful test memes, and ∆ denotes HarmTrace minus GRPO.

0.5

4.34

4.33

3.65

3.88

0.6

4.11

3.42

3.65

3.66

4.4 4.2 0.7

B.4 Threshold Sensitivity and Visual Matching Robustness Entity and text threshold sensitivity. We set τe = τs = 0.7 for target-entity and textual-mention matching. To assess sensitivity to these choices, we vary τe and τs over {0.5, 0.6, 0.7, 0.8, 1.0} and evaluate JRA on the complete 5×5 grid. Figure 9 shows the JRA difference between HarmTrace and GRPO under each threshold combination. HarmTrace remains above GRPO for all 25 settings, with ∆JRA ranging from 3.20 to 4.80 points, indicating that the model ordering is not sensitive to the selected entity or textualmention threshold. Visual IoU threshold sensitivity. With τe = τs = 0.7 fixed, we vary the minimum-enclosing-rectangle IoU threshold. As shown in Table 13, HarmTrace remains above GRPO at every evaluated threshold. Visual-region matching robustness. We evaluate whether the JRA comparison depends on the visual-region matching rule. In addition to minimum-enclosing-rectangle IoU, we consider one-to-one bipartite matching with no unmatched boxes and geometric set IoU over the unions of predicted and gold regions. All rules use an IoU threshold of 0.5, with the remaining JRA criteria unchanged. HarmTrace

4.11

3.42

3.65

3.66

4.0 3.8

0.8

Visual-region reward and penalties. The visual component uses soft matching to provide graded credit for spatially close predictions, including imperfect box matches. For a predicted box b and gold box g, the pairwise score is   IoU,         IoU SB (b, g) = max γIoU min 1, + γCov Cov ,   τsoft       + γPrec Prec + γctr Sctr (19) where (γIoU , γCov , γPrec , γctr ) = (0.55, 0.20, 0.15, 0.10) and τsoft = 0.6. Here, Cov, Prec, and Sctr denote goldregion coverage, prediction precision, and center proximity. For multiple regions, predicted and gold boxes are greedily matched and their scores are averaged. The minimum enclosing rectangle is also considered when multiple predicted boxes jointly cover one gold region. Each additional predicted box with best-match score below 0.20 incurs a 0.05 penalty. If the gold annotation contains no visual region, the visual component is inactive and any predicted region sets the schema-consistency score to 0.

4.6

3.89

3.88

3.20

3.42

3.42 3.6

1.0

4.79

4.80

0.5

0.6

4.11

4.33

4.34

0.7 0.8 Text threshold τs

1.0

ΔJRA (HarmTrace − GRPO)

GRPO HarmTrace ∆

4.8

IoU 0.5 IoU 0.6 IoU 0.7 IoU 0.75 IoU 0.8

Entity threshold τe

Model

3.4 3.2

Figure 9: Sensitivity of the JRA improvement to the targetentity threshold τe and textual-mention threshold τs . Each cell reports the JRA difference between HarmTrace and GRPO in percentage points. Visual rule

GRPO

HarmTrace

∆JRA

Enclosing-rectangle IoU Bipartite set matching Geometric set IoU

49.09 46.12 49.77

52.51 50.00 52.28

+3.42 +3.88 +2.51

Table 14: JRA under alternative visual-region matching rules. HarmTrace remains above GRPO under all three rules. remains above GRPO under all three matching rules, indicating that the model ordering is not determined by the visual-region matching implementation.

B.5 Additional Ablations and Robustness Analyses We report supplementary statistical and robustness analyses that are not included in the main paper. Statistical significance analysis. We quantify uncertainty in the primary JRA metric using 10,000 paired percentilebootstrap resamples of the 438 gold-harmful test memes, applying the same resampled indices to both systems in each comparison. As shown in Table 15, HarmTrace achieves 52.51over E-SFT (45.89%) by 6.62 percentage points, with a paired 95interval of [2.51, 10.96]. Under the same E-SFT initialization, it also outperforms GRPO (49.09%) by 3.42 points, with a paired 95% confidence interval of [0.23, 6.85]. Category-wise target identification. To examine whether the overall JRA gains extend across target categories, we compare E-SFT, GRPO, and HarmTrace. Table 16 reports category-wise JRA on gold-harmful test memes using the same joint correctness criteria as the overall evaluation. HarmTrace achieves the highest overall JRA, outperforming E-SFT and GRPO by 6.62 and 3.42 points, respectively. Its gains are concentrated in group background and gender,

Comparator

JRA

HarmTrace

∆JRA

Paired 95% CI

E-SFT GRPO

45.89 49.09

52.51 52.51

+6.62 +3.42

[2.51, 10.96] [0.23, 6.85]

Table 15: Paired comparisons of JRA on the 438 goldharmful test memes. All values are percentages. Confidence intervals are percentile intervals from 10,000 paired bootstrap resamples of test instances. Cat.

E-SFT

GRPO

Ours

Group/bg. Religion/caste Gender Health cond.

146 111 103 78

46.58 60.36 44.66 25.64

50.00 63.96 47.57 28.21

56.16 62.16 56.31 26.92

Overall

438

45.89

49.09

52.51

Table 16: Category-wise JRA for E-SFT, GRPO, and HarmTrace on gold-harmful test memes. where it improves over GRPO by 6.16 and 8.74 points and over E-SFT by 9.58 and 11.65 points. On religion and caste and health condition, HarmTrace remains above E-SFT but is slightly below GRPO by 1.80 and 1.29 points. These differences correspond to only two and one test examples, respectively. Overall, the gains on group background and gender outweigh these minor decreases.

C. Qualitative Case Studies Figures 10 and 11 present two representative successful cases. In both cases, HarmTrace correctly predicts the harmfulness label and all target-identification fields. These fields include the target category, target entity, textual mention, and visual region. The resulting records therefore satisfy all JRA criteria. Figures 12 and 13 illustrate two different forms of crossmodal target misidentification. In the first case, the text explicitly mentions a man. However, “dishwasher” is used as a derogatory reference to the woman shown in the image. HarmTrace follows the explicit mention of the man. It therefore fails to associate the implicit textual attack with the woman. In the second case, the attacked target is a girl mentioned only in the text. The image, however, depicts a Black man. HarmTrace relies on the visible person and incorrectly predicts and grounds the man as the target. These cases suggest that the model may over-rely on a single modality when textual and visual cues are not directly aligned. This can lead to incorrect identification of the attacked target.

D. Ethical Considerations Data use and release. Meme3W is derived from four publicly released research datasets, namely PrideMM, MAMI, Hateful Memes, and Harm-C. We will comply with their respective licenses and redistribution requirements. When raw-image redistribution is not permitted, we will release only structured annotations, source identifiers, and processing code. Because the data may contain identifiable individ-

uals, slurs, or stigmatizing content, the release will include a content warning and a procedure for reviewing removal requests. Annotator welfare and oversight. Five graduate-student annotators were informed in advance that the task involved potentially offensive and discriminatory content. Participation was voluntary, and annotators could skip individual examples or withdraw from the task. They were compensated at a rate of $10 per hour, consistent with institutional requirements and above the applicable local minimum wage. MLLM outputs were used only as editable annotation candidates. Disagreements were resolved through independent human review. Intended use. Meme3W is intended to support research on harmful-content understanding, safety evaluation, model analysis, and content-moderation review. Its structured annotations make the attacked target and the associated textual and visual evidence explicit.

1 Case 1 Input

Gold annotation Model prediction

Gold vs. Model Grounding

2 Think:

1. **Analyze Visuals & Text**: The meme consists of two panels. The top panel shows lines of white powder (crack cocaine) and a plastic bag, labeled 'upper'. The bottom panel shows a man with Down syndrome smiling, labeled 'downer'. 2. **Decode Semantics**: The meme uses a pun on the word 'downer', which refers to a depressant drug, but also refers to Down syndrome. By pairing a person with Down syndrome with the term 'downer', the meme mocks and degrades the individual based on their genetic condition. This is a common harmful trope that uses people with disabilities as the punchline of a joke. 3. **Distinguish 'Perpetrator/Symbol' vs. 'Victim/Target'**: The man in the bottom panel is the victim/target. He is being mocked for his physical appearance and condition. 4. **Determine Where the Attacked Target Appears**: The attacked target (a person with Down syndrome) is explicitly shown in the image. The text 'downer' is a coded reference to the condition but does not explicitly name the group 'people with Down syndrome'. 5. **Classify the meme and target category**: The meme is harmful as it mocks a person based on a health/physical condition. It maps to 'Health condition'. The target entity is 'people with Down syndrome'.

Structured Prediction

Gold Record

"Label": "Harmful", "Target Category":"Health condition", "Target Entity": "people with Down syndrome", "Text Mention": null, "Visual Region":[120, 540, 890, 999]

"Label": "Harmful", "Target Category":"Health condition", "Target Entity": "people with Down syndrome", "Text Mention": null, "Visual Region":[100, 546, 894, 1000]

Figure 10: Case 1 demonstrates correct image-grounded target identification.

1 Case 2 Input

Gold annotation Model prediction

Gold vs. Model Grounding

Gold Record

"Label": "Harmful", "Target Category":"Group background", "Target Entity": "Black people", "Text Mention": black, "Visual Region":[131, 66, 887, 995]

2 Think: 1. **Analyze Visuals & Text**: The image shows a Black woman wearing clown makeup (white face paint, red nose, red lips, black teardrop) and a top hat. The text says 'how you look when youre black and you celebrate the 4th of july'. 2. **Decode Semantics**: The clown makeup references the racist history of minstrelsy, portraying Black people as clowns. The meme mocks Black people by linking them to a degrading stereotype during a national celebration. 3. **Distinguish 'Perpetrator/Symbol' vs. 'Victim/Target'**: The woman in the image is Black and is being used to represent the target group. The clown makeup is a derogatory symbol used to mock her and the group she represents. The attacked target is Black people. 4. **Determine Where the Attacked Target Appears**: The target is explicitly named in the text ('black') and a representative of the target group is shown in the image. Thus, the location is 'both'. 5. **Classify the meme and target category**: The meme is harmful as it uses a racist stereotype to mock a group based on their race. This falls under 'group background'. The target entity is 'Black people'. The text span is 'black'. The image target is the woman.

Structured Prediction "Label": "Harmful", "Target Category":"Group background", "Target Entity": "Black people", "Text Mention": black, "Visual Region":[135, 70, 890, 1000]

Figure 11: Case 2 demonstrates correct joint text–image target identification.

1 Case 3

Gold annotation Model prediction

Input

Gold vs. Model Grounding

2 Think: "1. **Analyze Visuals & Text**: The image shows a man and a woman in an intimate setting. The text says “a real man... loads the dishwasher every night!!”. 2. **Decode Semantics**: The phrase “a real man” with “loads the dishwasher” implies a sexist stereotype that defines masculinity through domestic roles, degrading men by reducing their identity to household labor. 3. **Distinguish 'Perpetrator/Symbol' vs. 'Victim/Target'**: The man in the image is the subject of the text's claim. He is being labeled as a 'real man' based on a sexist standard. The meme attacks men by reducing their identity to a domestic role. The woman is a visual element in the scene but is not the target of the attack; the attack is directed at the concept of masculinity. 4. **Determine Where the Attacked Target Appears**: The target (men) is explicitly referred to in the text ('a real man') and visually represented by the man in the image. 5. **Classify the meme and target category**: The meme is harmful because it promotes a sexist stereotype and degrades men based on their gender. It fits the 'gender' category. The target entity is 'men'. The text span is 'a real man'. The image target is the man.

Structured Prediction

Gold Record

"Label": "Harmful", "Target Category":"Gender", "Target Entity": "women", "Text Mention": null, "Visual Region":[319, 43, 684, 875]

"Label": "Harmful", "Target Category":"Gender", "Target Entity": "men", "Text Mention": null, "Visual Region":[270, 220, 992, 992]

Figure 12: Case 3 shows target-entity and visual-grounding errors despite correct harmfulness and target-category predictions.

1 Case 4

Gold annotation Model prediction

Input

Gold vs. Model Grounding

Gold Record

"Label": "Harmful", "Target Category":"Gender", "Target Entity": "women", "Text Mention": "girl", "Visual Region":[]

2 Think: "1. **Analyze Visuals & Text**: The meme has two panels. The top shows a Black man smiling with the text “Finally getting to date a girl with thick thighs.” The bottom shows him looking shocked with the text “But she won't crush your head in between them.” 2. **Decode Semantics**: The meme uses a harmful stereotype that hypersexualizes Black men and associates them with sexual violence. The phrase “crush your head in between them” is a coded reference to assault, reinforcing a racial stereotype and making light of such violence. 3. **Distinguish 'Perpetrator/Symbol' vs. 'Victim/Target'**: The Black man in the image is the visual vehicle for the stereotype. The meme attacks Black men as a group by associating them with this violent behavior. The 'girl' mentioned is the hypothetical victim of the implied violence. 4. **Determine Where the Attacked Target Appears**: The attacked target (Black men) is explicitly represented by the man in the image. The text does not explicitly name the racial group. 5. **Classify the meme and target category**: The meme is harmful as it promotes a violent stereotype against a group based on race. This falls under 'group background'. The target entity is 'Black men'.

Structured Prediction "Label": "Harmful", "Target Category":"Group background", "Target Entity": "Black men", "Text Mention": null, "Visual Region":[500, 0, 990, 495], [500, 500, 990, 990]

Figure 13: Case 4 shows cascading target-identification errors despite a correct harmfulness prediction.

HarmTrace: Anchor-Calibrated Decoupled Optimization for Fine-Grained Target Identification in Harmful Memes

Analyze & explore related research in Conceptio

Related documents