Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection Ahmad Dawar Hakimi1,2,3 1
Lea Hirlimann1
Isabelle Augenstein2
Hinrich Schütze1,3
Center for Information and Language Processing, LMU Munich, Germany 2 Department of Computer Science, University of Copenhagen, Denmark 3 Munich Center for Machine Learning, Germany [email protected]
arXiv:2604.13899v1 [cs.CL] 15 Apr 2026
Abstract Instruction-tuned LLMs can annotate thousands of instances from a short prompt at negligible cost. This raises two questions for active learning (AL): can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be labelled at once? We investigate both questions on a new dataset of 277,902 German political TikTok comments (25,974 LLM-labelled, 5,000 human-annotated), comparing seven annotation strategies across four encoders to detect anti-immigrant hostility. A classifier trained on 25,974 GPT-5.2 labels ($43) achieves comparable F1-Macro to one trained on 3,800 human annotations ($316). Active learning offers little advantage over random sampling in our preenriched pool and delivers lower F1 than full LLM annotation at the same cost. However, comparable aggregate F1 masks a systematic difference in error structure: LLM-trained classifiers over-predict the positive class relative to the human gold standard. This divergence concentrates in topically ambiguous discussions where the distinction between anti-immigrant hostility and policy critique is most subtle, suggesting that annotation strategy should be guided not by aggregate F1 alone but by the error profile acceptable for the target application.
1
Introduction
Annotated data remains a bottleneck for supervised learning in NLP. A recent community survey finds that 94% of practitioners still consider annotation a limiting factor, and that active learning (AL) remains widely used (Romberg et al., 2026). Recent work has begun placing LLMs inside the AL loop as annotators (Zhang et al., 2023; Kholodna et al., 2024), but these studies do not compare human and LLM labels under matched conditions, nor examine how annotation source affects classifier behaviour. When an instruction-tuned LLM can label an entire corpus from a short prompt at a fraction of human annotation cost, the comparison that matters is no
Figure 1: Experimental framework. Human and LLM (Llama-3.3-70B prefiltering + GPT-5.2 classification) annotations feed into the same active learning loop, evaluated across seven conditions, four German encoder models, and 10 seeds on a party-stratified test set.
longer AL versus human annotation but AL versus LLM annotation at scale. We test whether AL’s cost premium survives that comparison. We investigate this question on detecting antiimmigrant hostility in German TikTok comments collected from the official accounts of six major political parties. The task is challenging to annotate: it requires differentiating hostility directed at immigrant groups from criticism of immigration as a policy issue, a culturally specific distinction where annotation source is likely to matter. It also represents a common scenario in which a lightweight supervised model must be trained from limited labels for efficient inference at scale. Our experimental design (Figure 1) compares seven labelling conditions that vary annotation source (human vs. LLM), sampling strategy (active learning vs. random vs. full pool), and label volume (530 to 25,974 instances) across four encoder models and 10 random seeds. The labelling pool comprises 25,974 immigration-relevant comments constructed via a two-stage LLM pipeline, of which 5,000 are independently annotated by six human crowdworkers, yielding 1,200 gold evaluation items and 3,800 training labels. We organise the paper around three research questions:
RQ1: Does human-in-the-loop active learning remain cost-competitive with LLM annotation at scale, and does it provide any performance advantage that justifies its cost premium? RQ2: What characteristic errors and confidence patterns emerge in classifiers trained on LLMversus human-labelled data? RQ3: Where in the discourse space do human and LLM annotations diverge, and what does this reveal about how each source operationalises the annotation boundary? Our central finding is that LLM annotation at scale achieves comparable F1-Macro to human annotation at one-seventh the cost, but with systematic differences in error structure. We release a new dataset of 277,902 German political TikTok comments (25,974 LLM-labelled and 5,000 humanannotated) with experimental code.
2
Related Work
2.1
Active Learning and LLMs as Annotators
Active learning (AL) reduces annotation cost by selecting only the most informative instances for human labelling (Settles, 2009), but practical adoption remains limited by unreliable performance gains, hyperparameter sensitivity, poor tooling, and difficulty obtaining reliable uncertainty estimates for acquisition functions (Lowell et al., 2019; Romberg et al., 2026; Schröder and Niekler, 2020). Instruction-tuned LLMs offer a complementary cost reduction: zero-shot ChatGPT achieves higher annotation agreement than crowd-workers at roughly one-thirtieth the cost (Gilardi et al., 2023). These findings extend to open-source models and other domains (Alizadeh et al., 2025; Törnberg, 2025), including German migration discourse (Kostikova et al., 2024). Recent work places LLMs inside the AL loop as annotators (Zhang et al., 2023; He et al., 2024; Xia et al., 2025), though Pangakis et al. (2023) caution that quality is task- and prompt-sensitive and requires validation against human gold labels. Yet most studies evaluate annotation agreement rather than downstream classifier performance or error profiles (Heseltine and Clemm von Hohenberg, 2024; Fillies et al., 2025); none compare human versus LLM labels across matched AL, random, and full-pool conditions with error-structure analysis. This is the gap we address.
2.2
Hate Speech Detection and Annotation
Hate speech detection is well studied (Schmidt and Wiegand, 2017; Fortuna and Nunes, 2018), but annotation difficulty increases sharply when hostility toward a group must be distinguished from criticism of a policy (Waseem and Hovy, 2016) or when bias is implied rather than explicit (Sap et al., 2020), the central challenge in our task. This difficulty affects LLMs as well as humans: Vallejo Vera and Driggers (2025) show that LLMs are sensitive to party cues when annotating political text. German-language resources remain limited: Ross et al. (2017) demonstrate low inter-annotator agreement even with detailed guidelines, and the most related dataset (Fillies et al., 2025) covers far-right ecosystems rather than mainstream party accounts. Annotator disagreement in subjective tasks is increasingly understood as signal rather than noise (Plank, 2022; Röttger et al., 2022; Davani et al., 2022; Wright and Augenstein, 2025), raising the question of whether LLMs can capture such human disagreement patterns (Ni et al., 2025).
3
Methodology
3.1
Dataset Collection and Preprocessing
We collected public comments from the official TikTok accounts of six major German political parties (CDU/CSU, Bündnis 90/Die Grünen, SPD, FDP, AfD, and Die Linke) via the TikTok Research API, covering January 2024 to September 2025, yielding 467,660 comments. We applied a four-stage preprocessing pipeline: (1) removal of duplicate comment IDs; (2) content filtering to remove non-informative comments (emoji-only, empty, or single-token sequences such as mentions, URLs, hashtags, and numbers); (3) exact text deduplication within and across party accounts; and (4) language identification via GlotLID (Kargaran et al., 2023) to retain only German-language comments. This yielded a filtered corpus of 277,902 German-language comments (median 14 words per comment); full perstep counts are provided in Table 13 (App. D.1). This corpus forms the input to the LLM-based prefiltering stage described in Section 3.3. 3.2
Label Space Definition
We define a binary classification task. A comment is labelled ANTI - IMMIGRANT if it expresses hostility, fear, or opposition toward immigrants, refugees, or asylum seekers as a group, including xenophobic
language, stereotypes, deportation rhetoric, or attribution of social problems to these groups. All other comments are labelled NOT ANTI - IMMIGRANT. Two-Question Annotation Framework. For human annotation, we operationalise this label through two sequential yes/no questions rather than a direct label assignment: • Q1: Does this comment refer to immigrants, refugees, asylum seekers, foreigners, or immigration policy? • Q2: Are these groups portrayed negatively through hostility, stereotypes, deportation rhetoric, or attribution of social or economic problems? The final label is derived programmatically: ANTI - IMMIGRANT ⇔ Q1 = YES ∧ Q2 = YES; otherwise NOT ANTI - IMMIGRANT. This decomposition embeds the core decision logic directly into the annotation interface and enables diagnostic tracing of disagreements to the reference step (Q1) or the valence step (Q2). 3.3
LLM-Based Pool Construction and Annotation
We construct the labelling pool via a two-stage LLM pipeline. Stage 1: Topic Prefiltering. We applied Llama-3.3-70B-Instruct (Team, 2024) to all 277,902 comments using a binary filtering prompt asking whether each comment discusses immigration, migrants, refugees, asylum seekers, or related policies (full prompt and configuration: App. A.1). This reduced the corpus to 25,974 comments (9.3% of the filtered corpus). Stage 2: Annotation. We applied GPT-5.2 via the OpenAI Batch API to annotate each prefiltered comment as ANTI - IMMIGRANT or NOT ANTI IMMIGRANT using the label definitions from Section 3.2 (full prompt and configuration: App. A.2. The model was instructed to default to NOT ANTI IMMIGRANT when uncertain. GPT-5.2 labelled 7,847 comments (30.2%) as ANTI - IMMIGRANT and 18,127 (69.8%) as NOT ANTI - IMMIGRANT. Table 1 shows the distribution across party accounts. Annotation Corpus. From the labelled pool, we sampled 5,000 comments for the human annotation study (Section 3.4) using party-stratified sampling at a target ratio of 40% ANTI - IMMIGRANT to 60%
Party
Total
Anti
%
AfD CDU/CSU SPD Die Linke Grüne FDP
9,379 7,316 4,324 3,311 1,244 400
2,691 2,508 1,430 831 246 141
28.7 34.3 33.1 25.1 19.8 35.2
Total
25,974
7,847
30.2
Table 1: GPT-5.2 label distribution after annotation of the Llama-prefiltered pool, by party account. Phase
Overlap
Items
Use
Test Pilot Main (triple) Main (single)
6-way 6-way 3-way 1-way
50 200 950 3,800
Eval Eval Eval Train
Total eval Total train
1,200 3,800
Table 2: Annotation study structure. Multiply-annotated items form the gold evaluation set; single-annotated items form the active learning training pool.
NOT ANTI - IMMIGRANT , a moderate enrichment over the natural 30.2% base rate to ensure sufficient positive instances for active learning. This same set is independently annotated by crowdworkers, enabling direct comparison between LLM and human labels under matched conditions. The LLM receives a single prompt rather than the two-question decomposition, targeting the same label space.
3.4
Human Annotation Study
We recruited annotators through Prolific, restricting participation to native or highly proficient German speakers. Eight were recruited; six were retained beyond the qualification round. Annotations were collected via Argilla,1 hosted on Hugging Face Spaces, using the two-question interface described in Section 3.2. The study comprised three phases with decreasing annotation overlap (Table 2), yielding 1,200 multiply-annotated items for the gold evaluation set and 3,800 single-annotated items for the active learning training pool. Details on qualification criteria, attention checks, and item assignment are provided in App. C. For the derived binary label, inter-annotator agreement is moderate: Krippendorff’s α = 0.49 on the pilot (6-way, 200 items) and α = 0.43 on the triple-annotated main items (3-way, 950 items). Agreement is higher on Q1 (immigra1
https://argilla.io/
Condition
Source
Sampling
ntrain
AL-H UMAN AL-LLM R ANDOM -H UMAN R ANDOM -LLM F ULL -H UMAN F ULL -LLM F ULL -LLM-26K
Human LLM Human LLM Human LLM LLM
Entropy / BALD Entropy / BALD Random Random Full pool Full pool Full pool
≤530 ≤530 ≤530 ≤530 3,800 3,800 25,974
Table 3: Seven experimental conditions. AL conditions use two acquisition functions (Section 3.6) and report maximum ntrain at round 10 (30 seed + 500 acquired).
tion reference, α = 0.57–0.59) than on Q2 (negative portrayal, α = 0.43–0.49), confirming that the primary source of disagreement is judging valence rather than identifying the topic, consistent with prior work on German hate speech annotation (Ross et al., 2017). Labels are aggregated via majority vote (full pairwise agreement in App. C). 3.5
Experimental Conditions
We compare seven conditions spanning three design axes: annotation source (human vs. LLM), sampling strategy (AL vs. random vs. full pool), and label volume (Table 3). Within-pair comparisons isolate label source; AL-vs-Random comparisons isolate acquisition value; and F ULL -LLM26K tests whether scale compensates for label quality. All conditions share the same classifier, training recipe, and evaluation set (Section 3.7); costs are reported in Section 3.8. 3.6
Active Learning Design
We implement pool-based AL for AL-H UMAN and AL-LLM. Both conditions start from identical initial states: the same unlabelled pool and a class-balanced, party-stratified seed set of 30 comments (5 per party), following Fairstein et al. (2024). Since labels are collected prior to experiments, we simulate the AL loop by revealing preexisting labels when queried. In each of 10 rounds, the top 50 most informative instances are selected, yielding a maximum of 530 labelled examples at round 10 (30 seed + 500 acquired). We evaluate two acquisition strategies: Entropy, selecting instances with the highest predictive entropy over the softmax output; and BALD (Houlsby et al., 2011), selecting instances maximising mutual information between predictions and model parameters, estimated via MC Dropout (Gal and Ghahramani, 2016) with 10 forward passes. After each round, the classifier is retrained from scratch and evaluated
on the held-out test set; learning curves report mean and standard deviation across 10 random seeds. 3.7
Model, Training, and Evaluation
Models. We evaluate four pre-trained transformer encoders: german-bert,2 ModernGBERT,3 gbert-base,4 and xlm-r-base.5 The latter serves as a multilingual baseline. Each model is fine-tuned with a randomly initialised binary classification head. Comments are tokenised and truncated to 128 tokens. Training. We use AdamW (Loshchilov and Hutter, 2019) with learning rate 2×10−5 , batch size 16, and weight decay 10−2 , with linear warmup (10%) and decay. To address class imbalance we apply focal loss (Lin et al., 2017) with γ = 2.0 and perclass weights recomputed from class frequencies at each AL round. For full conditions and the AL seed round, training runs for up to 10 epochs with early stopping patience of 5 on development set macro-F1. For AL rounds 1+, patience is reduced to 3 to limit overfitting on small labelled sets. Evaluation. The 1,200-item gold evaluation set is partitioned into a development set (15%, n≈180, for early stopping) and a test set (85%, n≈1,020, for final reporting), both stratified by class. We report macro-F1 as the primary metric, supplemented by class-specific F1 for ANTI - IMMIGRANT (F1Anti). Across AL conditions we additionally report the Area under the Learning Curve (ALC), normalised by the number of AL rounds. All experiments are repeated with 10 random seeds;6 we report mean and standard deviation across seeds. 3.8
Cost-Effectiveness Analysis
We compare annotation costs across conditions (Table 4). All costs are reported in USD (£1 = $1.334). Human annotators were paid at Prolific’s “good rate” of £9/hour across three phases with 6 annotators, totalling $416 ($0.083 per label across 5,000 annotations).7 The full LLM pipeline (Llama prefiltering + GPT-5.2 annotation) cost $45.72 ($0.002 per label). Full token estimates and infrastructure details are provided in App. B. Cost-performance tradeoffs are analysed in Section 4.3. 2
https://hf.co/dbmdz/bert-base-german-uncased https://hf.co/LSX-UniWue/ModernGBERT_134M 4 https://hf.co/deepset/gbert-base 5 https://hf.co/FacebookAI/xlm-roberta-base 6 Seeds: 42, 123, 1997, 2026, 1337, 7, 183, 56, 919, 777. 7 Phase breakdown: £18 (test, 20 min) + £54 (pilot, 60 min) + £162 (main, 3 hours) = £234, plus £77.92 platform fee. 3
Cost
$/label
F ULL -H UMAN F ULL -LLM F ULL -LLM-26K AL-H UMAN AL-LLM
3,800 3,800 25,974 530 530
$316 $6.34 $43.30 $44 $0.88
$0.083 $0.002 $0.002 $0.083 $0.002
Full human study
5,000
$416
$0.083
Table 4: Annotation costs per condition (£1 = $1.334). Human costs are computed proportionally from the $416 total; LLM annotation costs from the $43.30 GPT-5.2 total. All LLM conditions additionally depend on a onetime Llama prefiltering cost of $2.42 (not included per condition). R ANDOM conditions share the cost structure of their AL counterparts and are omitted.
4
Results
We evaluate all seven conditions across four encoders and 10 seeds on the held-out test set. The relative ordering of annotation strategies is stable across all four architectures. German-specific models consistently outperform the multilingual baseline (xlm-r-base) by up to 0.035 F1-Macro points, a gap comparable in magnitude to the annotation strategy effects reported below. We focus on german_bert and ModernGBERT in the main text: german_bert achieves the highest absolute performance, while ModernGBERT shows a different pattern on the scale–quality tradeoff where F ULL LLM-26K exceeds F ULL -H UMAN. Results for all four models are provided in App. G.1. 4.1
Learning Curves and Annotation Source (RQ1)
Key finding. Human labels yield a small F1Macro advantage at matched volume. At scale (25,974 vs. 3,800 labels), F ULL -LLM-26K achieves comparable F1-Macro and higher F1-Anti than F ULL -H UMAN across models. AL-H UMAN (Entropy) outperforms all AL conditions on both metrics. AL-LLM (Entropy) tracks within 0.018–0.040 F1-Macro points at round 10. The gap is most pronounced in early rounds (ntrain < 200) and narrows as data accumulates, indicating that volume only partially compensates for label quality within the AL budget. F ULL H UMAN (0.725–0.740 F1-Macro across models) sets the performance ceiling at matched volume. F ULL -LLM falls 0.016–0.035 points below, isolating the label quality effect. F ULL -LLM-26K recovers this gap with approximately 7× more training examples (0.730–0.735 vs. 0.725–0.740). On
F1 Macro
0.8
F1 Anti-Immigrant
0.7 0.6
0.7
0.5
German BERT
ntrain
0.4
0.6
0.3
0.5
0.2 0.1 0.7
0.8
0.6
0.7
ModernGBERT
Condition
0.5
0.6
0.4 0.3
0.5
0.2 100
200
AL-Human (Entropy)
300
400
500
100
Number of Training Instances
AL-Human (BALD) Full-Human Full-LLM
200
AL-LLM (Entropy) Full-LLM-26K
300
400
500
AL-LLM (BALD)
Figure 2: Learning curves for german_bert (top) and ModernGBERT (bottom) on F1-Macro (left) and F1Anti (right). Shaded regions indicate ±1 SD across 10 seeds. Horizontal lines mark full-pool baselines. Results for gbert-base and xlm-r-base are in App. G.1.
F1-Anti, F ULL -LLM-26K achieves the highest scores across all four models (0.627–0.632), exceeding F ULL -H UMAN (0.573–0.603) by 0.025– 0.059 points, reflecting higher recall at the cost of more false positives (Section 4.4). 4.2
Acquisition Functions and Sampling Strategy
Key finding. Entropy-based acquisition outperforms BALD, but neither provides advantage over random sampling in a pre-enriched pool. AL-H UMAN (Entropy) > AL-H UMAN (BALD) and AL-LLM (Entropy) > AL-LLM (BALD) across all four models (Table 6). The gap is larger within LLM-labelled conditions (up to 0.035 on xlm-r-base), suggesting that BALD’s epistemic uncertainty estimates are more sensitive to systematic divergence between training labels and the evaluation standard. Entropy, operating on mean softmax outputs, is more robust to this effect. However, the AL–Random gap is more consequential than the Entropy–BALD gap. The ∆ rows in Table 6 show that AL (Entropy) provides near-zero advantage over random sampling for german_bert, gbert-base, and ModernGBERT. For xlm-r-base, AL performs worse than random in both regimes (∆ = −0.007 and −0.009), likely reflecting poor uncertainty calibration. This pattern is consistent with our pre-enriched pool construction (Section 3.3), which reduces the marginal benefit of uncertainty-based selection.
germ.
Mod.
gbert
F1-Macro 0.740 0.726 0.705 0.710 0.730 0.735 0.714 0.703 0.674 0.685 0.695 0.677 0.670 0.668
0.735 0.706 0.732 0.706 0.677 0.690 0.664
0.725 0.703 0.730 0.704 0.677 0.702 0.677
F ULL -H UMAN F ULL -LLM F ULL -LLM-26K AL-H UMAN (Ent.) AL-LLM (Ent.) R AND .-H UMAN R AND .-LLM
F1-Anti 0.603 0.573 0.587 0.585 0.628 0.632 0.554 0.537 0.539 0.532 0.527 0.492 0.517 0.531
0.599 0.587 0.627 0.545 0.533 0.529 0.520
0.594 0.586 0.627 0.562 0.528 0.548 0.531
Table 5: F1-Macro and F1-Anti (mean across 10 seeds) for all seven conditions. AL and Random conditions report round 10 values. Bold and underline indicate best and second-best per model. Acq.
germ. Mod. gbert xlm-r
AL-H UM . Entropy .668 .668 .659 .638 AL-H UM . BALD .658 .649 .652 .616 R AND .-H UM . — .658 .642 .643 .645 ∆ +.010 +.026 +.015 −.007 AL-LLM Entropy .642 .653 .643 .611 AL-LLM BALD .629 .632 .623 .576 R AND .-LLM — .641 .640 .628 .621 ∆ +.001 +.013 +.015 −.009
Table 6: Normalised ALC (mean, 10 seeds). ∆ = AL (Entropy) minus Random; positive values favour AL.
4.3
< $10
$10 100
Annotation Cost Performance Frontier
$100 400
0.74
F ULL -H UMAN F ULL -LLM F ULL -LLM-26K AL-H UMAN (Ent.) AL-LLM (Ent.) R AND .-H UMAN R AND .-LLM
Condition
0.76
xlm-r
Cost-Effectiveness and the Scale–Quality Tradeoff (RQ1)
Key finding. F ULL -LLM-26K matches F ULL H UMAN on F1-Macro at one-seventh the cost, but with a different error profile (Section 4.4). Figure 3 visualises the cost–performance frontier. At matched volume (3,800 comments each, $6 vs. $316), F ULL -LLM falls 0.016–0.035 F1Macro points below F ULL -H UMAN, isolating the pure label quality effect. F ULL -LLM-26K recovers this gap by providing 7× more training examples at $43 (0.730–0.735 vs. 0.725–0.740 across models). Label quality can be compensated with volume when LLM annotation is cheap enough to scale. The cost comparison between AL and LLM annotation is striking. Active learning acquires a maximum of 530 human labels at a proportional cost of 530/5,000 × $416 = $44, virtually identical to F ULL -LLM-26K ($43). At equal absolute cost, F ULL -LLM-26K provides 49× more train-
0.72
Macro-F1
Condition
0.70 0.68 0.66
AL-LLM (Ent) (530 labels) AL-LLM (BALD) (530 labels) Random-LLM (530 labels) Full-LLM (3,800 labels) Full-LLM-26K (26K labels)
0.64 0.62
0
50
100
150
200
Annotation budget ($)
250
AL-Human (BALD) (530 labels) Random-Human (530 labels) AL-Human (Ent) (530 labels) Full-Human (3,800 labels)
300
350
Figure 3: Annotation cost–performance frontier for german_bert (mean ± 1 SD across 10 seeds). Budget zones mark three regimes: under $10 (LLM-only), $10–100 (LLM at scale), and $100–400 (human annotation).
ing labels and higher final F1-Macro (Table 5). Active learning reduces the cost of human annotation relative to a full human study; it does not reduce annotation cost below LLM annotation at scale. 4.4
Error Profiles and Confidence Analysis (RQ2)
Key finding. Comparable aggregate F1 masks a systematic difference in error structure: LLMtrained classifiers over-predict the positive class relative to the human gold standard with higher confidence than human-trained models (Fig. 4). FP/FN asymmetry. Figure 4 (left) shows that F ULL -H UMAN produces approximately balanced FP and FN counts across seeds for german_bert (124 FPs, 87 FNs). F ULL -LLM-26K produces 243 FPs but only 21 FNs, a nearly 12:1 ratio. Whether this reflects genuine misclassification or a broader but defensible interpretation of anti-immigrant content is examined in Section 4.5. This asymmetry traces to how each annotation source operationalises the policy-versus-people distinction. The human two-question framework explicitly separates immigration reference (Q1) from negative group portrayal (Q2), requiring annotators to judge each step independently. GPT-5.2’s single holistic prompt evaluates both dimensions simultaneously, and appears to draw a broader boundary for what constitutes group-directed negativity in the presence of policy-adjacent framing. Training on 25,974 examples with this consistently broader boundary embeds the LLM’s decision rule into the classifier, producing high recall but low precision for the ANTI - IMMIGRANT class when evaluated against human labels. F ULL -LLM at matched volume shows an intermediate FP skew, confirming the effect scales with LLM-labelled data volume.
Confident Errors
German BERT
Random-LLM
Full-Human
1.0
Random-Human
Prediction confidence
AL-LLM (BALD) AL-LLM (Ent) AL-Human (BALD) AL-Human (Ent) Full-LLM-26K Full-LLM False Positives False Negatives
Full-Human 300
200
100
False Positives
0
100
200
False Negatives
300
0.9 0.8
Confidence > 0.9: 13/211
0.7 0.6 0.5
0
20
40
60
80
100
Error instances (ranked by confidence)
120
German BERT Full-LLM-26K
1.0
FP (n=124) FN (n=87) Confidence > 0.9
Prediction confidence
FP/FN Butterfly
FP (n=243) FN (n=21) Confidence > 0.9
0.9 0.8
Confidence > 0.9: 2/264
0.7 0.6 0.5
0
50
100
150
200
Error instances (ranked by confidence)
250
Figure 4: Left: FP/FN counts per condition for german_bert (mean ± 1 SD across 10 seeds; dots = individual seeds). Right: Errors ranked by prediction confidence for F ULL -H UMAN and F ULL -LLM-26K.
Confidence on errors. Figure 4 (right) shows that F ULL -LLM-26K’s FP confidence curve is shifted higher relative to F ULL -H UMAN. F ULL H UMAN-trained models show lower confidence on incorrect predictions, making them more amenable to threshold tuning and confidence-based quality assurance. In deployment scenarios where prediction confidence is used for human review triage, LLM-trained models would route fewer errors to review because the errors appear confident, a potentially costly failure mode for content moderation. Annotation boundaries. Cross-seed analysis reveals instances that are systematically misclassified across all seeds and conditions. Many reflect genuine annotation ambiguity rather than model failure. Consider: Wir nehmen die Menschen, wie sie sind und nicht wie sie sein sollten. Leitkultur (‘We take people as they are and not as they should be. Leitkultur
)
The first clause paraphrases a well-known quote by Konrad Adenauer (“Nehmen Sie die Menschen, wie sie sind, andere gibt’s nicht” — ‘Take people as they are, there are no others’), juxtaposed with Leitkultur, a politically charged concept implying cultural assimilation, and a clown emoji signalling sarcasm. The comment is genuinely ambiguous: if the emoji targets Leitkultur, the author is mocking the demand for cultural assimilation (NOT ANTI - IMMIGRANT); if it targets the acceptance statement, the author is mocking tolerance toward immigrants as naive (ANTI - IMMIGRANT). An LLM with knowledge of German political rhetoric may recognise the Adenauer reference and interpret the comment as a political citation being ironically reframed, while crowdworkers without this background are more likely to rely on surface-level cues. This illustrates how annotation source can
affect not only the rate but also the type of labelling decisions on culturally embedded content. 4.5
Topical Clustering and Annotation Disagreement (RQ3)
Key finding. Human–LLM annotation disagreement concentrates in themes where the distinction between anti-immigrant hostility and antiimmigration policy critique is most ambiguous. We embed all 25,974 prefiltered comments using paraphrase-multilingual-MiniLM-L12-v2 and apply BERTopic (Grootendorst, 2022), yielding 22 named topics after merging four semantically overlapping pairs (clustering details and tSNE visualization in App. E). Agreement ranges from 45% (Border Control & Grenzschutz) to 93% (Pro-AfD Mobilization; Table 7), with the lowest agreement in topics where policy-adjacent negativity is most prevalent. Islam & Islamization shows moderate agreement (71%) but the largest label-rate gap: GPT-5.2 labels 82% as ANTI - IMMIGRANT versus 9% by humans. The LLM ANTI - IMMIGRANT rate exceeds the human rate for all 22 topics without exception, confirming systematic directional divergence. This propagates into classifier error rates: FP increases under LLM annotation are largest in Border Control & Grenzschutz, Criminal Violence & Public Safety, and Welfare State & Economic Competition, topics where negative framing targets policies or institutions rather than immigrant groups directly (pertopic error rates in App. G.3). Two mechanisms of disagreement. Manual inspection of all disagreement cases in the two most informative clusters reveals two distinct mechanisms (full inventory in App. F). In Border Control & Grenzschutz (45% agreement, 18 disagreements), all 18 cases target political actors or policy decisions rather than immi-
Topic
n Agr. H% L%
Border Control & Grenzschutz 225 45% Welfare & Econ. Competition 1,476 59% Migration Policy & Terminology 1,050 69% Islam & Islamization 195 71%
6 5 8 9
69 52 54 82
Pro-AfD Mobilization Racism & Fascism Discourse
1 1
7 11
2,688 93% 309 93%
Table 7: Selected discourse topics with lowest and highest human–LLM agreement. Agr. = label agreement rate; H% and L% = human and LLM ANTI - IMMIGRANT rates. Islam & Islamization has moderate agreement but the largest label-rate gap (+0.72). Full table in App. 14.
grant groups e.g. “Wer hat die Grenzen geöffnet?” (‘Who opened the borders?’) and “wann kommen die Grenzkontrollen?” (‘When are the border controls coming?’). Human annotators classify these as NOT ANTI - IMMIGRANT because Q2 (negative portrayal of immigrants) is not satisfied; GPT-5.2 treats immigration-adjacent vocabulary combined with negative framing as sufficient evidence for ANTI - IMMIGRANT, conflating policy critique with group-directed hostility. In Islam & Islamization (71% agreement, largest gap at +0.72, 7 disagreements), a different pattern emerges. Disagreement comments target Muslims as a religious group e.g. “der Islam gehört nicht zu Deutschland” (‘Islam does not belong in Germany’) without explicit immigration framing. The distinguishing factor is the Q1 gateway: comments framing hostility in purely religious terms may receive Q1 = NO, producing an automatic NOT ANTI - IMMIGRANT regardless of the comment’s hostility. GPT-5.2 treats antiMuslim and anti-immigrant hostility as overlapping categories. Whether this broader scope is a labelling error or a defensible interpretation depends on whether the framework intends to capture hostility toward immigration-associated groups or only toward groups explicitly framed as immigrants.
5
Discussion
The case for LLM annotation at scale. For research groups operating under tight budgets, common in computational social science and lowresource NLP, large-scale LLM annotation can substitute for human annotation on aggregate performance metrics without requiring crowdsourcing infrastructure or extended recruitment timelines. LLM annotation also enables broader coverage of the target distribution, which may contribute to the strong F1-Anti performance of F ULL -LLM-
26K (Table 5). Beyond cost, our human annotation pipeline required several weeks across setup, recruitment, qualification, pilot, and main phases; GPT-5.2 annotated all 25,974 comments within hours. This speed difference enables rapid iteration over label definitions and prompt designs before committing to a full annotation run. When human annotation remains essential. LLM-trained classifiers systematically over-predict the positive class with higher confidence (Section 4.4). We do not claim human labels are uniformly more accurate. The Adenauer example and the Islam & Islamization cluster suggest cases where the LLM may capture hostility that the twoquestion framework excludes. However, where false positives carry real costs, the FP-skewed error profile may be unacceptable regardless of aggregate F1. Which source better captures the intended category depends on the research question and the definition of harm being operationalised. LLM annotation does carry an additional ethical advantage: it eliminates human exposure to harmful content, a non-trivial consideration given documented psychological costs (Steiger et al., 2021). Active learning in pre-enriched pools. The near-zero AL advantage over random sampling reflects a successful pool construction strategy rather than a failure of AL. Practitioners should treat pool construction as a first-class design decision: investment here yields more reliable returns than investment in acquisition function sophistication.
6
Conclusion
We compared human and LLM annotation across seven conditions, four encoders, and 10 seeds for detecting anti-immigrant hostility in German political TikTok comments. LLM annotation at scale achieves comparable F1-Macro to human annotation at approximately one-seventh the cost, while AL with human labels delivers lower F1 at essentially the same price and provides little advantage over random sampling in our pre-enriched pool. However, comparable aggregate performance masks systematic differences in error structure that concentrate in topically ambiguous discourse, where the distinction between group-directed hostility and policy critique is most subtle. The choice between human and LLM annotation should be guided not by aggregate F1 alone but by the error profile acceptable for the target application.
7
Limitations
Pre-enriched annotation pool. Our annotation pool was constructed via LLM prefiltering, which concentrates the pool around immigration-relevant content and reduces the marginal benefit of uncertainty-based acquisition. Our finding that AL provides little advantage over random sampling (Section 4.2) should be interpreted in this context; results may differ with an unfiltered corpus. Crowdsourced rather than expert annotation. Human annotations were collected from Prolific crowdworkers, not domain experts in hate speech research. Crowdworker judgements may differ systematically from those of trained experts, particularly for borderline cases involving implicit hostility or culturally specific references. Expert annotations might shift the decision boundary and alter the relative performance of human versus LLM conditions. Evaluation against human gold labels. All conditions are evaluated against the human gold standard, which systematically disadvantages LLMtrained models whose annotation boundary diverges from the human one. Instances where GPT5.2 and the downstream classifier agree on ANTI IMMIGRANT but the human majority votes NOT ANTI - IMMIGRANT are counted as false positives regardless of which label is more appropriate. An expert adjudication study on contested instances would help disentangle genuine model errors from evaluation bias. Single task and domain. All experiments are conducted on a single binary classification task in a single language (German) on a single platform (TikTok). Findings may not generalise to other subjective annotation tasks, languages, or platforms. Single LLM annotator and prompt design. LLM labels are produced by a single model (GPT5.2) with a single classification prompt. We tested multiple prompt formulations during development, varying label definitions and instruction specificity, before selecting the current design based on agreement with a pilot subset of human labels. Nevertheless, the systematic annotation boundary divergence we observe (Section 4.5) may be specific to this model and prompt design rather than a general property of LLM annotation. Moreover, our human annotators use a two-question decomposition whereas the LLM receives a single holistic prompt.
This asymmetry reflects a practical constraint: pilot testing showed that annotators who received a holistic label definition did not consistently follow the guidelines. Part of the observed FP asymmetry may therefore stem from this difference in annotation interface rather than from a fundamental difference between human and LLM judgment. Simulated active learning. We simulate the AL loop by pre-collecting both human and LLM labels and revealing the appropriate label when an instance is queried, rather than collecting annotations on-demand. In a true online setting, annotators may calibrate their understanding over rounds, which could affect the human–LLM gap in ways our simulation does not capture.
8
Ethical Considerations
Annotator exposure and compensation. Our annotation task required crowdworkers to read and classify comments containing xenophobic language, stereotypes, and dehumanising rhetoric. Annotators were informed about the nature of the content before accepting the task, could pause or withdraw at any time without penalty, and were compensated at Prolific’s recommended “good rate” of £9/hour regardless of completion speed (full cost breakdown in Section 3.8 and App. B). We did not use rejection-based payment models. Nonetheless, sustained exposure to hostile content carries documented psychological risks (Steiger et al., 2021), and reducing the need for human annotation of harmful content is an explicit motivation for this work. Data collection and privacy. Comments were collected via the TikTok Research API under its terms of service for academic research. All comments are publicly posted content. We do not collect or store user profile information beyond the comment text and the party account it appeared under. Usernames and user IDs are stripped from the released dataset. The annotation study was conducted with informed consent from all annotators. Use of generative tools. We used LLM-based tools to assist with literature retrieval in the related work section and as coding assistants during experiment implementation. All generated content was verified by the authors.
References Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohammadmasiha Zahedivafa, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2025. Open-source llms for text annotation: a practical guide for model setting and finetuning. J. Comput. Soc. Sci., 8(1):17. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Trans. Assoc. Comput. Linguistics, 10:92– 110. Yaron Fairstein, Oren Kalinsky, Zohar Karnin, Guy Kushilevitz, Alexander Libov, and Sofia Tolmach. 2024. Class balancing for efficient active learning in imbalanced datasets. In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 77–86. Jan Fillies, Esther Theisen, Michael Hoffmann, Robert Jung, Elena Jung, Nele Fischer, and Adrian Paschke. 2025. A novel german tiktok hate speech dataset: far-right comments against politicians, women, and others. Discover Data, 3(1):4. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Comput. Surv., 51(4):85:1–85:30. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, JMLR Workshop and Conference Proceedings, pages 1050–1059. JMLR.org. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for textannotation tasks. CoRR, abs/2303.15056. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based TF-IDF procedure. CoRR, abs/2203.05794. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. Annollm: Making large language models to be better crowdsourced annotators. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 165–190. Association for Computational Linguistics. Michael Heseltine and Bernhard Clemm von Hohenberg. 2024. Large language models as a substitute for human experts in annotating political text. Research & Politics, 11(1):20531680241236239. Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Máté Lengyel. 2011. Bayesian active learning
for classification and preference learning. CoRR, abs/1112.5745. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. 2023. Glotlid: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Findings of ACL, pages 6155–6218. Association for Computational Linguistics. Nataliia Kholodna, Sahib Julka, Mohammad Khodadadi, Muhammed Nurullah Gumus, and Michael Granitzer. 2024. Llms in the loop: Leveraging large language model annotations for active learning in low-resource languages. In Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track - European Conference, ECML PKDD 2024, Vilnius, Lithuania, September 9-13, 2024, Proceedings, Part X, Lecture Notes in Computer Science, pages 397– 412. Springer. Aida Kostikova, Dominik Beese, Benjamin Paassen, Ole Pütz, Gregor Wiedemann, and Steffen Eger. 2024. Fine-grained detection of solidarity for women and migrants in 155 years of german parliamentary debates. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 5884–5907. Association for Computational Linguistics. Woosuk Kwon. 2025. vLLM: An Efficient Inference Engine for Large Language Models. Ph.D. thesis, UC Berkeley. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. David Lowell, Zachary C. Lipton, and Byron C. Wallace. 2019. Practical obstacles to deploying active learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 21– 30. Association for Computational Linguistics. Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Miserlis Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, and Elliott Ash. 2025. Can large language models capture human annotator disagreements? CoRR, abs/2506.19467. Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. Automated annotation with generative AI requires validation. CoRR, abs/2306.00176.
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 10671–10682. Association for Computational Linguistics.
Llama Team. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.
Julia Romberg, Christopher Schröder, Julius Gonsior, Katrin Tomanek, and Fredrik Olsson. 2026. Reassessing active learning adoption in contemporary NLP: A community survey. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2026 - Volume 1: Long Papers, Rabat, Morocco, March 24-29, 2026, pages 2621–2647. Association for Computational Linguistics.
Sebastián Vallejo Vera and Hunter Driggers. 2025. Llms as annotators: the effect of party cues on labelling decisions by large language models. Humanities and Social Sciences Communications, 12(1):1–11.
Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. CoRR, abs/1701.08118. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet B. Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 175–190. Association for Computational Linguistics. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5477–5490. Association for Computational Linguistics. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, SocialNLP@EACL 2017, Valencia, Spain, April 3, 2017, pages 1–10. Association for Computational Linguistics. Christopher Schröder and Andreas Niekler. 2020. A survey of active learning for text classification using deep neural networks. CoRR, abs/2008.07267. Burr Settles. 2009. Active learning literature survey. Miriah Steiger, Timir J. Bharucha, Sukrit Venkatagiri, Martin J. Riedl, and Matthew Lease. 2021. The psychological well-being of content moderators: The emotional labor of commercial moderation and avenues for improving support. In CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama, Japan, May 8-13, 2021, pages 341:1–341:14. ACM.
Petter Törnberg. 2025. Large language models outperform expert coders and supervised classifiers at annotating political social media messages. Social Science Computer Review, 43(6):1181–1195.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the Student Research Workshop, SRW@HLT-NAACL 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 88–93. The Association for Computational Linguistics. Dustin Wright and Isabelle Augenstein. 2025. Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift. PLoS One, 20(6):e0323064. Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Dernoncourt, Branislav Kveton, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Sungchul Kim, Zhengmian Hu, Yue Zhao, Nedim Lipka, Seunghyun Yoon, Ting-Hao Kenneth Huang, Zichao Wang, Puneet Mathur, Soumyabrata Pal, Koyel Mukherjee, Zhehao Zhang, Namyong Park, Thien Huu Nguyen, Jiebo Luo, Ryan A. Rossi, and Julian J. McAuley. 2025. From selection to generation: A survey of llm-based active learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 August 1, 2025, pages 14552–14569. Association for Computational Linguistics. Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. Llmaaa: Making large language models as active annotators. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, Findings of ACL, pages 13088–13103. Association for Computational Linguistics.
A
LLM Annotation Prompts
B
A.1
Stage 1: Topic Prefiltering Prompt
Llama-3.3-70B-Instruct (Stage 1). The model was run locally on four NVIDIA A100 80GB GPUs using vLLM (Kwon, 2025) with tensor parallelism across all four devices, bfloat16 precision, and a maximum sequence length of 512 tokens. Filtering all 277,902 comments completed in 1,982 seconds (0.55 wall-clock hours; 2.20 GPU-hours total) at a throughput of approximately 140 comments/second. At the representative Lambda Labs A100 spot rate of $1.10 per GPU-hour, the estimated infrastructure cost is $2.42 USD. Researchers without GPU access can replicate Stage 1 via the DeepInfra API (meta-llama/Llama-3.3-70B-Instruct) at approximately $5.28 for the same job ($0.10/1M input tokens, $0.40/1M output tokens; 41.7M input and 2.8M output tokens estimated).
Prompt A.1: Llama-3.3-70B-Instruct Topic Prefiltering Does this German TikTok comment discuss immigration, migrants, refugees, asylum seekers, foreigners, or related policies? Output JSON: {"label": "YES" or "NO"} Comment: {comment}
Figure 5: LLM prompts. A.1: Prefiltering prompt applied to all 277,902 comments (temperature = 0, JSON output).
A.2
Stage 2: Anti-Immigrant Annotation Prompt
Prompt A.2: GPT-5.2 Anti-Immigrant Annotation You are an expert in sentiment analysis and political discourse, specializing in German-language text. Your task is to classify TikTok comments posted under videos from German political parties. Classify the comment as either "ANTI-IMMIGRANT" or "NOT ANTI-IMMIGRANT". Definitions: ANTI-IMMIGRANT: Expresses negative sentiments, hostility, fear, or opposition towards immigrants, refugees, asylum seekers, or immigration policies. Includes xenophobic language, stereotypes, calls for stricter borders, deportation advocacy, or blaming immigrants for social/economic issues. Subtle implications count if they clearly lean negative. - NOT ANTI-IMMIGRANT: Neutral, positive towards immigrants/immigration, or unrelated to immigration altogether. Guidelines: - Analyze the comment in its original German. - Base classification solely on the comment’s content, not party context. - If ambiguous or sarcastic, default to "NOT ANTI-IMMIGRANT". - Base decisions only on visible text; no external inference. Output JSON: {"label": "ANTI-IMMIGRANT" or "NOT ANTI-IMMIGRANT"} Comment: {comment}
Figure 6: A.2: Classification prompt applied to 25,974 prefiltered comments via GPT-5.2 Batch API (temperature = 0, structured JSON output).
LLM Infrastructure Details
GPT-5.2 (Stage 2). Each request comprised the fixed classification prompt (≈240 tokens), comment text (≈20 tokens on average), and a JSON schema definition (≈50 tokens), yielding approximately 310 input tokens per call. Each response consisted of a structured JSON object with a binary label and brief explanation (≈80 output tokens). Across 25,974 comments this amounts to approximately 8.1M input tokens and 2.1M output tokens. At GPT-5.2 Batch API rates of $1.75 per 1M input tokens and $14.00 per 1M output tokens: $14.18 (input) + $29.12 (output) = $43.30 USD ($1.67 per 1,000 comments). Human Annotation Cost Breakdown. Annotators were compensated at £9/hour across three phases. Phase-level costs: £18 (test, 20 min per annotator × 6) + £54 (pilot, 60 min × 6) + £162 (main, 3 hours × 6) = £234 base compensation. With Prolific’s 33.3% academic platform fee (£77.92), the grand total was £312 ($416 USD). Per-condition costs are computed proportionally: F ULL -H UMAN uses 3,800 of 5,000 labels (3,800/5,000 × $416 = $316); AL-H UMAN uses a maximum of 530 (530/5,000 × $416 = $44).
C
Annotation Study Details
C.1
Annotator Demographics
The six annotators who completed all phases ranged in age from 19 to 60 (mean 40), with four identifying as female and two as male. All six were residents in Germany at the time of the study. Four
were native German speakers; the remaining two were highly proficient non-native speakers (one Japanese national, one Portuguese national), both listing German among their fluent languages on Prolific. C.2
Attention Checks
Instruction manipulation checks (IMCs) were interleaved throughout all phases at densities of approximately 10 per 100 items (test), 5 per 100 (pilot), and 3 per 100 (main). Each check instructs the annotator to select specific Q1 and Q2 responses, testing whether comment text is being read. A minimum of two checks was enforced per annotator per phase. In the qualification test, Annotator_7 failed to meet the ≥80% accuracy threshold on attention checks (2/3, 67%) and was excluded. Annotator_6 passed attention checks but showed extremely low inter-annotator agreement on the valence question (average pairwise κ = 0.13 on Q2) and was also excluded. The remaining six annotators achieved 83–100% accuracy on attention checks across all phases, with all six scoring 100% in the main round. C.3
Item Assignment
In the main round, 950 items received triple annotation for the gold evaluation set. Items were distributed via round-robin assignment over all 6 3 = 20 possible 3-annotator combinations to ensure balanced workload and annotator-pair coverage. The remaining 3,800 items received single annotation and were distributed evenly across annotators. C.4
Inter-Annotator Agreement
Tables 8 and 9 report agreement statistics for the pilot and main phases. Table 10 and Table 11 provide full pairwise Cohen’s κ matrices for the derived binary label. Metric Krippendorff’s α Cohen’s κ (avg) Cohen’s κ range
Q1
Q2
Label
0.574 0.580 .356–.807
0.493 0.488 .310–.758
0.494 0.490 .320–.741
Table 8: Inter-annotator agreement on the pilot phase (200 items, 6-way overlap).
Metric Krippendorff’s α Cohen’s κ (avg) Cohen’s κ range
Q1
Q2
Label
0.591 0.599 .384–.704
0.432 0.437 .221–.588
0.433 0.439 .221–.588
Table 9: Inter-annotator agreement on the main phase (950 triple-annotated items, 3-way overlap). Krippendorff’s α is used as Fleiss’ κ is not applicable for variable annotator subsets.
A1 A2 A3 A4 A5 A8
A1
A2
A3
A4
A5
A8
— .470 .515 .518 .386 .320
.470 — .738 .741 .545 .408
.515 .738 — .615 .551 .395
.518 .741 .615 — .425 .351
.386 .545 .551 .425 — .363
.320 .408 .395 .351 .363 —
Table 10: Pairwise Cohen’s κ for the derived binary label on the pilot phase (200 items, 6-way overlap).
A1 A2 A3 A4 A5 A8
A1
A2
A3
A4
A5
A8
— .535 .419 .272 .358 .357
.535 — .588 .471 .560 .480
.419 .588 — .536 .487 .521
.272 .471 .536 — .416 .359
.358 .560 .487 .416 — .221
.357 .480 .521 .359 .221 —
Table 11: Pairwise Cohen’s κ for the derived binary label on the main phase (950 triple-annotated items, 3way overlap). Each pair is computed over items where both annotators provided labels.
C.5
Per-Annotator Statistics
Table 12 reports per-annotator average κ (with all other annotators) and ANTI - IMMIGRANT label rates across the pilot and main phases. Annotator_1 shows a consistently higher positive label rate (34– 37%) than other annotators (16–29%), reflecting a more inclusive interpretation of what constitutes anti-immigrant content. Pilot
Main
Annotator
κ̄
ANTI%
κ̄
ANTI%
A1 A2 A3 A4 A5 A8
.442 .581 .563 .530 .454 .367
34.0 26.5 29.0 30.0 18.5 15.5
.388 .527 .510 .411 .408 .388
37.0 28.7 21.6 16.5 16.3 17.0
Table 12: Per-annotator average Cohen’s κ (on the derived binary label, averaged over all pairwise comparisons) and ANTI - IMMIGRANT label rate, by study phase.
C.6
Class-Conditional Agreement
Agreement is higher for the majority class across both phases. On the pilot (200 items, majority vote), items labelled NOT ANTI - IMMIGRANT (n=156) show 90.2% average agreement with 63% unanimity, while items labelled ANTI - IMMIGRANT (n=44) show 81.4% average agreement with 30% unanimity. On the main round (950 tripleannotated items), agreement is high overall: 96.3% for ANTI - IMMIGRANT items (89% unanimous) and 98.5% for NOT ANTI - IMMIGRANT items (96% unanimous), reflecting that the main round’s 3-way overlap yields higher raw agreement than the pilot’s 6-way overlap.
D
Dataset Statistics
D.1
Preprocessing Pipeline Step
Removed
Duplicate IDs Emoji-only Empty Mention-only URL-only Hashtag-only Number-only Laughter-only Duplicate text (within-party) Duplicate text (cross-party) Non-German (GlotLID)
20,390 86,883 1,989 1,273 447 406 783 270 28,543 4,068 44,706
Total removed
189,758
Remaining
277,902
Table 13: Per-step preprocessing counts applied to 467,660 raw TikTok comments collected via the TikTok Research API. GlotLID (Kargaran et al., 2023) is used for language identification to retain only Germanlanguage comments.
E
Discourse Topic Structure
Figure 7 and Table 14 provide the full topic structure underlying the error analysis in Section 4.5. Each cluster is labelled by its dominant keyword pattern; human and LLM label rates (H% and L%) are reported for the annotated subset.
Figure 7: t-SNE projection of all 25,974 immigration-relevant comments, clustered into 22 named discourse topics via BERTopic (Grootendorst, 2022). Colours denote topic membership; grey points indicate the noise category. Topics span explicit hostility (C10: Criminal Violence & Public Safety; C14: Islam & Islamization), coded political language (C05: Merkel/CDU & 2015 Legacy; C16: Female Politicians), policy-adjacent discourse (C02: Democracy & Governance; C06: Migration Policy & Terminology), and international framing (C11: Racism & Fascism Discourse; C18: Israel, Palestine & Middle East; C19: War & Militarism). The clusters showing the strongest FP-dominant bias under LLM conditions are discussed in Section 4.5.
F
Disagreement Examples by Cluster
This section provides the full inventory of human– LLM disagreement cases summarised in Section 4.5. Border Control & Grenzschutz (45% agreement, 18 disagreements). Comments fall into two groups. The majority (10/18) are rhetorical questions directed at politicians over the 2015 border opening: “Wer hat die Grenzen geöffnet?” (‘Who opened the borders?’), “Wer war denn 2015 der Auslöser?” (‘Who caused this in 2015?’), “merkel 2015. . . wir vergessen nicht” (‘Merkel 2015. . . we don’t forget’). The remainder (6/18) demand border controls without referencing any group: “wann kommen die Grenzkontrollen?” (‘When are the border controls coming?’), “Diese Grenze war vorher auch geschützt” (‘This border was protected before too’). In all 18 cases, the negativity targets political actors or policy decisions rather than immigrant groups. Islam & Islamization (71% agreement, largest gap at +0.72, 7 disagreements). Disagreement comments target Muslims as a religious group: “der Islam gehört nicht zu Deutschland” (‘Islam does not belong in Germany’), “muslim free bitte” (‘Muslim-free please’), “Die CDU hat Millionen Moslems reingeholt” (‘The CDU brought in millions of Muslims’). By contrast, the 17 cases where both sources agree on ANTI - IMMIGRANT typically combine religious references with explicit immigration vocabulary (“muslimische Flüchtlinge”, “Islamistenwanderung”) or with clearly dehumanising language.
ID
Topic
C0 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13 C14 C15 C16 C17 C18 C19 C20 C21
Germany in Decline & National Identity Pro-AfD Mobilization Democracy & Governance Critique Welfare State & Economic Competition Directed Personal Hostility Merkel/CDU & 2015 Legacy Migration Policy & Terminology Ukraine & Ukrainians Afghanistan, Syria & Turkey Left-Wing Party Critique (Grüne/Linke) Criminal Violence & Public Safety Racism & Fascism Discourse Refugees & Asylum Policy Border Control & Grenzschutz Islam & Islamization EU & European Politics Female Politicians (Faeser/Storch) Poll Numbers & Electoral Math Israel, Palestine & Middle East War & Militarism Rejection Rhetoric Brandmauer & Wall Metaphors
n
Agr.
H%
L%
5,489 2,688 1,915 1,476 1,188 1,072 1,050 876 356 345 333 309 308 225 195 172 162 156 126 123 112 100
71% 93% 87% 59% 74% 70% 69% 76% 76% 85% 70% 93% 75% 45% 71% 73% 62% 87% 80% 81% 70% 77%
3 1 1 5 2 2 8 10 8 3 8 1 9 6 9 3 2 0 5 7 1 2
32 7 16 52 25 33 54 50 51 28 57 11 65 69 82 22 25 12 20 43 27 20
Table 14: Human–LLM label agreement across all 22 BERTopic discourse topics. Agr. = label agreement rate on the overlapping human-annotated subset; H% and L% = human and LLM ANTI - IMMIGRANT label rates. Topics sorted by cluster ID.
G
Additional Results
G.1
All-Model Learning Curves
F1 Macro
0.8
F1 Anti-Immigrant 0.6
0.7
GBERT-base
0.5 0.4
0.6
0.3
0.5
0.2
0.4 0.7
XLM-R base
0.6 0.5 0.4 0.3 0.2 100
200
AL-Human (Entropy)
300
400
500
0.6 0.5 0.4 0.3 0.2 0.1 0.0
100
Number of Training Instances
AL-Human (BALD) Full-Human Full-LLM
200
300
AL-LLM (Entropy) Full-LLM-26K
400
500
AL-LLM (BALD)
Figure 8: Learning curves for gbert-base (top) and xlm-r-base (bottom) on F1-Macro (left) and F1-Anti (right). Shaded regions indicate ±1 SD across 10 seeds. The relative ordering of annotation strategies is consistent with german_bert and ModernGBERT (Figure 2). xlm-r-base shows higher variance and lower absolute performance, confirming that German-specific models are more robust on this task.
G.2
Confidence Distributions
Figure 9 shows the predicted P(A NTI I MMIGRANT) distributions split by error type for german_bert. FP distributions for LLM-trained conditions peak substantially higher than for human-trained conditions, confirming that LLM-trained classifiers assign high confidence to their false positives. The pattern is consistent across all four encoder models.
Confidence by Error Type german_bert Full-Human Full-LLM-26K
AL-Human (Ent)
P(ANTI-IMMIGRANT)
1.0 0.8 0.6 0.4 0.2 0.0 Correct
FP
Correct
FP
FN
Correct
FN
Correct
AL-Human (BALD)
FP
FN
Correct
FP
FN
FP
FN
Correct
FP
FN
AL-LLM (Ent)
AL-LLM (BALD)
P(ANTI-IMMIGRANT)
1.0 0.8 0.6 0.4 0.2 0.0 Figure 9: Confidence distributions for german_bert.
G.3
Topic Error Rates
Figure 10 shows per-topic FP–FN imbalance across all nine conditions for german_bert. The FPdominant pattern is consistent across annotation strategies. Notable exceptions are AL Human (Ent), which is the coldest column overall, and EU & European Politics, which is consistently FN-dominant under all human-trained conditions.
0.4
0.2
0.0
FP-dominant 0 FN-dominant
AL-Human (Ent)
0.6
0.2
0.4
Random-LLM
Full-LLM-26K
Random-Human
-0.20 -0.30 -0.30 -0.10 -0.30 -0.40 -0.22 -0.11 0.00 0.11 -0.22 -0.11 -0.17 -0.17 -0.03 0.00 -0.15 -0.09 -0.05 -0.06 0.00 0.02 -0.06 -0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 -0.02 0.01 0.06 0.08 -0.02 0.05 -0.11 -0.09 0.07 0.09 -0.09 0.02 -0.02 -0.07 0.12 0.20 0.00 0.05 -0.02 0.01 0.12 0.16 -0.03 0.09 -0.02 0.05 0.13 0.22 -0.01 0.10 0.00 0.00 0.17 0.00 0.00 0.08 -0.33 0.00 0.67 0.67 0.00 0.00 0.00 0.00 0.29 0.43 0.00 0.14 0.22 0.11 0.33 0.33 0.11 0.33 0.23 0.23 0.18 0.32 0.09 0.14 0.20 0.20 0.20 0.30 0.15 0.20 0.06 0.15 0.25 0.33 0.06 0.21 -0.15 0.31 0.46 0.54 0.15 0.31 0.25 0.25 0.62 0.44 0.06 0.25 0.22 0.20 0.45 0.62 0.20 0.47 0.50 0.50 0.50 0.50 0.50 0.50 0.67 0.33 0.67 0.67 0.67 0.67 AL-LLM (BALD)
0.20 0.22 0.22 0.05 0.00 0.14 0.12 0.33 0.05 0.17 0.17 0.42 0.00 0.43 0.11 0.18 0.10 0.27 0.62 0.56 0.33 0.00 0.33
AL-LLM (Ent)
0.20 0.11 0.11 0.03 0.00 0.14 0.14 0.22 0.10 0.17 0.14 0.17 0.00 0.14 0.22 0.23 0.30 0.38 0.38 0.31 0.33 0.50 0.67
AL-Human (BALD)
0.20 -0.17 -0.08 -0.02 0.00 0.00 0.00 0.07 0.15 0.02 -0.02 0.00 0.00 0.14 -0.11 0.14 0.10 0.17 0.08 0.25 0.25 0.00 0.33
Full-LLM
Islam & Islamization Left-Wing Party Critique (Grüne / Linke) Merkel/CDU & 2015 Legacy Pro-AfD Mobilization Racism & Fascism Discourse Poll Numbers & Electoral Math Democracy & Governance Critique Directed Personal Hostility Ukraine & Ukrainians Noise Germany in Decline & National Identity Rejection Rhetoric Israel / Palestine & Middle East Female Politicians EU & European Politics Specific Nationalities (Afghanistan / Syria / Turkey) Refugees & Asylum Policy Migration Policy & Terminology Border Control & Grenzschutz Criminal Violence & Public Safety Welfare State & Economic Competition Brandmauer & Wall Metaphors War & Militarism
Full-Human
cluster_topic
FP-FN Imbalance by Topic × Condition german_bert (positive = FP-dominant, negative = FN-dominant)
Figure 10: FP–FN imbalance by topic and condition for german_bert. Positive values indicate FP-dominant conditions; negative values indicate FN-dominant conditions. AL-Human (Ent) is consistently the least FP-skewed condition across topics.