arXiv:2606.06879v1 [cs.CL] 5 Jun 2026
An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection Carl Lochstampfor
Ayan Roy
Department of Computer Science Old Dominion University Norfolk, VA, USA [email protected]
Department of Computer Science Christopher Newport University Newport News, VA, USA [email protected]
Abstract—Our prior work introduced COVA, a synthetically generated multi-turn conversational smishing dataset of 3,201 labeled conversations, establishing baseline detection benchmarks across eight models. While XGBoost with TF-IDF features achieved the best performance (72.5% accuracy, 0.691 macro F1), transformer models underperformed—attributed to input truncation and insufficient training data. We present COVAX, an expanded dataset of 10,985 conversations spanning eight elder-targeted scam categories, produced by an improved generation pipeline addressing contamination, label mismatch, stagedirection bleed, and prompt-design failures from the first iteration. Retraining all classifiers on the expanded dataset yields the central finding of this work: Longformer now surpasses XGBoost on all evaluation metrics (79.71% accuracy, 0.7786 macro F1 vs. 78.43% and 0.7563), directly confirming that transformer models require larger conversational corpora to realize their contextual advantages. We additionally document a quality lifecycle including a 12.7× improvement in label correction rate (3.9% vs. 49.8%), an architectural intervention reducing virtualkidnapping artifact rates (67.1%→46.5%), and a per-scamtype outcome analysis showing scam categories modulate results in mechanism-consistent ways. A pre/post-cleanup sensitivity analysis confirms that dataset refinement recovers genuine labelrelevant signal across all three classifier architectures. Index Terms—Social Engineering, elder fraud, synthetic data, scam detection, multi-agent LLM, conversation classification, Smishing Detection, Longformer
I. I NTRODUCTION Conversational social engineering attacks targeting elderly individuals represent a growing and underserved threat in the cybersecurity landscape. Unlike single-message phishing, these attacks unfold across multiple dialogue turns, gradually building trust before extracting financial or personal information [2], [3]. Real-world data on such interactions is scarce due to privacy and ethical constraints, motivating the use of synthetic data generation. Earlier we introduced COVA [1] , the first publicly available multi-turn conversational smishing dataset, comprising 3,201 labeled conversations across eight elder-targeted scam categories. That work established eight baseline classifiers and identified a critical limitation: despite their superior contextual language understanding, transformer models (DistilBERT and Longformer) underperformed compared to XGBoost with TFIDF features. We attributed this gap to two factors: (1) input truncation discarding outcome-critical final turns in Distil-
BERT, and (2) insufficient training data for effective transformer fine-tuning. We explicitly predicted that “transformer models would likely benefit from larger training corpora (>5,000 conversations).” This paper presents COVA-X, the expansion of the COVA dataset to 10,985 conversations (3.43× increase),1 and reports the results of retraining all classifiers on the expanded dataset. The central finding confirms the first paper’s hypothesis: Longformer now surpasses XGBoost on all metrics for the first time, achieving 79.71% accuracy and 0.7786 macro F1 on the 3-class outcome prediction task. Beyond model performance improvements, this paper presents a comprehensive dataset-quality lifecycle analysis. This includes pipeline failures identified and corrected during dataset expansion, a three-point evidence chain evaluating an architectural intervention for reducing virtual-kidnapping artifacts, improvements in label correction rates, the quantitative contribution of each pipeline stage to dataset integrity, and a pre/post-cleanup classifier sensitivity analysis demonstrating that the cleanup improved label-relevant signal across three classifier architectures. We argue that transparent documentation of these iterative challenges constitutes a methodological contribution in its own right, offering a practical roadmap for future synthetic dataset construction efforts. The main contributions of this paper are: We expand the COVA dataset from 3,201 to 10,985 conversations across eight elder-targeted scam categories, with five victim and attacker profiles per type. • We document and quantify a full quality lifecycle pipeline including contamination scanning, stage-direction stripping, automated relabeling, and a three-role architectural intervention for virtual-kidnapping generation, achieving a 3.9% label correction rate versus 49.8% in the first iteration and a reduction in virtual-kidnapping artifact rates (67.1%→46.5%). •
1 Per-scam-type generation targets summed to approximately 11,400; the realized production count of 10,985 reflects normal generator retry overhead and minimum-turn-floor discards. Comprehensive dataset documentation, including version history, per-scam-type counts, and known limitations, is provided in the COVA-X data sheet, released as a companion artifact alongside the dataset.
We report a per-scam-type outcome distribution analysis showing that scam-type characteristics modulate outcomes in mechanism-consistent ways: virtual kidnapping has the highest successful_scam rate (33%) due to emotional-pressure framing; grandparent has the highest verification_attempt rate (63%) consistent with verification-call-back scripts; and medicare/bank have substantial quick_rejection populations reflecting public awareness. • We retrain all classifiers and confirm that Longformer surpasses XGBoost on the expanded dataset on every reported metric, validating the data-size hypothesis from the first paper. A pre/post-cleanup sensitivity analysis shows all three architectures improve on the cleaned dataset, evidence the cleanup affected genuine labelrelevant signal. • We characterize Qwen 2.5 14B generation behavior under sustained emotional pressure, documenting a five-instance pattern of explicit prompt-directive override across distinct directive types (structural-tag adoption, victimprofile compliance, address-term parameterization, nameidentity rules, and conversation-termination directives) that motivates the model-comparison study proposed in future work.
•
II. R ELATED W ORK A. Social Engineering Detection Detection of social engineering attacks has been studied across multiple modalities. In the SMS domain, Seo et al. [4] proposed on-device smishing classifiers resistant to text evasion, while Patra et al. [5] developed prediction models combining machine learning with text analysis. These approaches address single-message classification and do not capture multiturn dynamics. Derakhshan et al. [6] introduced ASsET, a detection system based on scam signatures for telephone-based attacks. Wood et al. [7] analyzed scam-baiting calls from YouTube, identifying scam stages and scripts at scale. Jain et al. [8] demonstrated that BERT outperforms TF-IDF for single-message smishing detection, a result that motivates investigating whether this advantage transfers to conversational detection with sufficient data. Recent work has extended detection beyond conventional text classification. Park et al. [16] present an LLM-based framework for voice-phishing detection that combines caseinformed transcript generation with domain-expert prompt engineering, achieving substantially improved classification accuracy on Korean voice-phishing data through synthetic augmentation. Their work establishes synthetic transcript generation as a viable strategy for addressing phishing data scarcity. Closely related, Li et al. [15] examine the adversarial dimension of LLM-generated phishing content, demonstrating that LLMs can be used by attackers to construct adversarial transcripts capable of evading ML-based detectors. Their results inform our own threat-model framing: the same
generation capability that enables defensive dataset construction also enables adversarial transcript synthesis. Gressel et al. [17] address the romance-scam category specifically, examining LLMs’ role in detecting emotionally manipulative scam dialogue, a category with particular relevance to our COVA-X romance subset (the largest single category at 1,934 conversations). For real-time deployment, the AI-in-the-Loop framework [18] proposes federated and privacy-preserving classification of scam dialogue from partial transcripts, which informs the incremental-classification direction proposed in Section VII. B. Synthetic Data and Multi-Agent Simulation Basta et al. [9] presented “Bot Wars,” a framework using competing LLMs as scam-baiters, validated against 179 hours of human interactions. Kumarage et al. [10] proposed SE-VSim, an LLM-agentic framework for simulating social engineering in multi-turn conversations with varying victim personality traits. Spokoyny et al. [11] developed CHATTERBOX for automated long-term engagement with online scammers. These frameworks share with our work a multiagent design, but differ in deployment posture: prior work focuses on adversarial engagement against active scammers, whereas COVA-X uses multi-agent simulation as a datasetconstruction methodology for downstream classifier training. A practical concern relevant to our prompt-engineering methodology is what we term a salience-bias effect in instruction-tuned LLMs: enumerating prohibited patterns as negative examples can increase rather than decrease their production rate, a phenomenon documented across both general instruction-following [19], [20] and negation-specific generation contexts [21]. We observe this effect directly in our virtual-kidnapping attacker prompt iterations (Section VI, Salience Bias in Prompt Engineering subsection). C. Gap in the Literature The COVA-X expansion addresses gaps identified in our prior work and the broader literature: (1) limited dataset scale constraining transformer evaluation; (2) single victim profiles limiting behavioral diversity; (3) undocumented pipeline quality lifecycles making reproducibility difficult; (4) limited characterization of generation-model capability limits encountered during synthetic dataset construction. Herrera et al. [12] highlight that older adults remain disproportionately vulnerable to AI-enhanced scams, reinforcing the importance of large-scale, multi-category elder fraud datasets. III. DATASET E XPANSION M ETHODOLOGY A. Generation Infrastructure The COVA-X expansion used the same core framework as the first dataset: two independent LLM agents (attacker and victim) exchanging turns using Qwen 2.5 14B via Ollama on local GPU hardware, with a three-role extension for virtual kidnapping (Section VI). All generation uses local inference rather than commercial API access, for three reasons: research freedom on simulated scam dialogue, zero marginal cost for
bulk generation, and reproducibility through Ollama-pinned model weights. Expansion runs were distributed across two workstations: Workstation A: NVIDIA RTX 4080 Super 16GB GDDR6X — primary generation and classifier-training machine; used for romance, government impersonation, medicare, and virtual kidnapping. • Workstation B: NVIDIA RTX 5060 Ti 16GB GDDR7 (Blackwell architecture) — secondary generation machine; used for grandparent, bank, lottery, and investment.
•
Both workstations ran Qwen 2.5 14B at 16k context. Sampling parameters were temperature 0.8 for attacker and victim turns, and temperature 0.7 with a 60-token cap for virtual-kidnapping three-role hostage utterances (the token cap enforces the 3–12 word brevity constraint for hostage voice). These settings produce stochastic variation across regeneration runs even with identical seeds; reproducibility of exact conversation content is therefore not guaranteed, but distributional characteristics (per-scam-type counts, outcome distribution, average turn lengths, quality-flag rates) are reproducible within sampling-noise margins. Conversations under 10 turns were discarded as insufficiently developed (skip rate approximately 2% overall, 25–30% for virtual kidnapping).
Romance Govt. Impersonation
14.1% 13.6%
Medicare Grandparent Bank Impersonation Virtual Kidnapping
17.6%
12.5%
8.9%
12.4%
Investment Lottery
8.9% 12.1%
Fig. 1. COVA-X per-scam-type distribution (n = 10,985). TABLE I COVA-X OUTCOME DISTRIBUTION BY SCAM TYPE ACROSS FIVE TERMINAL STATES : succ., part., ver., det., AND rej. Scam Type
succ.
part.
ver.
det.
rej.
Romance Government Imp. Medicare Virtual Kidnapping Grandparent Bank Impersonation Lottery Investment
254 202 254 443 63 173 196 86
921 468 376 659 296 377 477 705
481 551 268 110 868 569 290 146
267 294 159 107 144 29 10 5
11 30 433 11 5 212 2 33
B. Profile Expansion Cova [1] used a single victim profile per scam type. COVAX introduces five profiles per type: two outcome-pinned profiles (Carol pinned to quick_rejection, George pinned to scam_detected) and three general-pool profiles covering the remaining outcomes. This design provides sufficient behavioral variety while maintaining enough conversations per profile for classifier signal strength. Profile-pinning is implemented through outcome-target dictionaries in the generator, with hard-stop directives written into the system prompt for the pinned outcome targets. C. Dataset Targets and Distribution Per-scam-type generation targets summed to approximately 11,400 conversations across the eight scam categories. The realized production output, after normal generator retry overhead and minimum-turn-floor discards, is 10,985 conversations. Fig 1 shows the per-scam-type production counts; the COVAX data sheet documents the per-scam-type generation targets and the production-vs-target reconciliation in detail. D. Prompt Engineering Improvements Several prompt-level fixes were introduced for the expansion based on failures observed in the first dataset: Name Identity Lock: A zero-tolerance name rule added to all attacker and victim prompts, preventing character name mixing across scam types. • ABSOLUTE OUTPUT RULE: Injected at runtime into victim prompts to suppress stage-direction bleed. Most effective for low-emotional-pressure scam types; partially •
effective for virtual kidnapping and romance, where additional architectural and post-processing measures were needed (Section IV). • Grandparent-Opener Contamination Fix: Attacker prompts for government impersonation, medicare, and virtual kidnapping were rewritten to eliminate “Hi Grandma/Grandpa” openers that had infected all nongrandparent types in [1]. Residual legacy contamination from pre-fix runs is documented in Section IV. • Pipeline Bug Fix—Prompt Loading: A hardcoded _v1.md suffix in load_prompt_template() silently prevented updated prompts from loading in the first dataset. Fixed via glob-pattern matching for all prompt versions. E. Per-Scam-Type Outcome Distribution A property of the COVA-X dataset that emerges from the production-count review is the per-scam-type outcome distribution. Rather than appearing as a roughly uniform pattern across types, outcomes are strongly modulated by scam-type characteristics: scams that exploit emotional pressure (virtual kidnapping) produce different outcome distributions than scams that exploit institutional trust (medicare, bank), which themselves differ from scams that exploit gradual relationship formation (romance). Table I reports the per-scam-type, peroutcome counts. Three patterns are worth noting because they shape the classifier analysis in Section V. First, virtual kidnapping has the highest successful_scam rate of any type (443 of 1,330, or 33%), more than double the dataset-
wide successful_scam rate (1,671 of 10,985, or 15.2%). This is consistent with the high-emotional-pressure framing exploited by VK scams: simulated victims under acute stress are markedly more likely to comply with attacker demands. Second, grandparent has by far the highest verification_attempt rate (868 of 1,376, or 63%), consistent with the grandparent-scam pattern in which victims attempt to verify the call back to the supposed grandchild before acting. Third, medicare and bank scams have substantial quick_rejection populations (433 and 212 respectively), reflecting the public salience of medicare and bank scams and victims having institutional alternatives (call the bank directly, hang up). These patterns motivate the per-scam-type breakdown reported alongside classifier results in Section V: a classifier’s overall accuracy is shaped by the dataset’s outcome distribution, and per-type breakdowns reveal architecture-specific strengths obscured by aggregate metrics. The outcomedistribution heterogeneity motivates the stratified train/val/test split (scam type × outcome class, seed 42), preserving the joint distribution of these two variables across splits. IV. DATASET Q UALITY L IFECYCLE A. Post-Generation Quality Scan All 10,985 preprocessed conversations were scanned using scan_generation_quality.py, which detects five artifact types: DIALOGUE COLLAPSE (victim writing attacker lines in their own turn), STAGE DIRECTION (bracket/parenthetical stage directions), TURN OVERFLOW (excessively long individual turns), META BLEED (model instruction echo), and LOOP DETECTION (repetitive content). Across the full preprocessed dataset, the post-cleanup flag rate is 14.5%, dominated by virtual-kidnapping conversations whose post-cleanup flag rate is 46.5% (Section IV-B). Without virtual kidnapping, the flag rate across all other types is approximately 3.5%. Table II reports the per-scam-type breakdown of post-cleanup flag rates and shows the heterogeneity that motivated the architectural intervention reported in Section IV-B. TABLE II P OST-C LEANUP F LAG R ATES BY S CAM T YPE . V IRTUAL KIDNAPPING IS THE DOMINANT SOURCE OF ARTIFACT CONCENTRATION ; REMAINING TYPES ARE ALL BELOW 6%. Scam Type
Flagged
Total
Rate
Virtual Kidnapping Romance Investment Lottery Bank Impersonation Grandparent Government Impersonation Medicare
619 78 47 36 47 38 41 87
1,330 1,934 975 975 1,360 1,376 1,545 1,490
46.5% 4.0% 4.8% 3.7% 3.5% 2.8% 2.7% 5.8%
Total Excl. VK
993 374
10,985 9,655
9.0% 3.9%
The dataset-wide rate of 14.5% reported in earlier analysis includes multiple flag types per conversation; Table II reports
unique conversations flagged at least once. Both framings agree on the central observation: VK is the dominant driver of artifact concentration, and remaining scam types are individually below 6%. B. Virtual-Kidnapping Three-Point Evidence Chain Virtual kidnapping is uniquely structured among the eight scam types: it involves three parties (kidnapper, victim, and a supposedly-kidnapped hostage). Initial generation used a tworole architecture, in which the attacker model was directed to produce hostage utterances within its own turns using a structural [HOSTAGE VOICE]: tag. This produced a 67.1% flag rate on virtual-kidnapping conversations, dominated by DIALOGUE COLLAPSE (the attacker model writing all three voices within a single turn) and STAGE DIRECTION leakage. Section VI (Architectural Workaround and Its Residual Limits) documents the three-role generation architecture (Path B) we introduced to address this. Path B treats the hostage as a distinct agent during generation, with a dedicated system prompt and its own turn; hostage turns are folded backward into the preceding attacker turn at JSON-write time with the [HOSTAGE VOICE]: tag added by the generator itself. Figure 2 contrasts the two architectures. two-role
victim
attacker writes hostage voice within turn three-role (Path B)
attacker
hostage
victim
distinct agent; folded at write time
Fig. 2. Generation architectures for virtual kidnapping. In the two-role design, the attacker model generates hostage speech within its own turns, increasing the risk of dialogue collapse. In three-role Path B, the hostage is modeled as a separate agent with its own system prompt; hostage turns are later merged into the preceding attacker turn using a [HOSTAGE VOICE]: tag.
Table III reports the three-point evidence chain documenting Path B’s effect on virtual-kidnapping flag rates: the prePath-B baseline, the post-Path-B raw rate (after architectural intervention but before post-processing), and the post-Path-B processed rate (after a bare-colon stripping filter and a nonLatin-script filter applied during cleaning). TABLE III T HREE -P OINT E VIDENCE C HAIN : VK F LAG R ATE BY P IPELINE S TAGE Pipeline Stage Pre-Path-B (two-role baseline) Post-Path-B raw (three-role, no cleanup) Post-Path-B processed (after cleaning)
VK Flag Rate 67.1% 46.7% 46.5%
Figure 3 visualizes the trajectory across these three pipeline points. The visual representation makes vivid that the archi-
67.1
60
46.7
46.5
Post-Path-B raw
Post-Path-B proc.
40 20 0 Pre-Path-B
Fig. 3. Virtual-kidnapping flag rates across pipeline stages: the three-role intervention reduced flags by 20.4 points, with post-processing reducing them by only 0.2 more, indicating that most removable artifacts were eliminated by construction rather than downstream cleanup.
tectural intervention accounts for the bulk of the artifact-rate reduction (20.4 % points), while post-processing contributes a much smaller incremental effect (0.2% points). Majority of the reduction (20.4 % points) comes from the architectural change. The post-processing stage contributes a smaller 0.2% point further reduction, indicating that most cleanable artifact patterns were eliminated by construction once the three-role separation was in place. The residual 46.5% flag rate reflects artifact categories that the architectural intervention does not address (repetition loops under emotional pressure, address-term distribution bias) and is documented as a Qwen 2.5 14B capability limit in Section VI. C. Contamination Scan Scanning for grandparent-opener bleed and cross-scam contamination revealed that the contamination was successfully eliminated for bank, lottery, and investment (0 grandparent openers). Government impersonation (357 conversations), medicare (348), and virtual kidnapping (355) retain grandparent-opener contamination from pre-fix pipeline runs performed during the initial expansion. These are documented as legacy contamination rather than corrected post-hoc, preserving data integrity and providing a measurable comparison point for future pipeline iterations. D. Stage-Direction Stripping A post-processing pass using strip_stage_directions() was applied to all preprocessed conversations. The pass affected approximately 0.6% of dataset turns (concentrated in virtual kidnapping) with minimal modification to other scam types. This minimal impact on non-VK types confirms that the ABSOLUTE OUTPUT RULE injection during generation was largely effective for six of the eight scam types; the residual VK rate is the focus of the three-role architectural intervention reported above. E. Label Quality Audit and Relabeling The full dataset was audited using audit_outcome_labels_v2.py and relabeled via relabel_outcomes_v2.py. Of the 10,985 conversations processed by the audit, 439 (3.9%) received corrected labels. Per-scam-type breakdown of the 439 corrections: medicare 195, bank 95, investment 50, virtual kidnapping
32, romance 18, government impersonation 17, grandparent 16, lottery 16. The most common single correction was verification_attempt → quick_rejection (121 instances). A second-pass audit using updated v2 heuristics identified seven high-confidence Carol-profile compound-response conversations that had been mislabeled as verification_attempt; these were corrected to quick_rejection. An additional 5,734 v2-audit findings reflecting the v2 audit’s expanded detection rules were deferred to future work pending manual classification of true mismatches versus rule false positives. This represents a dramatic improvement over the first dataset’s 49.8% mismatch rate—a 12.7× improvement in label self-consistency. The improvement is attributable to three pipeline changes: better prompt behavioral guidance, outcomepinned profiles for Carol and George, and the expanded attacker/victim profile set producing more self-consistent conversations. Figure 4 visualizes the comparison. Label-correction rate (%)
VK flag rate (%)
80
60
49.8
40 20 3.9 0 Paper 1
COVA-X
Fig. 4. Label-correction rates fell from 49.8% in Paper 1 to 3.9% in COVAX, a 12.7× improvement. This reduction reflects three pipeline changes: outcome-pinned victim profiles, runtime ABSOLUTE OUTPUT RULE injection, and an expanded attacker-victim profile set that produced more selfconsistent conversation arcs.
A pipeline bug was also discovered and fixed during this phase: relabel_outcomes_v2.py used glob() rather than rglob(), silently skipping all subdirectory files. The fix was applied before the relabeling run. V. E XPERIMENTATION A. Experimental Setup 1) Data Splits: The 10,985-conversation dataset2 was partitioned into stratified train (80%), validation (10%), and test (10%) splits using random seed 42, maintaining proportional representation of scam categories and outcome classes. This yielded 8,787 training conversations, 1,099 validation conversations, and 1,099 test conversations. Total turn count after preprocessing is 88,852. 2) Label Mapping: The same 3-class collapse from the first paper was applied: complied (successful scam), partial (partial compliance), and rejected (verification attempt + scam detected + quick rejection). Class distribution in the training set: rejected 46.4%, partial 37.5%, complied 16.1%. 2 The COVA-X dataset is fully documented in a companion data sheet covering identity, composition, provenance, classifier results, and known limitations. The data sheet will be released alongside the dataset. Per-scam-type generation targets summed to approximately 11,400; the realized production count of 10,985 reflects normal generator retry overhead and minimum-turnfloor discards rather than separate preprocessing-filter loss.
3) Models: We retrained the same model families as the first paper, focusing on the three best-performing configurations: XGBoost + TF-IDF (sklearn), DistilBERT v2 (tail truncation, 512 tokens), and Longformer (full context, 1,024 tokens). The DistilBERT v2 tail-truncation strategy was introduced in the first paper and retains the last 512 tokens rather than the first, preserving the outcome-critical final turns of each conversation. Hyperparameters and architectures are unchanged between the pre-Path-B first-pass training run and the post-cleanup retrain on the post-Path-B dataset, supporting direct dataset-state comparison between the two runs (see Section V-E). Table IV summarizes the configurations. TABLE IV C LASSIFIER C ONFIGURATIONS ( POST- CLEANUP RETRAIN ; UNCHANGED FROM PRE -PATH -B BASELINE ).
(0.7753, 0.7632) on every reported metric. The first paper’s hypothesis that transformer architectures would benefit from larger conversational training corpora is directly confirmed. C. Per-Class Analysis Table VI reports test-set precision, recall, and F1 by class for all three models. We report all three classifiers rather than Longformer alone because the cross-architecture perclass comparison constitutes part of the cross-architecture consistency finding discussed in Section VI-C. TABLE VI P ER -C LASS T EST P ERFORMANCE ON COVA-X T EST S ET ( N =1,099, POST- CLEANUP RETRAIN ). B EST F1 PER CLASS IN BOLD . Model
Class
n
Prec.
Rec.
F1
Model
Configuration
XGBoost + TF-IDF
complied partial rejected
166 429 504
0.706 0.739 0.843
0.651 0.718 0.885
0.677 0.728 0.864
XGBoost + TF-IDF
sklearn defaults; TF-IDF unigrams + bigrams, 5,000 max features, sublinear TF; plus 28 engineered features from the first paper’s feature set
DistilBERT v2 (tail)
complied partial rejected
166 429 504
0.738 0.752 0.803
0.747 0.678 0.867
0.743 0.713 0.834
Longformer (full)
complied partial rejected
166 429 504
0.696 0.790 0.837
0.771 0.702 0.887
0.731 0.743 0.861
DistilBERT v2 (tail)
distilbert-base-uncased (66M params); tail truncation to last 512 tokens; −5 batch size 16, learning rate 2 × 10 , 8 epochs, balanced class weights; best-byval-F1m model selection (best epoch 4)
Longformer (full)
allenai/longformer-base-4096 (148M params); full-context input (max 1,024 tokens, no truncation); batch size 4, learning rate 2 × 10−5 , 8 epochs, balanced class weights; best-by-val-F1m model selection (best epoch 5)
B. Results Table V presents the complete 3-class test results for all three models across both datasets. The key finding is preserved from the pre-Path-B first-pass run and strengthened by the post-cleanup retrain: Longformer surpasses XGBoost on all three metrics, reversing the ranking from the first paper.
Longformer achieves the best F1 on every class. The transformer advantage is most pronounced on the minority class complied (Longformer 0.731 vs. XGBoost 0.677 F1, a 5.4 point gap), consistent with prior findings that contextual encoders handle class imbalance better than bag-of-words approaches at sufficient training-data scale. XGBoost’s weakness on complied stems from low recall (0.651) despite moderate precision (0.706) — it under-predicts the minority class. DistilBERT v2 is intermediate on complied but underperforms both other classifiers on partial (F1 0.713), the largest non-rejected class, reflecting its 512-token truncation limit discarding midconversation context that Longformer’s full-context encoding preserves. D. Confusion Matrix Analysis
TABLE V C OMPLETE 3-C LASS T EST R ESULTS : F IRST PAPER VS . COVA-X ( POST- CLEANUP RETRAIN , POST-VK-PATH -B). B EST RESULTS IN BOLD . Model
F1-m
F1-w
COVA-X (10,985 conversations, this paper) Longformer (full) 0.7971 0.7786 XGBoost + TF-IDF 0.7843 0.7563 DistilBERT v2 (tail) 0.7753 0.7632
Acc.
0.7956 0.7825 0.7730
First Paper Baseline (3,201 conversations) XGBoost + TF-IDF 0.7250 0.6910 DistilBERT v2 (tail) 0.6980 0.6740 Longformer (full) 0.6980 0.6670
0.7180 0.6950 0.7010
Improvement (COVA-X vs. First Paper, same model) Longformer +9.91pp +11.16pp +9.46pp XGBoost + TF-IDF +5.93pp +6.53pp +6.45pp DistilBERT v2 +7.73pp +8.92pp +7.80pp
Longformer achieves 0.7971 accuracy and 0.7786 macro F1, exceeding XGBoost (0.7843, 0.7563) and DistilBERT v2
Table VII presents the test-set confusion matrices for all three classifiers on the post-cleanup retrain (n=1,099). Reporting all three matrices side-by-side supports the crossarchitecture consistency analysis in Section VI-C: the error structure should be similar across architectures if the residual classifier errors reflect dataset-level boundary ambiguity rather than architecture-specific failure modes. The error structure is broadly consistent across all three classifiers: the largest off-diagonal cells are in the partial row, distributed between complied predictions (38–48 conversations) and rejected predictions (78–100 conversations). This boundary confusion is expected: victims who engage extensively before rejecting produce dialogue that is lexically similar to those who engage without fully committing. The cross-architecture similarity of this error pattern argues that the boundary cases are genuinely ambiguous at the data level, not artifacts of any one architecture.
TABLE VII C ONFUSION M ATRICES ON COVA-X T EST S ET ( N =1,099, POST- CLEANUP RETRAIN ). ROWS = TRUE CLASS , COLUMNS = PREDICTED CLASS .
True \ Pred.
compl.
partial
rejected
108 43 2
53 308 56
5 78 446
compl.
partial
rejected
124 38 6
35 291 61
7 100 437
complied partial rejected
Improvement (pp)
XGBoost + TF-IDF
DistilBERT v2 (tail) True \ Pred. complied partial rejected
complied partial rejected
compl.
partial
rejected
128 48 8
31 301 49
7 80 447
To isolate the contribution of the dataset cleanup pipeline (Section IV) from architectural and hyperparameter choices, we report results at two pinned dataset states: a pre-PathB first-pass run on the initial COVA-X dataset, and a postcleanup retrain on the post-Path-B dataset reported above. Hyperparameters and training procedures are unchanged between the two runs. The only varying input is dataset state: the cleanup pipeline replaced the COVA-X expansion’s initialbatch (two-role architecture) virtual-kidnapping conversations with the three-role Path B regenerations described in Section IV; the 1,330 virtual-kidnapping conversations in the dataset are now exclusively three-role Path B output. The cleanup pass also applied seven outcome-label corrections to Carol-profile conversations. TABLE VIII P RE VS . P OST-PATH -B C LASSIFIER P ERFORMANCE ( HELD - OUT TEST SET, N =1,099). S AME HYPERPARAMETERS ; VARYING INPUT IS DATASET STATE .
XGBoost + TF-IDF DistilBERT v2 (tail) Longformer (full)
Pre acc. Post acc.
∆ acc.
0.7764 0.7625 0.7880
+0.79pp +0.0198 +1.28pp +0.0220 +0.91pp +0.0083
0.7843 0.7753 0.7971
1.28
1
0.79
0.910.83
Fig. 5. Cross-architecture classifier improvement between the pre-Path-B firstpass run and the post-cleanup retrain (post-Path-B). All three architectures improve on both metrics. Macro F1 improvement scaled by 102 for visual comparability with accuracy in % point terms.
E. Pre/Post-Path-B Sensitivity
1
1.98
XGBoost DistilBERT v2 Longformer ∆ accuracy (pp) ∆ macro F1 (×102 )
Two architecture-specific patterns are also worth noting. First, DistilBERT v2 has the highest partial → rejected misclassification rate (100 of 429, or 23.3%), consistent with its 512-token truncation discarding mid-conversation engagement signals that distinguish partial compliance from outright rejection. Second, XGBoost has the highest complied → partial misclassification rate (53 of 166, or 31.9%), reflecting the bag-of-words representation’s difficulty distinguishing fullycommitted from partially-committed victims when both produce similar lexical patterns.
Model
2.2
2
0
Longformer (full) True \ Pred.
All three architectures improve on the post-Path-B dataset on both accuracy and macro F1. Longformer remains the bestperforming model at both dataset states; the cross-architecture ranking is preserved across the cleanup. Figure 5 visualizes the cross-architecture improvement pattern.
∆ F1-m
We discuss the implications of this consistency in Section VI-C. F. Five-Class XGBoost Comparison The classifier results above use the 3-class collapse standardized since the first paper (Section V). For completeness, we report the corresponding 5-class XGBoost result on the pre-Path-B baseline run, which was the first sklearn training run on the COVA-X dataset and used the original 5-class label set (successful_scam, partial_compliance, verification_attempt, scam_detected, quick_rejection). On the same n=1,099 test set with seed 42 stratification, XGBoost + TF-IDF achieves accuracy 0.7498, macro F1 0.7745, weighted F1 0.7489. The macro F1 is higher in the 5-class formulation than 3-class, which is an artifact of two small classes (quick_rejection n=73, scam_detected n=97) being highly identifiable through distinct lexical patterns; macro-averaging over the 5 classes upweights these strong-performing minority classes. Substantively the 5-class results are consistent with the 3-class headline: XGBoost performs strongly on quick_rejection (precision 0.784, recall 0.945) and scam_detected (precision 0.914, recall 0.763), but the boundary between successful_scam/partial_compliance and verification_attempt remains the dominant error mode. The 3-class formulation collapses these into the more interpretable rejected category and is retained as the primary reporting basis for direct comparison to the first paper. 1 The XGBoost pre-Path-B baseline shown here was not separately serialized as a standalone JSON; its values are preserved in the comparison sections of the April 4, 2026 DistilBERT v2 and Longformer training output files, which independently logged identical XGBoost reference metrics (0.7764 / 0.7365 / 0.7713). The 5-class XGBoost run from April 2 is preserved with the same records for reference. The DistilBERT v2 and Longformer prePath-B baselines are anchored to JSON training-output files preserved in the same records.
VI. D ISCUSSION A. Hypothesis Confirmation The first paper’s central hypothesis—that transformer models would benefit from larger training corpora—is directly confirmed by the COVA-X results. Longformer’s improvement from 0.667 to 0.7786 macro F1 (+11.16 % points) substantially exceeds XGBoost’s improvement from 0.691 to 0.7563 (+6.53 % points). With 8,787 training conversations, Longformer’s full-context architecture is able to exploit conversation-level signals that TF-IDF representations compress into bag-of-words statistics. The cross-paper improvement gap (Longformer +11.16pp vs. XGBoost +6.53pp on macro F1) is the primary quantitative evidence supporting this paper’s central claim. B. Why Longformer Surpasses XGBoost Firstly,full-context encoding without truncation: Longformer’s 4,096-token capacity accommodates all conversations without information loss (maximum observed token count: 874, well within the 1,024-token training limit). In contrast, DistilBERT’s 512-token limit still forces truncation for approximately 20% of sampled conversations even with the tail strategy, discarding outcome-critical signal located in midconversation turns. Secondly, sufficient data for fine-tuning: Longformer exhibited overfitting by epoch 4 (training loss 0.18 vs. validation loss 2.30) in [1]. On COVA-X, Longformer’s best epoch is 5, and the validation loss remains below 1.0 at that point, indicating substantially better generalization from the larger training set. C. Cross-Architecture Cleanup-Effect Consistency The Pre/Post-Path-B Sensitivity results in Table VIII provide a methodological observation distinct from the headline hypothesis confirmation. All three classifier architectures improve on the post-Path-B dataset relative to the pre-Path-B first-pass run, on both accuracy and macro F1. The cleanup intervention—reduction in stage-direction artifacts, dialoguecollapse instances, and repetition loops in virtual-kidnapping conversations, plus seven high-confidence outcome relabels— produces measurable signal recoverable across architectural families that operate on fundamentally different representational substrates (bag-of-words TF-IDF, transformer embeddings with tail truncation, transformer embeddings with full context). This consistency is informative because it argues against the cleanup operating on architecture-specific noise. If the post-Path-B improvements were concentrated in tokens visible only to one model (for example, only to the tail-truncation window of DistilBERT v2), we would expect asymmetric improvement patterns across the three architectures. Instead, all three improve, suggesting the cleanup affected genuine labelrelevant signal accessible at multiple levels of representational granularity. We note, however, that the improvement magnitudes are not uniform across architectures. DistilBERT v2 shows the
largest absolute change on both accuracy (+1.28pp) and macro F1 (+0.0220). XGBoost shows a smaller swing on accuracy (+0.79pp) and a comparable swing on macro F1 (+0.0198). Longformer shows the smallest swing on macro F1 (+0.0083). One plausible reading is that DistilBERT v2’s larger swing reflects the proximity of cleanup to its truncation window: the cleanup primarily affected end-of-conversation tokens (stage directions, dialogue-collapse text, repetition loops in late VK turns), which are precisely the tokens the tail-truncation strategy preserves. Cleaner end-of-conversation tokens produce an outsized DistilBERT improvement.3 Longformer’s smaller swing is consistent with its architecture being less sensitive to artifact concentration in any particular conversational region: with full context available, it already extracted most of the available signal at the pre-Path-B state, so cleanup adds proportionally less. We treat this as a methodological observation rather than a causal claim. Cleanly establishing causality between cleanupby-token-region and per-architecture improvement magnitude would require a controlled ablation isolating cleanup effects to specific token windows, proposed as future work in Section VII. The cross-architecture consistency claim is supported by direct JSON-anchored evidence for two of three classifiers. The DistilBERT v2 and Longformer pre-Path-B baselines are preserved as pre-cleanup training-output JSON files, paired against the post-cleanup retrain JSONs. D. Quality Pipeline as Methodology The 3.9% relabeling correction rate in COVA-X versus 49.8% in the first dataset represents a 12.7× improvement in label self-consistency. This improvement was not a single fix but the compounded result of five independent pipeline improvements: profile pinning for Carol and George, the ABSOLUTE OUTPUT RULE injection, the glob→rglob bug fix in the relabeling script, expanded victim/attacker profiles providing more behavioral variety, and revised attacker prompt templates eliminating cross-type contamination. The Pre/Post-Path-B Sensitivity results (Section VI-C) provide additional methodological evidence beyond the labelcleanliness story. The cleanup pipeline—a separate intervention from the audit-and-relabel pass that yielded the 3.9% rate—produced cross-architecture classifier improvements indicating that the cleanup affected classifier-recoverable signal. We argue that documenting these improvements quantitatively, with before/after deltas at each pipeline stage and at multiple classifier architectures, is itself a methodological contribution. Synthetic dataset construction is an iterative process, and 3 It is worth flagging that XGBoost’s small swing despite substantial cleanup could be read either as capability ceiling (the model has extracted what bagof-words can extract from the lexical signal, and additional cleanup does not unlock additional features) or as a robustness property (the model was less sensitive to the artifact noise removed by cleanup in the first place). We do not adjudicate between these readings; the relevant claim for this paper is the cross-architecture direction of improvement, not its precise mechanism per architecture.
the growing pains encountered are informative for future researchers. E. Limitations First, the dataset remains English-only and US-centric. Second, all models still treat conversations as complete documents; real-time detection would require incremental classification (see Section VII). Third, legacy grandparent-opener contamination persists in government impersonation, medicare, and virtual kidnapping files from pre-fix pipeline runs; specific counts (gov imp 357, medicare 348, VK 355) are documented in the companion data sheet and preserved as legacy contamination rather than corrected post-hoc. Fourth, we document below a set of generation-model capability limits identified through iterative prompt engineering on the virtual kidnapping scam type. These findings are specific to Qwen 2.5 14B and motivate the model-comparison study proposed in Section VII. 1) Structural Tag Adoption Failure: Virtual kidnapping is uniquely structured among the eight scam types in COVA-X: it involves three parties (attacker, victim, and a supposedlykidnapped hostage whose voice is played by AI-cloned audio during the call). Our generation pipeline is architecturally two-role, producing alternating attacker and victim turns. We initially attempted to embed hostage voice inside attacker turns using structural tags such as [HOSTAGE VOICE]:, with the tag adopted by the attacker model during generation. Across four iterations of prompt engineering (v6 through v9 of the VK attacker prompt), we progressively intensified enforcement of this tag rule through explicit prohibitions, forbidden-format examples, repositioned rule priority, and recovery instructions. Despite these reinforcements, Qwen 2.5 14B produced the target tag in 0% of attacker turns across 30 test conversations. The model instead defaulted to a pretraining-learned bare-colon pattern (e.g., : ‘‘Mom! Help!’’) in every attacker turn regardless of the prompt structure. This suggests that the attacker-hostage voice embedding is not a prompt-engineering problem at the 14Bparameter scale under this emotional context. 2) Salience Bias in Prompt Engineering: During Path A iterations, we observed an instructive anti-pattern. Version 9 of the attacker prompt explicitly enumerated forbidden barecolon variants (e.g., : ‘‘Dad!’’, : ‘‘Please!’’) as negative examples, intending to discourage their production. Counterintuitively, the bare-colon leakage rate increased from approximately 15% of attacker turns under v8 to approximately 40% under v9. We interpret this as salience-bias in instruction-tuned LLMs: adding examples of prohibited patterns increases their attention-weight presence in the prompt context, making the model more likely to reproduce them rather than suppress them. This finding is methodologically useful for practitioners attempting prompt-level constraint enforcement on similar generation tasks. 3) Architectural Workaround and Its Residual Limits: To address the structural tag adoption failure, we introduced a three-role generation architecture treating the hostage as a
distinct agent during generation, with a dedicated system prompt and its own turn (Path B). At JSON-write time, hostage turns are folded backward into the preceding attacker turn with the [HOSTAGE VOICE]: tag added by the generator itself, preserving two-role format for downstream classifier training. This architectural change achieved 100% structural consistency by construction and eliminated the dialogue-collapse failure mode observed in earlier batches. Section IV reports the three-point evidence chain documenting Path B’s effect on VK-specific artifact rates (67.1% pre-Path-B → 46.7% postPath-B raw → 46.5% post-Path-B processed flag rate). The three-role architecture did not, however, address several additional Qwen 2.5 14B behaviors: Repetition loops under emotional pressure. Attacker turns frequently converge on near-verbatim repetition by turn 10 in high-emotional-pressure scenarios, consistent with context-window degradation rather than role confusion. • Profile compliance breakdown. The highly-skeptical and informed victim profiles (targeting quick rejection and scam detected outcomes, respectively) showed compliance breakdown under VK emotional pressure. The model treats personality-profile directives as soft guidance rather than hard constraints in emotionally charged contexts. This behavior is observed consistently in VK and partially in government impersonation; other scam types respect victim profiles. • Pretraining distribution bias on address terms. Regardless of the configured hostage_address_term parameter (Mom/Dad/Grandma/Grandpa/Honey), Qwen 2.5 14B defaults to “Mom!” address patterns in approximately 70% of grandchild and spouse relationships, reflecting a strong pretraining prior toward child-to-mother kidnapping-scenario audio. • Cross-conversation name drift. The most frequent dataset names (e.g., Jessica, Ashley) occasionally bleed across distinct victim profiles, producing concatenated-name artifacts (e.g., “JessicaEmily”) indicating the model’s output is influenced by recent training exposure despite explicit name-identity rules in the system prompt. • Conversation termination override. Government impersonation prompts include a CONVERSATION ENDING rule, added during the initial expansion run and subsequently strengthened, instructing termination after victim refusal or success. The model still overrides this rule in about 22.3% of conversations, producing farewell-andcompliance loops at turns 15–19. Because stronger enforcement risks the salience-bias regression noted above, these loops are treated as a documented capability limit rather than further prompt-engineered away. •
These five behaviors, observed independently across distinct prompt-directive types (structural tag adoption, victimprofile compliance, address-term parameterization, nameidentity rules, and conversation-termination directives), constitute a consistent pattern: Qwen 2.5 14B honors prompt directives in short conversations and low-pressure contexts,
but overrides them under sustained emotional or contextual pressure as conversation length increases. We treat this as a capability ceiling rather than a prompt-engineering deficiency, motivating the model-comparison study proposed in Section VII. The five-instance pattern provides a well-defined test target for future work: can a comparison model obey structural tags, sustain profile specifications under emotional pressure, parameterize address terms, maintain name identity, and honor conversation-termination directives across the same test conditions? Residual artifact rates are addressed post-generation through a bare-colon stripping filter and non-Latin-script filter in the cleaning pipeline, with the post-cleanup VK flag rate of 46.5% documented in Section IV. Mitigation through architectural and post-processing changes is partial; future work using alternative or larger generation models is the expected path to further reduction. VII. C ONCLUSION AND F UTURE W ORK This paper presented COVA-X, expanding the COVA conversational smishing dataset from 3,201 to 10,985 synthetic multi-turn conversations. Retraining all classifiers on the expanded corpus shows that Longformer now outperforms XGBoost + TF-IDF, achieving 79.71% accuracy and 0.7786 macro F1 compared with 78.43% and 0.7563 for XGBoost. This supports the hypothesis from our prior work that transformer models require larger conversational corpora to realize their contextual advantages. Across model families, the strongest gains were observed for transformer-based models, indicating that contextual representations become more effective as dataset scale and quality improve. COVA-X also contributes a synthetic-data quality lifecycle analysis. Pipeline refinements reduced relabeling corrections from 49.8% to 3.9%, while the virtual-kidnapping three-role architecture reduced artifact flag rates from 67.1% to 46.7%, with post-processing yielding a final rate of 46.5%. Per-scamtype outcome analysis shows mechanism-consistent patterns, including higher successful_scam rates for virtual kidnapping, higher verification_attempt rates for grandparent scams, and substantial quick_rejection rates for medicare and bank impersonation. Pre/post-cleanup sensitivity analysis further shows that cleanup improved label-relevant signal across multiple classifier architectures. Future work will extend COVA-X to audio-domain voicephishing detection using synthetic text-to-speech, conduct controlled ablations to separate dataset-scale effects from pipelinequality effects, and evaluate alternative generation models and grammar-constrained decoding to address observed generation failures. Additional directions include real-time classification from partial transcripts, cross-linguistic expansion, real-transcript validation, per-scam-type classifier analysis, and early-stopping studies for transformer fine-tuning. The COVAX dataset, accompanying data sheet, classifier baselines, and evaluation scripts will be made available for research use following acceptance.
R EFERENCES [1] C. Lochstampfor and A. Roy, “A Synthetic Conversational Smishing Dataset for Social Engineering Detection,” arXiv preprint arXiv:2604.11752, 2026. [Online]. Available: https://arxiv.org/abs/2604.11752 [2] G. Desolda, L. S. Ferro, A. Marrella, T. Catarci, and M. F. Costabile, “Human factors in phishing attacks: a systematic literature review,” ACM Computing Surveys, vol. 54, no. 8, pp. 1–35, 2021. [3] Ent Credit Union, “How to spot financial grooming scams,” May 2025. [Online]. Available: https://www.ent.com/education-center/ securitycenter/how-to-spot-financial-grooming-scams/ [4] J. W. Seo et al., “On-device smishing classifier resistant to text evasion attack,” IEEE Access, vol. 12, pp. 4762–4779, 2024. [5] C. Patra, D. Giri, M. S. Obaidat, and T. Maitra, “SMSDECT: A prediction model for smishing attack detection using machine learning and text analysis,” in Proc. IEEE GLOBECOM, 2023, pp. 3837–3842. [6] A. Derakhshan, I. G. Harris, and M. Behzadi, “Detecting telephonebased social engineering attacks using scam signatures,” in Proc. ACM Workshop on Security and Privacy Analytics, 2021, pp. 67–73. [7] I. D. Wood, M. Kepkowski, L. Zinatullin, T. Darnley, and M. A. Kaafar, “An analysis of scam baiting calls: Identifying and extracting scam stages and scripts,” Macquarie University, Tech. Rep., 2023. [8] A. K. Jain, K. Kaur, N. K. Gupta, and A. Khare, “Detecting smishing messages using BERT and advanced NLP techniques,” SN Computer Science, vol. 6, no. 2, p. 109, 2025. [9] N. Basta, C. Atkins, and D. Kaafar, “Bot Wars Evolved: Orchestrating Competing LLMs in a Counterstrike Against Phone Scams,” Macquarie University, Tech. Rep., 2025. [10] T. Kumarage et al., “Personalized Attacks of Social Engineering in Multi-Turn Conversations: LLM Agents for Simulation and Detection,” Arizona State University, Tech. Rep., 2024. [11] D. Spokoyny, N. Vogler et al., “Victim as a Service: Designing a System for Engaging with Interactive Scammers,” UC San Diego, Tech. Rep., 2025. [12] L. D. Herrera, L. V. Sickle, and A. Podhradsky, “Bridging the Protection Gap: Innovative Approaches to Shield Older Adults from AI-Enhanced Scams,” Dakota State University, Tech. Rep., 2024. [13] FBI Internet Crime Complaint Center, “2023 Elder Fraud Report,” FBI IC3, Tech. Rep., 2024. [14] H. Xu, A. Qadir, and S. Sadiq, “Malicious SMS Detection Using Ensemble Learning and SMOTE to Improve Mobile Cybersecurity,” Computers & Security, vol. 154, p. 104443, 2025. [15] W. Li et al., “Talking Like a Phisher: LLM-Based Attacks on Voice Phishing Classifiers,” Tech. Rep., 2025. [16] H. Park et al., “Enhanced Voice Phishing Detection Using an LLMBased Framework for Data Augmentation and Classification,” IEEE Access, vol. 11, 2025. [17] S. Gressel et al., “Love, Lies, and Language Models: Investigating AI’s Role in Romance Scam Detection,” Tech. Rep., 2025. [18] “AI-in-the-Loop: Privacy-Preserving Real-Time Scam Detection Leveraging LLMs and Federated Learning,” Proc. on Privacy Enhancing Technologies, 2026. [19] S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi, “Cross-Task Generalization via Natural Language Crowdsourcing Instructions,” in Proc. ACL, 2022, pp. 3470–3487. [20] S. Mishra, D. Khashabi, C. Baral, Y. Choi, and H. Hajishirzi, “Reframing Instructional Prompts to GPTk’s Language,” in Findings of ACL 2022, pp. 589–612. [21] J. Jang, S. Ye, and M. Seo, “Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts,” arXiv:2209.12711, 2022.