arXiv:2604.11752v1 [cs.CR] 13 Apr 2026
A Synthetic Conversational Smishing Dataset for Social Engineering Detection Carl Lochstampfor
Ayan Roy
Department of Cybersecurity Old Dominion University Norfolk, Virginia, United States [email protected]
School of Engineering and Computing Christopher Newport University Newport News, Virginia, United States [email protected]
Abstract—Smishing (SMS phishing) has emerged as a significant cybersecurity threat, particularly targeting elderly and cyber-unaware individuals, leading to substantial financial losses and erosion of trust. While prior research has proposed methods to classify individual text messages as smishing or benign, realworld attackers frequently employ multi-stage social engineering strategies, gradually grooming victims through extended conversational exchanges before extracting sensitive information. Despite the availability of datasets for single-message smishing detection, conversational smishing datasets remain largely absent, limiting research on multi-turn attack detection. To address this gap, this paper introduces a synthetically generated dataset consisting of 3,201 multi-round labeled conversations that emulate realistic conversational smishing attacks. The dataset captures diverse attacker tactics and victim responses across multiple stages of engagement. To establish baseline performance, we evaluate eight models spanning traditional machine learning (Logistic Regression, Random Forest, Linear SVM, XGBoost) and transformer architectures (DistilBERT, Longformer), using both engineered conversational features and TF-IDF textual representations. Experimental results show that models leveraging TF-IDF features consistently outperform those using engineered features alone. The best-performing model, XGBoost with TFIDF features, achieves an accuracy of 72.5% and a macro F1 score of 0.691, outperforming both transformer models despite their contextual language understanding. Analysis reveals that input length limitations and limited training data size are the primary factors constraining transformer performance on this task. These results demonstrate the importance of lexical signals in conversational smishing detection and highlight the potential of the proposed dataset for advancing research in social engineering defense. Index Terms—Social Engineering, elder fraud, synthetic data, scam detection, multi-agent LLM, conversation classification, Smishing Detection, Smishing Dataset
I. I NTRODUCTION Short Message Service (SMS) phishing, commonly referred to as smishing, has become one of the fastest growing forms of social engineering attacks. In these attacks, adversaries exploit the immediacy and perceived trustworthiness of text messaging to deceive victims into disclosing sensitive information or transferring funds, typically by impersonating trusted or reputable organizations [1], [2]. Recent cybersecurity reports indicate that smishing campaigns increasingly target elderly and technologically inexperienced individuals, who may be more susceptible to persuasion tactics and urgency-driven
manipulation. The consequences of such attacks often include financial loss, identity theft, and erosion of trust in digital communication systems. Smishing attacks have increased by 18% globally as of 2024, with 484,500 malicious smishing attempts reported in the United States, resulting in excess of $400 million in financial losses [3], [4]. Most existing research on smishing detection [5]–[7] focuses on classifying individual text messages as either malicious or benign. While this approach is useful for detecting isolated phishing attempts, it does not fully capture the behavior of modern attackers. In practice, many smishing attacks unfold as multi-step conversational interactions in which the attacker gradually builds trust with the victim before requesting sensitive information or payment [8]. A snapshot of such interactions is illustrated in Fig. 1. These conversational attacks may involve multiple rounds of dialogue, emotional manipulation, and incremental extraction of personal or financial information. As a result, detection methods designed for single-message analysis may fail to capture the broader conversational context that reveals malicious intent. Despite the growing prevalence of conversational social engineering attacks, publicly available datasets for studying multi-turn smishing interactions remain extremely limited. Most existing datasets consist of isolated SMS messages without conversational structure, which restricts the ability of researchers to study how attackers adapt their strategies over time or how victims respond during extended interactions. Furthermore, collecting real-world conversational smishing data presents significant privacy, ethical, and legal challenges, particularly when vulnerable populations such as elderly individuals are involved. To address this gap, this paper introduces a synthetically generated conversational smishing dataset consisting of 3,201 multi-round labeled conversations across multiple scam categories. The dataset is designed to simulate realistic attacker–victim interactions and capture the dynamics of conversational social engineering attacks, including varied attacker strategies, victim responses, and outcomes. By leveraging a controlled multi-agent generation process, the dataset provides diverse conversational patterns while avoiding the ethical and privacy concerns associated with real-world data collection. In addition to presenting the dataset, we establish baseline per-
Single-Message Smishing
Unknown Number Your bank account has been locked. Verify immediately at: secure-bankalert.com
Conversational Grooming Unknown Number Hi Mrs. Carter, this is Anna from Medicare support.
defensive strategies against social engineering attacks targeting vulnerable populations. II. R ELATED W ORKS A. Social Engineering Detection
I did get something in the mail. What is this about? There may be a verification issue with your coverage. Oh no, I don’t want that. What do you need from me? Please confirm your Medicare ID so I can verify your record.
Fig. 1. Comparison between single-message smishing and conversational grooming-based smishing. While the single-message attack (on the left) exhibits immediate malicious intent, conversational smishing (on the right) involves gradual trust-building and delayed elicitation of sensitive information across multiple turns, making detection significantly more challenging.
formance benchmarks using eight models spanning traditional machine learning (Logistic Regression, Random Forest, Linear Support Vector Machines, and XGBoost) and transformer architectures (DistilBERT and Longformer). We evaluate engineered conversational features, TF-IDF textual representations, and contextual embeddings to understand which approaches are most effective for detecting conversational smishing outcomes. Experimental results show that models leveraging TFIDF features consistently outperform both engineered features and fine-tuned transformers. The best-performing configuration, XGBoost with TF-IDF features, achieves an accuracy of 72.5% and a macro F1 score of 0.691 on the test set. The main contributions of this paper are summarized as follows: We introduce a synthetically generated dataset of 3,201 multi-turn smishing conversations designed to model realistic conversational social engineering attacks. • We provide a structured labeling framework capturing different victim response outcomes within conversational attacks. • We evaluate eight baseline models — six traditional ML classifiers and two transformer architectures — using both engineered conversational features and TF-IDF textual representations. • We demonstrate that TF-IDF-based models outperform transformer models on this dataset, and analyze the contributing factors of input truncation and training data limitations.
•
The proposed dataset and baseline benchmarks establish a foundation for future research on conversational smishing detection and aim to support the development of more effective
Detection of social engineering attacks has been studied across multiple modalities. In the SMS domain, Seo et al. [9] proposed on-device smishing classifiers resistant to text evasion, while Patra et al. [10] developed prediction models combining machine learning with text analysis. However, these approaches address single-message classification and do not capture the multi-turn conversational dynamics characteristic of phone-based social engineering. For telephone-based attacks, Derakhshan et al. [11] introduced ASsET, a detection system based on scam signatures — sets of speech acts that characterize different scam types. Their approach uses word embeddings to identify whether the semantic content of a scam signature appears in a conversation. Wood et al. [12] analyzed scam-baiting calls from YouTube, developing a methodology to semi-automatically identify scam stages and scripts at scale, providing insight into the sequential structure of phone scams. Lansley et al. [13] developed SEADer++, a machine learning framework for detecting social engineering attacks in online environments, demonstrating that content-based features can effectively identify manipulative communication patterns. Xu et al. [2] proposed an ensemble learning based model by combining Random Forest and Support Vector Machine, which helps in classifying a message as malicious or legitimate. Jain et al. [14] propose a smishing detection framework that integrates feature-based methods with advanced language models such as BERT. By evaluating both TF-IDF and contextual embeddings, their results demonstrate that BERT is more effective in capturing semantic and contextual relationships. Additionally, to improve classification performance, the original multi-class problem is reformulated into two binary classification tasks, each addressed using multi-layer neural networks. Oswald et al. [15] introduced Spot Spam, an intention analysis-based approach for detecting SMS spam that leverages BERT embeddings. As a pre-trained language model, BERT provides high-quality contextual representations that capture semantic meaning effectively. The proposed method encodes SMS messages using BERT embeddings and identifies patterns indicative of spam. Additionally, Akande et al. [16] presented a smartphone-based system for real-time smishing detection using rule-based techniques. B. Synthetic Data and Multi-Agent Simulation Recent work has explored LLM-based approaches to simulate and counter social engineering. Basta et al. [17] presented “Bot Wars,” a framework using competing LLMs as scambaiters against phone scams through simulated adversarial dialogues. Their two-layer prompt architecture enables demographically authentic victim personas, and they validated their synthetic dataset of 3,200 dialogues against 179 hours
Phase 1: Multi-Agent Generation
Phase 2: Data Quality Assurance
• Red AI attacker agent • Victim agent (age 65–85) • Qwen 2.5 14B (Ollama) • 8 scam categories • 16 prompt templates
• 3,201 conversations • 25,643 total turns • Audit pipeline v2 • 1,594 relabeled • 49.8% initial mismatch
Phase 3: Detection Baselines • 8 models (ML + Transformer) • TF-IDF + 28 features • Best: XGBoost (72.5%) • Macro F1: 0.691 • Longformer: 69.8%
Fig. 2. COVA framework architecture showing the three-stage pipeline: multi-agent data generation, data quality auditing, and machine learning baseline evaluation.
of human scam-baiting interactions. Kumarage et al. [18] proposed SE-VSim, an LLM-agentic framework for simulating social engineering attack mechanisms in multi-turn conversations, modeling victim agents with varying personality traits to assess susceptibility. Spokoyny et al. [19] developed CHATTERBOX, an LLM-based system that automates long-term engagement with online scammers, addressing pig-butchering and similar interactive scams. C. Gap in the Literature While prior work has advanced detection and simulation in related domains, several important gaps remain. First, publicly available multi-turn scam conversation datasets are scarce, with most comprehensive datasets maintained privately by telecom providers. Second, existing synthetic datasets primarily focus on single scam categories or text-based online scams, rather than telephone-based attacks spanning diverse categories. Third, there is a lack of datasets that explicitly model the conversational dynamics of elder-targeted scams with parameterized victim profiles. Herrera et al. [20] highlight this protection gap, noting that older adults remain disproportionately vulnerable to AI-enhanced scams, while existing protective technologies have not kept pace. To address these limitations, we introduce a publicly available, multi-category conversational smishing dataset focused on elder-targeted scenarios. The proposed dataset captures realistic multi-turn interactions and models attacker behavior across multiple conversational rounds aimed at financial exploitation. In addition, we provide baseline detection models to support future research on conversational smishing detection. III. P ROPOSED M ETHODOLOGY The COVA (Cognitive Operations Virtual Assistant) framework comprises three phases: synthetic conversation generation, data quality assurance, and detection model training. Figure 2 provides an overview of the complete pipeline. The framework is designed to be reproducible and extensible, with all code and generated data publicly available. A. Multi-Agent Conversation Generation Unlike single-prompt approaches that generate entire conversations from a monolithic template, our framework employs
two independent LLM agents — an attacker and a victim — that exchange dialogue turns iteratively. This design produces more natural conversation dynamics, including realistic resistance patterns, escalation tactics, and varied outcomes. • Attacker Agent: Each attacker agent is configured with a scam type, a specific persona (name, claimed role, backstory), tactical parameters (urgency level, payment method, requested amount), and a knowledge level reflecting how much the attacker knows about the victim prior to the call. We model three knowledge tiers: no prior knowledge (50%, cold calls), partial knowledge such as the victim’s name (30%), and full knowledge including family details (20%). • Victim Agent: Victim agents are parameterized with demographic attributes (age 65–85, living situation), personality traits (trust level, scam awareness, tech savviness, emotional tendency), and a target outcome that guides but does not guarantee the conversation trajectory. • Turn-Based Generation: The conversation proceeds as follows: (1) the victim initiates with a greeting, (2) the attacker delivers an opening line consistent with the scam type, (3) agents alternate turns with each response conditioned on the full conversation history, and (4) the conversation terminates when a natural endpoint is reached. Each turn is generated by a separate LLM inference call, preserving the independence of each agent’s perspective. B. Scam Category Coverage We selected eight scam categories that disproportionately affect elderly populations, based on the FBI’s Elder Fraud Report [21] and FTC complaint data. Table I summarizes the categories, key attacker tactics, and typical payment vectors. Each category has dedicated prompt templates for both attacker and victim agents, totaling 16 prompt templates. C. Local Inference Infrastructure We initially explored using commercial LLM APIs (Anthropic Claude) for conversation generation. However, the model’s safety guardrails prevented it from adequately roleplaying as a scam attacker, even when the research context was explicitly framed. We pivoted to local inference using Qwen
TABLE I S CAM CATEGORIES WITH ATTACKER TACTICS AND PAYMENT VECTORS
TABLE II S UMMARY STATISTICS OF THE COVA CONVERSATIONAL SMISHING DATASET
Category Grandparent Virtual Kidnapping Medicare Fraud Romance Government Impersonation Investment Lottery Bank Impersonation
Number of Conversations
400
Key Tactics Payment Emotional manipulation, ur- Wire, gift cards gency Terror, ransom demands Wire transfer Authority, trust exploitation Personal info Emotional bonding, extrac- Wire, crypto tion Fear of legal consequences Gift cards, wire Greed, FOMO, fake returns Crypto, wire Excitement, fee extraction Wire, prepaid Institutional trust, urgency Credentials
400
401
400
400
400
400
400
400
200
0
n Ba
k
G
.I ov
. t nt en ers are mp stm dp e n v a In Gr
ry
tte Lo
e
re
ica ed
M
c an
m Ro
.
ap
dn
Ki rt. Vi
Scam Category Fig. 3. Distribution of synthetic conversations across eight scam categories as mentioned in Table I (N = 3,201).
2.5 14B via Ollama, running on an NVIDIA RTX 4080 Super GPU. This approach offered three advantages: (1) no content restrictions for legitimate research, (2) zero API costs for bulk generation of over 3,000 conversations, and (3) complete data privacy with no data leaving the local machine. IV. DATASET DESCRIPTION AND QUALITY ASSURANCE A. Dataset Statistics The complete dataset comprises 3,201 synthetic multiturn conversations distributed approximately equally across all eight scam categories ( 400 per category). Each conversation is stored as a structured JSON file containing the full dialogue transcript, attacker and victim configurations, outcome labels, and generation metadata. The number of conversations corresponding to the category of scams as mentioned in Table I is highlighted in Figure 3. The dataset contains a total of 25,643 dialogue turns, with a mean of 16.2 turns per conversation and an average of 46.5 victim words per turn. All conversations are in English and simulate phone-based interactions. The summary of the dataset is provided in Table II. The dataset and generation
Metric
Value
Total conversations Total conversation turns Average turns per conversation Number of scam categories Victim age range Prompt templates used Relabeled conversations after audit Initial label mismatch rate
3,201 25,643 ∼16.2 8 65–85 16 1,594 49.8%
pipeline will be publicly released upon publication to support reproducibility and future research on conversational smishing detection. A snapshot of the dataset is shown in Figure 4. B. Outcome Classification The initial generation pipeline assigned one of five finegrained outcome labels: successful scam, partial compliance, verification attempt, scam detected, and quick rejection. Because the scam detected (n = 67) and quick rejection (n = 16) classes contain too few samples for reliable classification, we collapse the five labels into three classes for modeling: • Complied: The victim fully complies with the attacker’s requests. Corresponds to the original successful scam label (n=534). • Partial: The victim engages but does not fully commit. Corresponds to the original partial compliance label (n=1,748). • Rejected: The victim rejects the scammer or terminates the call. Merges verification attempt (n=836), scam detected (n=67), and quick rejection (n=16) (combined n=919). Average conversation length varies by outcome: quick rejections average 4.5 turns, while successful scams average 17.0 turns, reflecting the extended engagement required for full compliance. C. Label Quality Audit Initial outcome labels were assigned by a rule-based classifier during generation. Upon systematic validation, we discovered a 49.8% mismatch rate between the assigned labels and the actual conversation content. Of the 3,201 conversations audited, only 1,290 (40.3%) were confirmed correct; 1,594 (49.8%) required relabeling; and 317 (9.9%) were skipped due to low confidence. The primary failure modes were: • False compliance detection: Phrases like “I don’t have my credit card right now” were incorrectly classified as compliance, when the context indicated refusal. • Missed successful scams: Victims who explicitly committed to action were labeled as partial compliance because the classifier failed to recognize strong commitment language. • Context-insensitive keyword matching: The word “scam” in cautious victim phrases triggered a scam detected label even when the victim was still engaging.
TABLE III M ODELS EVALUATED WITH FEATURE TYPES Model Logistic Regression Random Forest Random Forest
Feature Type TF-IDF + features
Linear SVM XGBoost XGBoost
TF-IDF + features Features only TF-IDF + features
DistilBERT
Contextual embeddings Contextual embeddings
Longformer
Features only TF-IDF + features
Description Linear, L2 regularization 100-tree ensemble 100-tree ensemble, combined Linear kernel Gradient boosting Gradient boosting, combined Transformer, 66M params Transformer, 148M params
2) Feature Representations: We compared two feature extraction approaches: (1) Engineered Features — 28 numeric features including conversation length metrics, behavioral indicators (resistance/compliance counts and ratios), temporal patterns, financial signals, victim demographics, and stylistic features; and (2) TF-IDF Features — Term Frequency–Inverse Document Frequency vectors computed over the full conversation text using unigrams and bigrams with a maximum of 5,000 features and sublinear TF scaling, combined with the 28 engineered features. 3) Models: We trained six scikit-learn classifiers and finetuned two transformer models, spanning diverse algorithmic families, as summarized in Table III. B. Baseline Results Fig. 4. Dataset Snapshot.
Correction Pipeline: We developed an improved audit pipeline (v2) incorporating negation-aware matching, scamtype-specific heuristics, and weighted analysis of the final conversation turns. Of the 1,594 relabeled conversations, 1,339 (84.0%) were medium-confidence corrections and 255 (16.0%) were high-confidence corrections. Relabeling affected all eight scam categories, with grandparent scams (232 changes), romance scams (225), and virtual kidnapping (218) requiring the most corrections. Original labels were preserved in metadata for traceability. Manual validation of 28 stratified samples confirmed corrected labels aligned with human judgment in over 90% of cases. V. E XPERIMENTATION A. Experimental Setup 1) Data Splits: The corrected dataset was partitioned into stratified train (80%), validation (10%), and test (10%) splits using a random seed of 42, maintaining proportional representation of both scam categories and outcome classes. This yielded 2,556 training conversations (20,490 turns), 321 validation conversations (2,561 turns), and 324 test conversations (2,592 turns).
Fig. 5 presents the performance of all eight models on the held-out test set. The best-performing model was XGBoost with TF-IDF + engineered features, achieving 72.5% accuracy and a macro F1 score of 0.691. This substantially outperformed all other configurations, including both transformer models. The performance gap between XGBoost configurations is striking: XGBoost with TF-IDF achieved 72.5% accuracy compared to only 56.5% with engineered features alone — an improvement of 16.0 percentage points. This demonstrates that lexical patterns carry critical discriminative signal that hand-crafted features fail to capture. C. Error Analysis The confusion matrix depicted in Figure 6 for the XGBoost + TF-IDF reveals the following patterns: The partial class achieves the highest recall (89.7%), benefiting from larger representation and uniform language patterns. The complied class achieves 62.3% recall, with most errors involving misclassification as partial (16 of 53 cases) — expected since conversations where victims eventually comply contain extended engagement similar to partial compliance. The rejected class achieves 64.2% recall, with 28 of 95 cases misclassified as partial, reflecting lexical overlap between extensive engagement before rejection and engagement without commitment. The model achieves its highest precision on
Accuracy
150
Macro F1
0.8
True Label
Score
0.6 0.4 0.2 0 F ID TF R-
L
t t F F er ea ea RT DF ID ID -F -F FI orm BE f B l TF T TF i g RF t n s B M XG RF Lo Di XG SV
Fig. 5. Model comparison showing accuracy and macro F1 for all eight models on 3-class outcome prediction. The dashed horizontal line represents the random baseline (0.333). 150
True Label
Complied (n=53)
Partial (n=174)
Rejected (n=95)
33 (62.3%)
9 (5.2%)
16 (30.2%)
156 (89.7%)
partial comp.
152
1
0
10
11
quick rej.
0
2
0
0
0
scam det.
4
0
0
3
1
success scam
13
0
0
37
3
verif attempt
20
0
2
4
59
rif ve
em att
pt
t.
m
ss ce
c su
sca
m sca
de
k uic
. rej
q
r
pa
. mp co l a ti
100
50
0
Predicted Label Fig. 7. Five-class confusion matrix for XGBoost + TF-IDF (test set). Minority classes scam detected and quick rejection show near-zero recall.
4 (7.5%)
100
verification behavior (e.g., “official,” “hang up,” “the card”) and scam awareness (e.g., “be aware,” “trick,” “due diligence”) were most discriminative.
50
F. Transformer Results
0
To investigate whether contextual language models can improve upon bag-of-words representations, we fine-tuned two transformer architectures on the same 3-class task.
9 (5.2%)
6 (6.3%)
28 (29.5%)
61 (64.2%)
Comp.
Part.
Rej.
Predicted Label Fig. 6. Confusion matrix for XGBoost + TF-IDF on the 3-class test set (n = 324). Percentages indicate row-wise recall.
rejected (82.4%), indicating high confidence when predicting rejection. D. Five-Class Analysis We also evaluated all models on the original five-class labels. The best model remained XGBoost + TF-IDF, achieving 72.5% accuracy but a substantially lower macro F1 of 0.614. The drop is attributable to near-zero performance on minority classes as depicted in Figure 7: quick rejection (n = 2 in test) and scam detected (n = 8 in test) have insufficient samples for reliable classification. This motivated the 3-class collapsed evaluation as primary. E. Feature Importance The top features identified by XGBoost + TF-IDF include both engineered and lexical features. The topranked engineered features were victim turn count and attacker turn count, confirming that conversation length is a strong predictor. Among TF-IDF features, terms related to
1) DistilBERT: We fine-tuned DistilBERT-base-uncased (66M parameters) for 8 epochs with a learning rate of 2×10−5 , batch size 16, and balanced class weights. DistilBERT has a maximum input length of 512 tokens, but our conversations average 377 tokens with a maximum of 874. We evaluated three truncation strategies: head-only (default, keeping the first 512 tokens), tail-only (keeping the last 512 tokens). The tail-only (v2) strategy performed best, achieving 69.8% accuracy and 0.674 macro F1 on the test set. The tail-only strategy retains the outcome-critical final turns, and consistently outperformed head-only configurations across validation runs. 2) Longformer: To isolate the impact of truncation, we fine-tuned Longformer-base (148M parameters, 4,096-token capacity) with a maximum input length of 1,024 tokens — sufficient to fit all conversations without any truncation. We used a batch size of 4, learning rate of 2 × 10−5 , and trained for 8 epochs with balanced class weights. Longformer achieved 69.8% accuracy and 0.667 macro F1, substantially outperforming all DistilBERT configurations but falling short of XGBoost + TF-IDF. Table IV summarizes the complete results across all eight model configurations. XGBoost + TF-IDF remains the bestperforming model, with Longformer as the strongest transformer baseline.
TABLE IV C OMPLETE 3- CLASS TEST RESULTS (n = 324). B EST IN BOLD . Model XGBoost + TF-IDF Longformer (1024 tok.) DistilBERT (tail, v2) Linear SVM + TF-IDF Logistic Reg. + TF-IDF Random Forest + TF-IDF Random Forest (feat.) XGBoost (feat. only)
Accuracy
F1 Macro
F1 Weighted
0.725 0.698 0.698 0.679 0.679 0.682 0.605 0.565
0.691 0.667 0.674 0.667 0.659 0.652 0.571 0.534
0.718 0.701 0.695 0.678 0.676 0.678 0.593 0.557
VI. D ISCUSSION A. Dataset Contribution The primary contribution of this work is the synthetic dataset itself. With 3,201 conversations across eight scam categories, this represents — to our knowledge — the largest publicly available multi-turn social engineering conversation dataset targeting elderly populations. The multi-agent generation approach produces conversations with natural turn-taking dynamics, varied resistance patterns, and realistic outcomes that would be impractical to collect from real-world interactions due to privacy and ethical constraints. The 49.8% initial label mismatch rate underscores an important finding: automated outcome labeling of conversational data is substantially harder than labeling single messages. Context-dependent phrases, negation, and the temporal progression of a conversation all contribute to labeling difficulty. Our improved audit pipeline demonstrates that systematic, heuristic-based correction can substantially improve label quality, but also highlights the need for future work on more sophisticated labeling approaches. B. Baseline Analysis The 72.5% accuracy achieved by XGBoost + TF-IDF on a 3-class prediction task represents a meaningful baseline above the 33.3% random chance level, but leaves substantial room for improvement. The dominance of TF-IDF over engineered features (16.0-point accuracy gap for XGBoost) demonstrates that lexical patterns carry critical discriminative signal that hand-crafted features fail to capture. Contrary to initial expectations, transformer-based models did not outperform TF-IDF-based classifiers on this dataset. We attribute this to two factors. First, input length limitations: DistilBERT’s 512-token limit forces truncation of conversations that average 377 tokens with a maximum of 874. This truncation discards critical late-conversation signals — the compliance or rejection indicators that determine the outcome label. The progressive improvement from DistilBERT headonly to tail-only (69.8%) to Longformer with no truncation (69.8%) confirms that context window size directly impacts performance. Second, dataset scale: with only 2,556 training conversations, transformer models lack sufficient data for effective fine-tuning. Longformer’s training loss fell to 0.18 by epoch 8 while validation loss climbed to 2.30, exhibiting clear
overfitting characteristic of data-limited regimes. In contrast, XGBoost operates over pre-computed TF-IDF features and does not need to learn language representations from scratch, making it inherently more sample-efficient. These findings suggest that for small conversational datasets, traditional ML with TF-IDF features provides a strong and practical baseline. Transformer models would likely benefit from larger training corpora (>5,000 conversations) and longer-context architectures, which we identify as directions for future work. The confusion between partial and rejected classes remains a challenge across all model families. Victims who engage at length before ultimately rejecting a scam produce dialogue that is lexically similar to those who engage without ever fully committing. This boundary may require turn-level temporal modeling to resolve effectively. C. Limitations Several limitations should be acknowledged. First, the conversations are synthetic and may not fully capture the linguistic patterns of real scam interactions, particularly the emotional dynamics of actual elderly victims. Second, the dataset is English-only and US-centric. Third, the label correction pipeline was validated on a relatively small stratified sample (28 conversations) and may contain residual errors — 317 low-confidence conversations (9.9%) were left unchanged. Fourth, the current baseline models treat each conversation as a complete document, whereas a real-time detection system would need to classify incrementally. Fifth, the 3-class collapse merges meaningful behavioral distinctions that may be important for intervention systems. Sixth, the dataset size of 3,201 conversations may be insufficient for effective transformer fine-tuning, as evidenced by the overfitting observed in both DistilBERT and Longformer experiments. VII. C ONCLUSION AND F UTURE W ORK This paper presents three contributions to the field of AI-driven social engineering defense. First, we introduce a multi-agent LLM framework for generating synthetic scam conversations and the resulting dataset of 3,201 conversations across eight elder-targeted scam categories — the first publicly available resource of its kind. Second, we establish detection baselines using eight models spanning traditional ML and transformer architectures, with XGBoost + TF-IDF achieving 72.5% accuracy and 0.691 macro F1 on 3-class outcome prediction. Third, we provide an empirical analysis showing that TF-IDF-based models outperform fine-tuned transformers on conversational datasets of this scale, identifying input length limitations and training data size as the primary constraining factors. Future work will proceed along three directions. First, we plan to expand the dataset to over 10,000 conversations to investigate whether larger training corpora enable transformer models to surpass traditional ML baselines. Second, we will explore incremental classification — predicting conversation
outcomes from partial transcripts to enable real-time intervention. Third, we intend to expand the dataset with additional scam categories and non-English conversations to improve generalizability. The COVA conversational smishing dataset, generation pipeline, and detection baselines will be made publicly available upon acceptance of this paper.
[20] L. D. Herrera, L. V. Sickle, and A. Podhradsky, “Bridging the protection gap: Innovative approaches to shield older adults from ai-enhanced scams,” Dakota State University, Tech. Rep., 2024. [21] FBI Internet Crime Complaint Center, “2023 elder fraud report,” FBI IC3, Tech. Rep., 2024.
R EFERENCES [1] G. Desolda, L. S. Ferro, A. Marrella, T. Catarci, and M. F. Costabile, “Human factors in phishing attacks: a systematic literature review,” ACM Computing Surveys (CSUR), vol. 54, no. 8, pp. 1–35, 2021. [2] H. Xu, A. Qadir, and S. Sadiq, “Malicious sms detection using ensemble learning and smote to improve mobile cybersecurity,” Computers & Security, vol. 154, p. 104443, 2025. [3] Keepnet Labs, “Smishing statistics: The latest trends and numbers in sms phishing,” 2024, accessed: 2026-03-12. [Online]. Available: https://keepnetlabs.com/blog/ smishing-statistics-the-latest-trends-and-numbers-in-sms-phishing [4] CNET, “Americans lost $470 million to text scams in 2024: Here’s how to protect yourself,” 2024, accessed: 2026-03-12. [Online]. Available: https://www.cnet.com/tech/services-and-software/ americans-lost-470-million-to-text-scams-in-2024-heres-how-to-protect-yourself/ [5] A. Alhuzali et al., “Phishnet: A real-time, scalable ensemble framework for smishing attack detection using transformers and llms,” Computers, Materials & Continua, vol. 86, no. 1, pp. 1–?, 2026. [6] S. K. Birthriya, P. Ahlawat, and A. K. Jain, “Machine learning-based smishing detection using fuzzy logic and tf-idf feature engineering,” Franklin Open, p. 100506, 2026. [7] G. Li, K. Selvaraj, and M. Singh, “Collaborative dual-framework defense: Cti and llm-based enhanced smishing detection,” International Journal of Advanced Computer Science & Applications, vol. 17, no. 1, 2026. [8] Ent Credit Union, “How to spot financial grooming scams,” May 2025, accessed: 2026-03-12. [Online]. Available: https://www.ent.com/education-center/security-center/ how-to-spot-financial-grooming-scams/ [9] J. W. Seo et al., “On-device smishing classifier resistant to text evasion attack,” IEEE Access, vol. 12, pp. 4762–4779, 2024. [10] C. Patra, D. Giri, M. S. Obaidat, and T. Maitra, “Smsdect: A prediction model for smishing attack detection using machine learning and text analysis,” in Proc. IEEE Global Communications Conference (GLOBECOM), 2023, pp. 3837–3842. [11] A. Derakhshan, I. G. Harris, and M. Behzadi, “Detecting telephonebased social engineering attacks using scam signatures,” in Proceedings of the 2021 ACM workshop on security and privacy analytics, 2021, pp. 67–73. [12] I. D. Wood, M. Kepkowski, L. Zinatullin, T. Darnley, and M. A. Kaafar, “An analysis of scam baiting calls: Identifying and extracting scam stages and scripts,” Macquarie University, Tech. Rep., 2023. [13] M. Lansley, F. Mouton, S. Kapetanakis, and N. Polatidis, “Seader++: Social engineering attack detection in online environments using machine learning,” Journal of Information and Telecommunication, vol. 4, no. 3, pp. 346–362, 2020. [14] A. K. Jain, K. Kaur, N. K. Gupta, and A. Khare, “Detecting smishing messages using bert and advanced nlp techniques,” SN Computer Science, vol. 6, no. 2, p. 109, 2025. [15] C. Oswald, S. E. Simon, and A. Bhattacharya, “Spotspam: Intention analysis–driven sms spam detection using bert embeddings,” ACM Transactions on the Web (TWEB), vol. 16, no. 3, pp. 1–27, 2022. [16] O. N. Akande, H. B. Akande, A. A. Kayode, A. A. Adeyinka, F. Olaiya, and G. Oluwadara, “Development of a real time smishing detection mobile application using rule based techniques,” Procedia Computer Science, vol. 199, pp. 95–102, 2022. [17] N. Basta, C. Atkins, and D. Kaafar, “Bot wars evolved: Orchestrating competing llms in a counterstrike against phone scams,” Macquarie University, Tech. Rep., 2025. [18] T. Kumarage et al., “Personalized attacks of social engineering in multiturn conversations: Llm agents for simulation and detection,” Arizona State University and Columbia University, Tech. Rep., 2024. [19] D. Spokoyny, N. Vogler et al., “Victim as a service: Designing a system for engaging with interactive scammers,” University of California, San Diego, Tech. Rep., 2025.