LLM-based annotation and token-augmented modeling for emotional tone classification in online cancer peer-support posts.

LLM-based annotation and token-augmented modeling for emotional tone classification in online cancer peer-support posts - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice PLOS Digit Health . 2026 May 29;5(5):e0001235. doi: 10.1371/journal.pdig.0001235 LLM-based annotation and token-augmented modeling for emotional tone classification in online cancer peer-support posts Shuo Xu 1, # , Zhongyan Wang 2, # , Hailiang Wang 3, ‡ , Zhanyi Ding 2, ‡ , Yue Zou 4, ‡ , Yuchen Cao 5, ‡, * Editor: Zahra Hoodbhoy 6 Author information Article notes Copyright and License information 1 Department of Computer Science and Engineering, University of California San Diego, La Jolla, California, United States of America 2 Center for Data Science, New York University, New York, New York, United States of America 3 College of Computing, Georgia Institute of Technology, Atlanta, Georgia, United States of America 4 Fu Foundation School of Engineering and Applied Science, Columbia University, New York, New York, United States of America 5 Khoury College of Computer Sciences, Northeastern University, Seattle, Washington, United States of America 6 Aga Khan University, PAKISTAN ✉ * E-mail: cao.yuch@northeastern.edu The authors have declared that no competing interests exist. ‡ These authors also contributed equally to this work. # Contributed equally. Roles Shuo Xu : Data curation, Methodology, Software, Writing – original draft, Writing – review & editing Zhongyan Wang : Investigation, Methodology, Software, Writing – original draft Hailiang Wang : Data curation, Formal analysis, Software, Writing – original draft Zhanyi Ding : Data curation, Methodology, Supervision, Writing – review & editing Yue Zou : Data curation, Methodology, Validation, Writing – review & editing Yuchen Cao : Conceptualization, Formal analysis, Methodology, Project administration, Writing – original draft, Writing – review & editing Zahra Hoodbhoy : Editor Received 2026 Jan 26; Accepted 2026 May 8; Collection date 2026 May. © 2026 Xu et al This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. PMC Copyright notice PMCID: PMC13221046  PMID: 42213728 Abstract Online cancer peer-support communities generate large volumes of patient-authored and caregiver-authored text that may reflect distress, coping, and informational needs. Automated emotional tone classification could support scalable monitoring, but supervised modeling depends on label quality and may benefit from explicit context features. Using the Mental Health Insights: Vulnerable Cancer Survivors & Caregivers dataset, we compared five model families (TF–IDF Logistic Regression, Random Forest, LightGBM, GRU, and fine-tuned ALBERT) on a three-class target (Negative/Neutral/Positive) derived from four original categories. We introduced two extensions: (i) LLM-based annotation to generate parallel “AI labels” and (ii) token-based augmentation that prepends LLM-extracted structured variables (reporter role and cancer type) to the post text. Models were trained with a 60/20/20 stratified train/validation/test split, with hyperparameters selected on validation data only. Test performance was summarized using weighted F1 and macro one-vs-rest AUC with bootstrap confidence intervals, with paired comparisons based on McNemar tests and false discovery rate adjustment. The LLM annotator produced substantial redistribution in the four-class label space, shifting prevalence toward very negative relative to the original labels; the shift persisted but attenuated after collapsing to three classes. Across all model families, token augmentation improved held-out performance, with the largest gains for GRU and consistent improvements for ALBERT. Augmentation also reduced polarity-reversing errors (Negative ↔ Positive) for ALBERT, while adjacent errors (Negative ↔ Neutral) remained the dominant residual failure mode. These results indicate that LLM-based supervision can introduce systematic measurement shifts that require auditing, yet LLM-extracted context incorporated via simple token augmentation provides a pragmatic, model-agnostic mechanism to improve downstream emotional tone classification for supportive oncology decision support. Author summary We studied how to better monitor emotional tone in posts from online cancer peer-support communities, where patients and caregivers share experiences that may signal distress, coping, or unmet needs. Automated classification could help organizations and moderators identify when additional support may be needed, but these systems depend on the quality of the labels used for training and may miss clinical context. Using a public dataset of cancer survivor and caregiver posts, we trained and compared several machine-learning and deep-learning models to classify each post as negative, neutral, or positive. We tested two practical improvements. First, we used a large language model to generate an additional set of “AI labels” and examined how these differed from the original categories. Second, we extracted simple context information—whether the writer was a patient or caregiver and what cancer type was mentioned—and added this context to the text before model training. We found that adding context consistently improved performance across model types. However, the AI-generated labels shifted class distributions, indicating that automated labeling can introduce systematic changes that should be audited. Overall, simple context extraction can make emotional tone monitoring more accurate and useful for supportive oncology decision support. 1 Introduction Cancer survivorship and treatment are frequently accompanied by substantial psychosocial burden, affecting both patients and informal caregivers. As survivorship populations grow, supportive care and distress monitoring have become central to quality oncology care, yet practical screening and follow-up remain resource constrained [ 1 , 2 ]. In parallel, many individuals turn to online peer-support communities to seek information, disclose symptoms, and process uncertainty. These posts constitute a form of patient-generated health data (PGHD) that can complement traditional clinical touchpoints, but their volume and heterogeneity make manual review difficult to sustain at scale [ 3 , 4 ]. Natural language processing (NLP) offers a pathway to summarize and triage supportive-care needs from non-clinical text, and sentiment/emotion modeling has become a common first step for identifying distress-related language in social media and other user-generated content [ 5 – 7 ]. Recent work has examined trade-offs between machine learning and deep learning approaches for mental illness detection on social media [ 8 ], highlighting the importance of systematic model comparison, and a systematic review has catalogued recurring methodological biases across the ML lifecycle, including inconsistent hyperparameter tuning, inadequate data partitioning, and insufficient handling of class imbalance [ 9 ]. Within oncology specifically, prior computational studies have applied lexicon-based scoring, classical machine learning, and fine-tuned transformers to cancer-related text from social media, patient forums, and clinical notes. For example, LSTM-based models have been used to classify sentiment in cancer community tweets [ 10 ]; BERT has been applied to extract patient worries from breast cancer blogs [ 11 ]; and recent work has examined how poster–patient relationships shape language and sentiment on social media platforms for breast cancer [ 12 ]. More recently, LLMs have been explored for cancer patient and caregiver support, including symptom triage and information delivery [ 13 ], and context-aware emotion analysis in cancer survivor diaries [ 14 ]. Despite this growing body of work, two specific gaps persist. First, existing oncology NLP studies have largely treated post text as the sole model input, without systematically incorporating structured clinical context, such as whether the author is a patient or caregiver and which cancer type is discussed, that may alter the semantics of emotional language (e.g., caregiving burden versus patient symptom distress). Second, while LLMs have been evaluated as question-answering tools and information extractors in oncology, their use as annotation instruments for emotional tone in cancer peer-support text has not been empirically characterized, and the distributional shifts they may introduce relative to existing labels remain unaudited. Recent work has proposed using large language models (LLMs) as scalable annotators for text classification tasks, sometimes achieving strong agreement with human judgments while reducing labeling time and cost [ 15 ]. At the same time, LLM-based labeling can introduce systematic shifts in decision boundaries, particularly for subjective constructs such as emotional intensity, and may exhibit overconfidence or error modes that are not immediately evident without auditing [ 16 , 17 ]. For mental-health-adjacent content, such shifts are especially consequential: a more conservative rater may inflate negative prevalence, alter apparent community needs, and bias models trained or evaluated on those labels. Thus, when LLMs are used to generate labels, it is important not only to report predictive performance but also to characterize label distribution changes and to study how downstream models respond under controlled evaluation. In this work, we investigate emotional tone classification in online cancer peer-support posts using the Mental Health Insights: Vulnerable Cancer Survivors & Caregivers dataset [ 18 ]. The dataset provides author-reported emotional tone labels across four categories (very negative, negative, neutral, positive). We extend this setting by applying an LLM annotator to produce (i) a parallel set of emotional tone labels (AI labels) and (ii) lightweight structured variables extracted from the same text (reporter role and cancer type). This design enables a focused methodological study of two questions: how LLM labeling reallocates emotional tone relative to existing labels, and whether LLM-derived structured context can be incorporated to improve downstream classification. Methodologically, we compare a range of commonly used model families for text classification, spanning TF–IDF-based classical baselines and neural encoders, including a fine-tuned transformer (ALBERT) [ 19 , 20 ]. We further evaluate a simple, model-agnostic mechanism for leveraging LLM-extracted variables: token-based augmentation, in which deterministic prefix tokens (e.g., ROLE_PATIENT , CANCER_LUNG ) are prepended to the post text. This approach allows structured context to be consumed by both sparse vectorizers and neural tokenizers without introducing multi-branch architectures, providing a pragmatic template for integrating upstream information extraction with downstream screening. Existing evaluations of LLM-based annotation have primarily quantified agreement with human raters or assessed cost–quality trade-offs in general NLP benchmarks [ 15 ]. Fewer studies have examined how LLM labeling reshapes class distributions in health-specific text, and none, to our knowledge, have jointly (a) characterized the distributional shift introduced by LLM annotation in a clinical-adjacent domain, (b) leveraged the same LLM as a structured-context extractor whose outputs are incorporated into downstream classification via token augmentation, and (c) analyzed the ordinal error structure of resulting classifiers to assess operational consequences for supportive care screening. This three-part combination is the methodological contribution of the present work. The primary contribution of this work is methodological, with supportive oncology decision support serving as the motivating application domain rather than the object of validation. Our contributions are threefold. First, we provide an empirical characterization of human versus LLM label distributions under both the original four-class scheme and a three-class collapse used for modeling, highlighting the extent and nature of labeling shift. Second, we quantify the effect of token-based augmentation across multiple model families under a controlled train/validation/test design, including uncertainty estimates and paired statistical comparisons. Third, we analyze error structure with attention to ordinal distance (adjacent vs. polarity-reversing errors), emphasizing practical implications for decision-support deployment in supportive oncology settings. Collectively, these results aim to clarify when LLMs function as helpful annotators and context extractors, when they induce systematic artifacts, and how simple augmentation strategies can improve robustness and minority-class behavior in emotionally salient health-related text. 2 Methods We conducted a comparative modeling study to classify the emotional tone of posts from online cancer peer-support communities. The overall experimental framework, including data source, train/validation/test splitting, preprocessing decisions by model family, model families, and evaluation procedures, is described in full below. The substantive extensions in this study are that (i) prediction targets were derived from an LLM-based annotation procedure (AI labels) and (ii) selected LLM-derived structured variables were incorporated as additional model inputs via token-based augmentation. Hyperparameter selection was performed using a held-out validation set only, and final performance was reported on an untouched test set. 2.1 Computational environment All experiments were implemented in Python. Data manipulation used NumPy and Pandas. Classical machine learning components (data partitioning, TF–IDF construction, model fitting, and hyperparameter search) were implemented with scikit-learn [ 21 ]. Gradient boosting was implemented using LightGBM [ 22 ]. Neural network models were developed in PyTorch , and transformer fine-tuning used the Hugging Face transformers library with the pretrained albert-base-v2 checkpoint [ 20 ]. Text standardization utilities used Python regular expressions and NLTK resources [ 23 ]. Training progress was tracked with tqdm . When available, CUDA acceleration was used for neural model training. The GRU architecture followed the standard gated recurrent unit formulation [ 24 ]. 2.2 Ethical considerations and data governance This study is a secondary analysis of a publicly available, de-identified dataset [ 18 ] and does not involve direct interaction with human participants; institutional review board (IRB) approval was therefore not required under the Common Rule (45 CFR 46.104(d)(4)). The original dataset was collected from public online cancer support communities and released under a CC BY 4.0 license. LLM-based annotation was performed via API on de-identified text using a deterministic configuration (temperature = 0), and no outputs were used for clinical decision-making or individual-level intervention. We recognize that automated annotation of health-related content carries risks, including systematic misclassification that could affect downstream interpretation of community needs. To mitigate these risks, we (i) frame all model outputs as decision-support tools requiring human oversight rather than autonomous classifiers (see Discussion), (ii) report label distribution comparisons to make annotation behavior transparent, and (iii) publicly share the annotated dataset and analysis code to enable independent auditing and replication. 2.3 Dataset We used the Mental Health Insights: Vulnerable Cancer Survivors & Caregivers dataset [ 18 ], a de-identified corpus of user-authored posts from online cancer support communities. The annotated dataset produced in this study, including LLM-derived labels, structured variables, and train/validation/test split assignments, is publicly available at GitHub and mendeley . Each instance consists of a free-text post and an author-provided emotional tone label spanning four categories: very negative , negative , neutral , and positive . In this study, we additionally applied an LLM annotator to the raw post text to generate (i) a parallel set of emotional tone labels (AI labels) and (ii) lightweight structured variables used for text augmentation (reporter role and cancer type). 2.4 Four-class annotation vs. three-class modeling target Although the source dataset is four-class, our downstream predictive experiments use a three-class target obtained by collapsing VERY_NEGATIVE and NEGATIVE into a single Negative class, while retaining NEUTRAL and POSITIVE as distinct classes. This reduction is motivated by three considerations. First, from a measurement perspective, the label distribution analysis (Section 3 ; Table 1 ) demonstrates that the boundary between VERY_NEGATIVE and NEGATIVE is the primary locus of human–LLM disagreement: the LLM assigned 46.3% of posts to very negative compared to 11.1% under human labels. After collapsing, the Jensen–Shannon divergence between human and AI distributions decreased from 0.1155 (four-class) to 0.0351 (three-class), confirming that most labeling disagreement was concentrated within the negative subspace. Second, from a clinical and operational standpoint, the primary screening decision in supportive oncology is whether a post signals distress (warranting review) versus neutral informational content versus positive coping; finer intensity grading within the negative spectrum is less actionable for triage workflows. Third, collapsing adjacent negative subcategories is standard practice in affect modeling for health-related social media, where such boundaries are often unstable across annotation protocols [ 25 , 26 ]. For the original (human) labels, the three-class target was derived by deterministic remapping of the intensity scale. For the AI labels, the three-class target was derived by first summing the VERY_NEGATIVE and NEGATIVE probabilities from the four-element distribution into a single Negative probability, then assigning the argmax of the resulting three-element vector. Both procedures yield the same three-class label space using the mapping Table 1. Dataset Characteristics and Class Distributions. Characteristic Human Labels AI Labels Overall Total posts ( N ) 10,392 Post length, words: mean (SD) 234.1 (289.1) Post length, words: median [IQR] 162 [93, 276] Post length, words: range 1–5,413 Four-class distribution, n (%) Very negative 1,155 (11.1%) 4,810 (46.3%) Negative 4,112 (39.6%) 2,591 (24.9%) Neutral 4,375 (42.1%) 2,498 (24.0%) Positive 750 (7.2%) 493 (4.7%) Three-class distribution (modeling target), n (%) Negative 5,267 (50.7%) 7,877 (75.8%) Neutral 4,375 (42.1%) 2,026 (19.5%) Positive 750 (7.2%) 489 (4.7%) Train/validation/test partition Train 6,234 (60.0%) Validation 2,079 (20.0%) Test 2,079 (20.0%) Open in a new tab Human labels are derived from author-provided intensity scores (–2–1). Four-class AI labels are defined as the argmax of the LLM’s four-element probability vector. Three-class AI labels (the modeling target) are derived by first summing VERY_NEGATIVE and NEGATIVE probabilities into a single Negative probability, then taking the argmax of the three-element distribution. Post length is computed as whitespace-delimited word count on the raw post text. Partitions are stratified by the three-class AI label with random_state = 42 . 0 = Negative ( VERY_NEGATIVE ∪ NEGATIVE ) , 1 = Neutral ( NEUTRAL ) , 2 = Positive ( POSITIVE ) . Unless otherwise stated, all reported predictive results use this three-class target. The original four-class labels and the LLM-produced four-class probabilities were retained for descriptive analyses of labeling behavior (Section 3 ). 2.5 Train/validation/test splitting We used a two-stage holdout design. First, 20% of the dataset was reserved as the final test set. The remaining 80% was split into training and validation subsets, producing an overall 60/20/20 train/validation/test split. All splits were generated with a fixed seed ( random_state = 42 ). To preserve class proportions under imbalance, splits were stratified by the three-class target. The validation set was used for hyperparameter selection; the test set was used only for final reporting. 2.6 Text processing and representations Model-family preprocessing. Preprocessing was aligned to common practice for different model families. For sparse lexical baselines (Logistic Regression, Random Forest, LightGBM), models were trained on TF–IDF representations derived from the input text with built-in English stopword removal and without additional lemmatization. For neural models (GRU and ALBERT), we applied a lightweight standardization pipeline: text was lowercased; URLs, user mentions, and hashtags were replaced with generic placeholders via regular expressions; non-alphanumeric characters were removed; and whitespace was normalized. This approach reduces noisy artifacts while preserving contextual cues for neural tokenization. Feature construction. For classical models, TF–IDF vectors were built using up to 10,000 features, including unigrams and bigrams ( ngram_range=(1,2) ). For neural models, ALBERT used AlbertTokenizer with a maximum sequence length of 200 tokens, and the GRU model used a vocabulary of the top 10,000 tokens with sequences padded/truncated to length 200. 2.7 LLM-based labeling and structured variable extraction Annotator and protocol. We used an LLM ( gpt-4o-mini ) as an automated annotator to extract emotional tone labels and a small set of auxiliary variables from each post. The annotator was instructed to avoid clinical diagnosis or medical advice and to base outputs strictly on the provided text. Posts were provided in batched JSON format (20 posts per batch), and the model returned a JSON array of structured outputs in the same order. To maximize completeness under occasional formatting failures, we implemented a recursive split-and-retry strategy in which a failed batch was subdivided until valid JSON was recovered or a single-post failure remained. Outputs were written with checkpointing to enable safe resumption. Ordering relative to data splitting. LLM-based labeling and structured variable extraction were applied to the full dataset ( N = 10,392) prior to the train/validation/test split. This ordering treats LLM annotation as a fixed data-preparation step—analogous to human annotation—that does not depend on or leak information from the evaluation partitions. The LLM prompt design, model version, and decoding parameters (temperature = 0) were identical across all posts; no validation- or test-set information influenced the annotation protocol. AI labels (four-class probabilities and discrete label). For each post, the annotator produced emo_probs , a probability distribution over four emotional tone categories: VERY_NEGATIVE , NEGATIVE , NEUTRAL , and POSITIVE , with probabilities constrained to be non-negative and to sum to 1 (within rounding tolerance). For the three-class predictive task, probabilities were first collapsed by summing VERY_NEGATIVE and NEGATIVE into a single Negative probability, and the three-class AI label was then assigned as the argmax of the resulting three-element distribution. This collapse-then-argmax procedure was applied consistently across all modeling conditions. For the four-class label shift analysis (Section 3 ), a discrete four-class label was defined as the argmax of the original four-element probability vector, using the mapping 0=VERY_NEGATIVE , 1=NEGATIVE , 2=NEUTRAL , 3=POSITIVE . Auxiliary variables for augmentation. In addition to emotional tone, the annotator extracted two auxiliary variables used for augmentation: (i) reporter role ( PATIENT , CAREGIVER , UNCLEAR ) and (ii) cancer type ( BRAIN , COLON , LIVER , LEUKEMIA , LUNG , OTHER , UNKNOWN ), along with an associated confidence score (0–1). Evidence strings returned by the annotator were retained for qualitative auditing but were not used as model inputs. Importantly, these structured variables are extracted from the same post text and therefore do not introduce external information; rather, they emulate an upstream information-extraction step that produces coarse context features to support downstream screening. Reliability of extracted variables. Formal inter-rater validation of the LLM-derived structured variables was not conducted. However, the cancer type distribution aligns with the known composition of the source dataset, which was curated from communities organized around five specific cancer types [ 18 ], and the cancer_type_confidence score exhibits appropriate calibration (posts assigned to UNKNOWN had systematically lower confidence). 2.8 Modeling conditions and token-based augmentation We fit two modeling conditions that differed only in the input text field consumed by the learning algorithms. Regular (text-only) models. In the regular condition, each model consumed the original post text ( posts ) without additional structured variables. Augmented models (reporter role and cancer type tokens). In the augmented condition, we prepended deterministic tokens encoding the LLM-derived reporter role and cancer type to the post text to form an augmented input string ( posts_aug ). This design allows structured variables to be consumed by both TF–IDF vectorizers and neural tokenizers without requiring multi-branch architectures. Token definitions and construction. Reporter role was encoded as ROLE_ < ROLE> (e.g., ROLE_PATIENT ). Values outside { PATIENT , CAREGIVER , UNCLEAR } were mapped to ROLE_UNCLEAR . Cancer type was encoded as CANCER_ < TYPE> (e.g., CANCER_LUNG ). Values outside the allowed set were mapped to CANCER_UNKNOWN . The final augmented input was constructed as posts _ aug = ROLE _ < ROLE > ‖ CANCER _ < TYPE > ‖ ( post text ) , where ‖ denotes string concatenation with a single separating space. All models in the augmented condition consumed posts_aug . For classical baselines, TF–IDF vectorization was applied directly to posts_aug , so covariate tokens entered the sparse vocabulary as additional lexical features. For neural models, posts_aug was passed through the same tokenization pipeline; covariate tokens therefore contributed as prefix tokens within the encoded sequence. 2.9 Models and hyperparameter optimization The five model families were selected to span the complexity–interpretability spectrum relevant to applied health-NLP deployment. TF–IDF Logistic Regression provides an interpretable linear baseline with strong performance on sparse text features. Random Forest and LightGBM represent non-linear ensemble methods that can capture feature interactions while remaining computationally efficient. The GRU offers a lightweight sequential neural encoder that processes token order but has fewer parameters than transformer architectures. ALBERT represents a parameter-efficient pretrained transformer that leverages transfer learning from large-scale corpora. This range allows assessment of whether token-based augmentation benefits are model-agnostic or concentrated in specific architecture classes, which has practical implications for deployment settings where computational resources, latency, or interpretability constraints may favor different model families. We evaluated five model families: multinomial Logistic Regression, Random Forest, LightGBM, a GRU sequence model, and a fine-tuned ALBERT transformer. To mitigate class imbalance, we used class-weighted objectives or balanced weighting options where supported. Classical models were tuned using grid search over standard regularization and tree/boosting parameters. Neural models (GRU and ALBERT) were tuned using random search over learning rate, dropout, and training epochs. For all model families, the primary selection criterion was validation weighted F1-score computed on the three-class target. The full search spaces and selected hyperparameters are summarized in Table 2 . Table 2. Hyperparameter Search Spaces for ML and DL Models. Model Hyperparameter Search Space Grid Search (manual enumeration on validation set) Logistic Regression Regularization ( C ) {0.1, 1, 10} Solver { lbfgs , saga } Class Weight { balanced , None } Penalty { l2 } (fixed) Multiclass { multinomial } (fixed) Random Forest Estimators ( n _ e s t i m a t o r s ) {50, 100, 200} Max Depth { None , 10, 20} Min Samples Split {2, 5, 10} Min Samples Leaf {1, 2, 4} Class Weight { balanced , None } LightGBM Estimators ( n _ e s t i m a t o r s ) {100, 200} Learning Rate {0.01, 0.1, 0.2} Number of Leaves ( num_leaves ) {31, 63} Max Depth { None , 10} Class Weight { balanced , None } Random Search (n = 10 samples) GRU Embedding Dimension uniform in [150, 250] Hidden Dimension uniform in [256, 768] Learning Rate log-uniform in [10 −4 , 10 −3 ] Epochs {5,6,7,8,9} ALBERT Learning Rate log-uniform in [10 −5 , 10 −4 ] Epochs {3,4,5} Dropout uniform in [0.0, 0.25] Open in a new tab 2.10 Evaluation and statistical comparison We reported accuracy and class-wise precision, recall, and F1-score, using weighted F1 as the primary aggregate metric. Discriminative ability was additionally summarized using macro-averaged one-vs-rest (OvR) ROC AUC. To quantify uncertainty in aggregate performance, 95% confidence intervals for weighted F1 and macro OvR AUC were computed using nonparametric bootstrapping with 2,000 resamples of the held-out test set. For each resample, metrics were recomputed and percentile-based intervals were formed from the empirical bootstrap distribution. For paired comparisons between models evaluated on the same test instances, we applied McNemar’s test to per-instance correctness using the continuity-corrected asymptotic formulation. To control multiplicity across pairwise tests within each modeling condition, p-values were adjusted using the Benjamini–Hochberg false discovery rate procedure, with significance assessed at α = 0.05 . 2.11 Ordinal-aware error analysis Although the downstream task uses a three-class target, emotional tone remains conceptually ordered from negative to positive. We therefore examined confusion matrices with attention to the distance of misclassification, distinguishing adjacent-category confusions from distant-category confusions to identify error modes with greater potential operational consequences, such as under-identification of positive coping language or misclassification of negative narratives as neutral. 3 Results We report (i) overall held-out test performance under regular (text-only) versus augmented (role+cancer-type tokens) inputs, (ii) class-specific effects of augmentation for the best-performing model family, and (iii) paired statistical comparisons based on per-instance correctness. Unless otherwise noted, results correspond to the three-class modeling target (Negative/Neutral/Positive) derived from the four-class label space as described in Section 2 . 3.1 Dataset overview Table 1 summarizes the dataset characteristics. The corpus comprised N = 10,392 posts with a median length of 162 words (IQR: 93–276). Under the original human labels, the four-class distribution was dominated by neutral (42.1%) and negative (39.6%), with very negative (11.1%) and positive (7.2%) as minority categories. The three-class modeling target (Negative/Neutral/Positive) yielded a 50.7%/42.1%/7.2% split under human labels and a 75.8%/19.5%/4.7% split under AI labels, reflecting the distributional shift examined in the following subsection. The stratified 60/20/20 split produced 6,234 training, 2,079 validation, and 2,079 test instances. 3.2 Human vs. LLM label distributions (4-class and collapsed 3-class) To characterize how the LLM annotator reallocated emotional tone relative to the dataset’s original labels, we compared class prevalence under the four-class scheme and under the downstream three-class collapse (Negative vs. Neutral vs. Positive). Substantial distributional shift was observed in the four-class comparison. Under human labels, the corpus was dominated by neutral (42.1%) and negative (39.6%), with very negative (11.1%) and positive (7.2%) as minority categories. In contrast, the LLM labeling reassigned nearly half of all posts to very negative (46.3%), while reducing the prevalence of negative (24.9%), neutral (24.0%), and positive (4.7%). This divergence was statistically pronounced (chi-square = 13022.21, p < 0.001) and reflected a moderate distributional distance (Jensen–Shannon divergence = 0.1155), indicating that the LLM applied a substantially different decision boundary for intensity between very negative and the mid-spectrum classes. When collapsing very negative and negative into a single Negative class (the modeling target derived via the collapse-then-argmax procedure described in Section 2 ), the shift remained but was attenuated. The human-label distribution was 50.7% Negative, 42.1% Neutral, and 7.2% Positive, whereas the AI distribution became 75.8% Negative, 19.5% Neutral, and 4.7% Positive. The collapsed comparison remained highly significant (chi-square = 2645.39, p < 0.001) with a Jensen–Shannon divergence of 0.0351, indicating that while most four-class disagreement was absorbed by the collapse, the LLM still assigned substantially more posts to Negative and fewer to Neutral/Positive than the human labels. Practically, this implies that analyses relying on the LLM labels will be more conservative with respect to distress detection (higher negative prevalence), while potentially under-representing neutral and positive content; downstream modeling and interpretation should therefore treat the AI labels as a methodological artifact and evaluate robustness to this labeling shift. 3.3 Overall test performance under regular vs. augmented inputs Table 3 summarizes test-set performance for each model family under two input conditions: (Panel A) regular text-only inputs and (Panel B) augmented inputs that prepend LLM-derived ROLE_ < ROLE> and CANCER_ < TYPE> tokens to each post. Across both panels, we report weighted-average precision and recall, along with weighted F1 and macro one-vs-rest (OvR) AUC with 95% bootstrap confidence intervals. Table 3. Overall Model Performance Summary on Test Set. Model Precision Recall Weighted F1 (95% CI) Macro AUC (95% CI) Panel A. Regular version Logistic Regression 0.8588 0.8615 0.8457 [0.8283, 0.8624] 0.8992 [0.8823, 0.9146] Random Forest 0.8495 0.8466 0.8406 [0.8237, 0.8561] 0.8684 [0.8476, 0.8885] LightGBM 0.8480 0.8567 0.8457 [0.8286, 0.8624] 0.8761 [0.8578, 0.8945] GRU 0.8072 0.7975 0.8013 [0.7835, 0.8181] 0.8051 [0.7800, 0.8295] ALBERT 0.8610 0.8393 0.8476 [0.8328, 0.8623] 0.9177 [0.9029, 0.9306] Panel B. Augmented version Logistic Regression 0.8629 0.8663 0.8513 [0.8339, 0.8682] 0.9059 [0.8895, 0.9215] Random Forest 0.8518 0.8581 0.8477 [0.8310, 0.8644] 0.8807 [0.8604, 0.8997] LightGBM 0.8485 0.8499 0.8472 [0.8317, 0.8635] 0.8870 [0.8678, 0.9046] GRU 0.8290 0.8201 0.8231 [0.8068, 0.8386] 0.8407 [0.8183, 0.8620] ALBERT 0.8654 0.8562 0.8598 [0.8454, 0.8748] 0.9284 [0.9164, 0.9399] Open in a new tab Precision and Recall correspond to weighted-average values on the test set. Weighted F1 and Macro AUC are reported with 95% confidence intervals. Across all five model families, augmentation improved aggregate performance on the held-out test set. Weighted F1 increased from Panel A to Panel B for Logistic Regression (0.8457 → 0.8513; Δ = + 0.0056 ), Random Forest (0.8406 → 0.8477; Δ = + 0.0071 ), LightGBM (0.8457 → 0.8472; Δ = + 0.0015 ), GRU (0.8013 → 0.8231; Δ = + 0.0218 ), and ALBERT (0.8476 → 0.8598; Δ = + 0.0122 ). Macro AUC increased in parallel: Logistic Regression (0.8992 → 0.9059; Δ = + 0.0067 ), Random Forest (0.8684 → 0.8807; Δ = + 0.0123 ), LightGBM (0.8761 → 0.8870; Δ = + 0.0109 ), GRU (0.8051 → 0.8407; Δ = + 0.0356 ), and ALBERT (0.9177 → 0.9284; Δ = + 0.0107 ). The uniform direction of these gains supports the central premise that lightweight structured context derived from the same post text (who is speaking and which cancer type is referenced) provides complementary signal that can be exploited by both sparse lexical models and neural encoders. Notably, GRU exhibited the largest absolute improvements on both weighted F1 and macro AUC, consistent with high-salience prefix tokens providing additional guidance to a lower-capacity sequential model. The consistency of these gains across all five model families also provides indirect evidence for the reliability of the LLM-extracted structured variables: if the derived reporter role and cancer type were predominantly noise, prepending them to the input text would not produce systematic improvements in discriminative performance across architecturally diverse classifiers. To contextualize these gains in practical terms: on a test set of 2,079 posts, the ALBERT augmentation improvement ( Δ weighted F1 = +0.012) corresponds to approximately 35 fewer misclassified posts, while the GRU improvement ( Δ = + 0.022 ) corresponds to approximately 53 fewer errors. In a monitoring deployment processing several thousand posts per week, reductions of this magnitude would meaningfully decrease the volume of incorrectly triaged content requiring human review. The clinical relevance of these gains is further amplified by their composition: as shown in the error analysis (Section 3 ), augmentation disproportionately reduced polarity-reversing errors (Negative ↔ Positive) for ALBERT, which represent the most operationally consequential failure mode in distress screening—cases where a post expressing acute distress would be classified as positive coping, or vice versa, potentially leading to missed intervention opportunities or misdirected resources. Model ordering remained broadly consistent across conditions. ALBERT achieved the highest point estimates of weighted F1 and macro AUC in both panels (Panel A: weighted F1 = 0.8476; macro AUC = 0.9177; Panel B: weighted F1 = 0.8598; macro AUC = 0.9284), indicating strong discriminative performance even prior to augmentation and additional gains after introducing context tokens. Differences among the top-performing approaches were modest in weighted F1, with overlapping confidence intervals among Logistic Regression, LightGBM, and ALBERT in both panels, suggesting that multiple architectures achieved comparable aggregate classification quality under this three-class target. Macro AUC more clearly separated ALBERT from classical baselines, indicating stronger global discrimination across classes. 3.4 Class-specific effects of augmentation in the best-performing model (ALBERT) To localize the source of aggregate gains, we examined class-specific precision, recall, and F1 for ALBERT under the regular and augmented conditions ( Table 4 ). Class-wise results indicate that augmentation reshaped error patterns rather than uniformly inflating performance. Table 4. ALBERT Class-Specific Performance on the Test Set (Regular vs. Augmented Inputs). Class Precision Recall F1-score Support Panel A. Regular version (text-only) Negative 0.9288 0.8999 0.9141 1609 Neutral 0.7139 0.6423 0.6762 369 Positive 0.3191 0.5941 0.4152 101 Accuracy 0.8393 2079 Macro avg 0.6539 0.7121 0.6685 2079 Weighted avg 0.8610 0.8393 0.8477 2079 Panel B. Augmented version (role+cancer-type tokens) Negative 0.9340 0.9142 0.9240 1609 Neutral 0.6932 0.6856 0.6894 369 Positive 0.4029 0.5545 0.4667 101 Accuracy 0.8562 2079 Macro avg 0.6767 0.7181 0.6933 2079 Weighted avg 0.8654 0.8562 0.8601 2079 Open in a new tab Class labels (0–2) correspond to the three-class target used for modeling (see Section 2 ). Precision, recall, and F1-score are computed on the held-out test set. “Regular” uses post text only; “Augmented” prepends LLM-derived reporter role and cancer type tokens to the text input. For the majority class (Negative; n = 1609), augmentation produced consistent improvement: precision increased from 0.9288 to 0.9340, recall from 0.8999 to 0.9142, and F1 from 0.9141 to 0.9240. This indicates reduced confusion of clearly negative posts with neutral/positive content after adding role and cancer-type context. For the intermediate class (Neutral; n = 369), augmentation primarily improved sensitivity: recall increased from 0.6423 to 0.6856 and F1 increased from 0.6762 to 0.6894, while precision decreased modestly from 0.7139 to 0.6932. This pattern suggests that the added context enabled the model to recover additional neutral instances, at the cost of slightly more neutral assignments among non-neutral posts. For the smallest class (Positive; n = 101), augmentation yielded the largest F1 gain (0.4152 → 0.4667). This improvement was driven chiefly by a substantial precision increase (0.3191 → 0.4029), while recall decreased slightly (0.5941 → 0.5545), indicating improved specificity for positive coping language and fewer false-positive positive predictions. In aggregate, these shifts align with the observed increases in test accuracy (0.8393 → 0.8562) and weighted F1 (0.8476 → 0.8598), and suggest that deterministic context tokens can meaningfully alter decision boundaries for minority and boundary-sensitive classes rather than merely reinforcing already-strong majority-class performance. 3.5 Paired statistical comparisons To assess whether observed differences reflected systematic changes in per-instance correctness, we conducted continuity-corrected McNemar tests for all model pairs within each condition, with Benjamini–Hochberg adjustment to control the false discovery rate at α = 0.05 (Section 2 ). Consistent with the aggregate metrics, the most pronounced and statistically reliable differences involved the GRU baseline. In the regular condition, GRU was significantly outperformed by the classical TF–IDF baselines after adjustment, while differences among Logistic Regression, Random Forest, LightGBM, and ALBERT were generally not statistically distinguishable given their close aggregate performance. In the augmented condition, paired comparisons continued to favor non-GRU models, but the number of significant adjusted differences decreased, consistent with augmentation lifting performance across model families and narrowing gaps in per-instance correctness. Taken together, the results indicate that (i) token-based incorporation of LLM-derived context improves performance consistently across model classes, and (ii) transformer fine-tuning (ALBERT) provides the strongest overall discrimination, particularly in macro AUC, while (iii) augmentation yields its most practically relevant gains by improving minority-class behavior and reducing specific confusions rather than producing uniform improvements across all classes. 3.6 Error analysis: Ordinal error structure and the impact of token augmentation To complement aggregate metrics, we analyzed error structure on the held-out test set using the regular (text-only) and augmented (role+cancer-type tokens) prediction files. Because the three-class target is ordinal, we distinguished adjacent errors (distance = 1 on the label scale) from distant errors (distance = 2, i.e., Negative↔Positive), which are operationally more consequential. Augmentation reduced the total number of incorrect predictions for most model families, with the largest absolute gains observed for the GRU and the best-performing transformer. Specifically, GRU errors decreased from 381 to 328 (net −53), reflecting 99 instances corrected by augmentation and 46 newly misclassified. ALBERT errors decreased from 334 to 299 (net −35), corresponding to 72 corrected instances and 37 newly incorrect. Classical models exhibited smaller shifts (Logistic Regression: net −16; Random Forest: net −24; LightGBM: net + 3), consistent with augmentation acting as a modest but generally helpful signal that is most beneficial for lower-capacity sequential modeling and contextual encoders. For ALBERT, augmentation yielded a meaningful reduction in the most severe ordinal errors. The number of distance-2 errors (Negative↔Positive) decreased from 107/334 (32.0%) in the regular setting to 71/299 (23.7%) in the augmented setting, indicating that the added context tokens helped the model avoid polarity-reversing mistakes. This improvement was driven primarily by fewer Negative→Positive confusions (83 → 47), alongside smaller reductions in Neutral→Positive (45 → 36) and Neutral→Negative (87 → 80). In contrast, adjacent errors remained the dominant failure mode in both conditions, reflecting intrinsic boundary ambiguity between Negative and Neutral discourse in cancer support narratives. To estimate irreducible difficulty, we identified test instances misclassified by all evaluated architectures (Logistic Regression, Random Forest, LightGBM, GRU, and ALBERT). We observed 127 such cases (6.1%) in the regular condition and 142 (6.8%) in the augmented condition, with 103 instances failing universally in both. These global-failure cases indicate a non-trivial subset of posts whose sentiment remains intrinsically ambiguous or context-dependent even after contextual modeling and token-based augmentation, reinforcing that deployment should be framed as decision support with human review—particularly for low-confidence or high-impact predictions. Overall, augmentation primarily improved performance by (i) reducing the most consequential polarity reversals for ALBERT and (ii) correcting a larger number of sequential-model errors for GRU, while leaving adjacent boundary ambiguity as the dominant residual challenge. This pattern suggests that future improvements should focus less on longer-context capacity and more on boundary-specific strategies (e.g., calibration, class-aware thresholds, and targeted labeling of borderline Negative/Neutral instances) to reduce systematic boundary drift without sacrificing sensitivity to high-need content. 4 Discussion This study evaluated emotional tone classification for posts in online cancer peer-support communities under two substantive methodological extensions: (i) supervision derived from an LLM-based annotation protocol (AI labels) and (ii) token-based incorporation of LLM-extracted structured context (reporter role and cancer type) as additional model input. Across a common experimental pipeline and a held-out test set, we observed two overarching findings. First, the LLM annotator induced a marked distributional shift relative to the dataset’s original labels, particularly by reallocating a large share of posts toward the very negative category, and this shift remained (though attenuated) after collapsing to a three-class modeling target. Second, despite this labeling shift, token-based augmentation provided consistent improvements in discriminative performance across diverse model families, with the largest gains for lower-capacity sequential modeling (GRU) and measurable improvements even for a fine-tuned transformer (ALBERT). 4.1 LLM labeling shift: interpretability and implications The most salient descriptive result is the divergence between human labels and LLM-derived labels in the four-class scheme. The LLM assigned nearly half of all posts to very negative , substantially increasing the prevalence of severe distress language relative to the original labels. This pattern is consistent with a systematic difference in intensity thresholds: the LLM appears to treat a wider set of worry, uncertainty, symptom narratives, and treatment-related difficulties as “very negative” rather than reserving that label for extreme affect. Prior work has emphasized that LLMs can behave as confident but systematically shifted raters depending on rubric framing and latent norms learned during pretraining [ 16 , 17 ]. From a measurement standpoint, this shift implies that AI labels should not be interpreted as a drop-in replacement for human labels without explicit calibration or auditing, particularly when analyses depend on absolute prevalence of distress categories. Importantly, the distributional divergence was reduced under the three-class collapse used for modeling (Negative/Neutral/Positive), suggesting that much of the disagreement arises within the negative subspace. This supports the practical motivation for collapsing adjacent negative categories when the downstream goal is screening rather than fine-grained affect intensity estimation. However, even under the collapsed target, the LLM assigned substantially more posts to Negative and fewer to Neutral/Positive, which may bias downstream summaries of community emotional tone toward higher distress. In applied settings, this could have both benefits and risks: a conservative rater may improve sensitivity to potentially high-need content, but may also increase false alarms and dilute prioritization capacity if operational triage thresholds are not adjusted. These findings reinforce broader concerns about label noise, disagreement, and learnability in subjective text annotation [ 25 , 26 ]. When supervision is produced by an LLM, label shift becomes a first-order methodological artifact. A practical implication is that evaluations should (i) report label distribution comparisons, (ii) quantify uncertainty and robustness to labeling choice, and (iii) frame model outputs as decision support rather than ground truth, especially for high-impact use cases in mental health and supportive oncology [ 5 , 7 ]. 4.2 Token-based augmentation as lightweight context integration Despite labeling shift, augmentation with LLM-extracted role and cancer-type tokens improved performance consistently across all model families. The uniform direction of gains across TF–IDF baselines, gradient boosting, GRU, and ALBERT suggests that these coarse context features provide complementary signal beyond lexical content alone. Conceptually, reporter role can change the semantics of negative language (e.g., caregiving burden vs. patient symptom distress), and cancer type may correlate with treatment trajectories and symptom narratives that shape tone expression. Notably, these structured variables were extracted from the same input text, so they do not add external information; rather, they function as an upstream information-extraction layer that makes salient context explicit to the classifier. The largest improvements occurred for the GRU model, which also had the lowest baseline performance. This is consistent with prefix tokens acting as high-salience anchors for a lower-capacity sequential encoder, helping it resolve ambiguity early in the sequence. For ALBERT, gains were smaller in magnitude but still consistent, indicating that even transformer-based encoders can benefit from explicit context markers that reduce the burden of implicitly inferring role/type from noisy and variable text. In practice, token augmentation is attractive because it is model-agnostic, requires minimal architectural change, and can be integrated into existing text pipelines with little engineering overhead. Class-wise results for ALBERT further suggest that augmentation altered decision boundaries in operationally meaningful ways. Improvements in Negative performance indicate reduced confusion for clearly distressed narratives, while changes for Neutral and Positive reveal a nuanced trade-off: augmentation increased Neutral recall (recovering more neutral posts) while slightly reducing precision, and improved Positive precision at some cost to recall. This pattern is compatible with augmentation helping the model become more selective about minority-class assignments, particularly for Positive where false positives can be common under imbalance. From a screening perspective, improved Positive precision may better preserve the interpretability of “positive coping” detection, while Neutral recall gains may reduce over-triage of informational posts as negative. 4.3 Error structure and operational consequences The ordinal-aware error analysis provides an additional perspective beyond aggregate metrics. Adjacent-category confusions remained the dominant error mode, reflecting inherent boundary ambiguity between Negative and Neutral discourse in cancer support narratives. However, augmentation reduced the most consequential polarity-reversing mistakes for ALBERT (Negative↔Positive), indicating that explicit context tokens can help avoid extreme misinterpretations that would be problematic in real-world triage or monitoring. The persistence of adjacent confusions suggests that future improvements may be better targeted toward boundary-sensitive strategies—for example, calibration and selective abstention, class-aware thresholds, or additional rubric-guided labeling for borderline Negative/Neutral instances. The subset of instances misclassified by all model families highlights another practical consideration: a non-trivial fraction of posts may be intrinsically ambiguous or require context not present in the text (e.g., prior thread history, temporal trajectory, or user baseline). This observation aligns with broader findings in mental-health NLP that model performance is often limited by construct ambiguity and annotation variability rather than solely by model capacity [ 5 , 6 ]. For deployment, this supports a workflow where low-confidence cases are routed to human review rather than forcing a single-label decision. 4.4 Model comparisons and the role of modern encoders Across both regular and augmented conditions, ALBERT achieved the strongest overall discrimination, particularly in macro OvR AUC, consistent with established advantages of pretrained transformers for sentence-level classification [ 19 , 20 ]. Nevertheless, classical TF–IDF baselines performed competitively in weighted F1, with overlapping confidence intervals among top models. This has practical implications: in settings where computational resources, latency, or interpretability constraints favor simpler models, TF–IDF baselines may provide strong performance, and augmentation can further narrow gaps. Conversely, when the goal is robust discrimination across classes (as reflected by macro AUC), transformers may provide additional benefit. 4.5 Clinical and implementation implications The findings suggest several practical pathways for supportive oncology. First, automated emotional tone monitoring could be integrated into community moderation dashboards to flag posts expressing acute distress for priority human review. The consistent performance gains from token augmentation indicate that incorporating even coarse structured context (who is writing and which cancer type is discussed) can reduce the volume of misclassified posts that moderators must manually re-evaluate. Second, the reporter-role distinction captured by the ROLE_ token may support differential routing of flagged content: caregiving burden and patient symptom distress often require different support resources, and a classifier that is aware of this distinction can direct posts to appropriate response pathways. Third, cancer-type context could facilitate integration with condition-specific clinical knowledge bases or resource directories, enabling more targeted supportive care recommendations. Fourth, the ordinal error analysis provides a template for deployment risk assessment: organizations considering automated monitoring should evaluate not only aggregate accuracy but also the rate and direction of the most consequential misclassifications (e.g., labeling distressed posts as positive). Any deployment should incorporate confidence thresholds and selective abstention for low-confidence predictions, human-in-the-loop review for high-impact cases, and transparent communication to community members about the role of automated monitoring [ 27 , 28 ]. 4.6 Limitations This work has several limitations. First, the LLM annotator was applied in a single configuration, and labeling behavior may vary with prompt design, temperature, model version, or rubric specificity. Systematic prompt ablations and multi-annotator consensus protocols could help disentangle whether the observed severity shift is intrinsic to the model or a consequence of instruction framing. Second, auxiliary variables were restricted to two coarse attributes; additional structured factors (e.g., treatment stage, symptom mentions, prognosis uncertainty) could further improve classification but may be harder to extract reliably and may raise additional ethical concerns. Third, while we used a held-out test set with bootstrapped confidence intervals and paired statistical tests, the results are specific to this dataset and to the three-class collapse; external validation on other cancer community corpora would strengthen generalizability [ 18 ]. Finally, posts are de-identified, but sentiment modeling in health-related communities raises ethical risks, including misclassification harms, privacy concerns, and downstream misuse; such models should be deployed only with clear governance, transparency, and human oversight [ 27 , 28 ]. Fourth, the LLM-derived structured variables (reporter role and cancer type) were not formally validated against manual coding. Although these tokens function as coarse categorical features and the observed performance gains suggest they carry useful signal, extraction errors could introduce noise or systematic bias into the augmented models. Future work should assess extraction reliability through comparison with human annotation. 4.7 Future directions Several directions follow from these findings. The label shift analysis revealed systematic intensity threshold differences between human and LLM raters, particularly within the negative subspace. Multi-annotator LLM protocols, using multiple models, prompt variants, or temperature settings as independent raters, could enable estimation of annotation uncertainty at the post level and identify instances where human adjudication would be most valuable. Rubric-guided or multi-pass annotation workflows may further reduce intensity threshold drift and improve label consistency. Future work should also evaluate model calibration and selective abstention on borderline posts, using proper scoring rules to assess probability quality. In a triage setting, routing low-confidence predictions to human review rather than forcing a single-label decision could reduce the operational risk of the adjacent-category confusions that remained the dominant residual error mode across all model families. Beyond the two coarse augmentation tokens evaluated here, additional structured variables (e.g., treatment stage, symptom mentions, prognosis language) could further improve classification but may require more reliable extraction pipelines and raise additional privacy considerations. Multi-task learning that jointly predicts emotional tone and extracted attributes may improve boundary resolution between adjacent classes, though at greater implementation complexity. The error analysis also revealed a non-trivial subset of posts misclassified by all architectures, suggesting that post-level text alone may be insufficient; integration of temporal features (e.g., position within a discussion thread, time since diagnosis disclosure) could address these contextual limitations. All results in this study are specific to a single dataset drawn from English-language cancer peer-support communities. External validation across platforms, languages, and cancer populations is essential to assess robustness to domain shift and to mitigate biases stemming from platform-specific language norms. Prospective evaluation in partnership with cancer support organizations would be necessary to assess real-world utility, moderator workflow integration, user acceptance, and potential unintended consequences of automated emotional tone monitoring in active peer-support communities. 5 Conclusion This study demonstrates that LLMs can serve a productive dual role in health-text classification, as both annotation instruments and structured-context extractors, but that each role introduces distinct methodological considerations. As annotators, LLMs reshape label distributions in ways that require explicit auditing: in this corpus, the LLM shifted emotional tone substantially toward greater negative prevalence, with most disagreement concentrated within the negative subspace. As context extractors, LLMs can produce lightweight structured variables that, when incorporated via simple token augmentation, improve downstream classification consistently across model families and reduce the most consequential polarity-reversing errors. For supportive oncology, these findings point toward a practical workflow in which automated emotional tone monitoring is deployed as a decision-support layer, flagging high-need content for human review, routing patient and caregiver posts to appropriate resources based on reporter role, and providing community-level summaries of emotional tone, while maintaining transparency about model limitations, incorporating confidence-based abstention for ambiguous cases, and preserving human oversight for high-impact predictions. Data Availability The original Mental Health Insights: Vulnerable Cancer Survivors & Caregivers dataset analyzed during the current study is available in the Mendeley Data repository (DOI: 10.17632/69dcnv2gzd.1 ). The annotated dataset produced in this study, which includes LLM-derived labels, structured variables, and train/validation/test split assignments, is publicly available on GitHub ( https://github.com/shxu7788/DigitalHealth ) and in the Mendeley Data repository (DOI: 10.17632/zftkdw3z7g.1 ). Funding Statement The author(s) received no specific funding for this work. References 1. Wagle NS, Nogueira L, Devasia TP, Mariotto AB, Yabroff KR, Islami F, et al. Cancer treatment and survivorship statistics, 2025. CA Cancer J Clin. 2025. doi: 10.3322/caac.70011 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 2. American Association for Cancer Research (AACR). Supporting Cancer Patients and Survivors. 2025. https://cancerprogressreport.aacr.org/progress/cpr25-contents/cpr25-supporting-cancer-patients-and-survivors/ 3. Lordon RJ, Mikles SP, Kneale L, Evans HL, Munson SA, Backonja U, et al. How patient-generated health data and patient-reported outcomes affect patient-clinician relationships: A systematic review. Health Informatics J. 2020;26(4):2689–706. doi: 10.1177/1460458220928184 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 4. Stetson PD, McCleary NJ, Osterman T, Ramchandran K, Tevaarwerk A, Wong T, et al. Adoption of Patient-Generated Health Data in Oncology: A Report From the NCCN EHR Oncology Advisory Group. J Natl Compr Canc Netw. 2022;20(13). doi: 10.6004/jnccn.2021.7088 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 5. Calvo RA, Milne DN, Hussain MS, Christensen H. Natural language processing in mental health applications using non-clinical texts. Nat Lang Eng. 2017;23(5):649–85. doi: 10.1017/s1351324916000383 [ DOI ] [ Google Scholar ] 6. Guntuku SC, Yaden DB, Kern ML, Ungar LH, Eichstaedt JC. Detecting depression and mental illness on social media: an integrative review. Curr Opin Behav Sci. 2017;18:43–9. doi: 10.1016/j.cobeha.2017.07.005 [ DOI ] [ Google Scholar ] 7. Shatte ABR, Hutchinson DM, Teague SJ. Machine learning in mental health: a scoping review of methods and applications. Psychol Med. 2019;49(9):1426–48. doi: 10.1017/S0033291719000151 [ DOI ] [ PubMed ] [ Google Scholar ] 8. Ding Z, Wang Z, Zhang Y, Cao Y, Liu Y, Shen X, et al. Trade-offs between machine learning and deep learning for mental illness detection on social media. Sci Rep. 2025;15(1):14497. doi: 10.1038/s41598-025-99167-6 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 9. Cao Y, Dai J, Wang Z, Zhang Y, Shen X, Liu Y, et al. Machine Learning Approaches for Depression Detection on Social Media: A Systematic Review of Biases and Methodological Challenges. JBDS. 2025;5(1):67–102. doi: 10.35566/jbds/caoyc [ DOI ] [ Google Scholar ] 10. Edara DC, Vanukuri LP, Sistla V, Kolli VKK. Sentiment analysis and text categorization of cancer medical records with LSTM. J Ambient Intell Human Comput. 2019;14(5):5309–25. doi: 10.1007/s12652-019-01399-8 [ DOI ] [ Google Scholar ] 11. Watanabe T, Yada S, Aramaki E, Yajima H, Kizaki H, Hori S. Extracting multiple worries from breast cancer patient blogs using multilabel classification with BERT. JMIR Cancer. 2022;8(2):e37840. doi: 10.2196/37840 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 12. Zhang Z, Liew K, Kuijer R, She WJ, Yada S, Wakamiya S, et al. Differing Content and Language Based on Poster-Patient Relationships on the Chinese Social Media Platform Weibo: Text Classification, Sentiment Analysis, and Topic Modeling of Posts on Breast Cancer. JMIR Cancer. 2024;10:e51332. doi: 10.2196/51332 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 13. Kouzy R, Cha EE, Rosen A, Bitterman DS. Review of Large Language Models for Patient and Caregiver Support in Cancer Care Delivery. JCO Clin Cancer Inform. 2025;9:e2500044. doi: 10.1200/CCI-25-00044 [ DOI ] [ PubMed ] [ Google Scholar ] 14. Yao T, Foo E, Binnewies S. CALLM: Context-Aware Emotion Analysis in Cancer Survivors Using LLMs and Retrieval-Augmented Mobile Diaries. 2025. https://doi.org/ArXiv:2503.10707 15. Gilardi F, Alizadeh M, Kubli M. ChatGPT outperforms crowd-workers for text-annotation tasks. Proc Natl Acad Sci U S A. 2023;120(30):e2305016120. doi: 10.1073/pnas.2305016120 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 16. Lu J, Ma K, Wang K, Xiao K, Lee RKW, Xu B, et al. Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement. In: Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics; 2025. p. 5609–26. [ Google Scholar ] 17. Nahum O, Calderon N, Keller O, Szpektor I, Reichart R. Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2025. p. 26770–97. [ Google Scholar ] 18. Orchi IH, Tabassum N, Hossain J, Tajrin S, Alam I. Mental Health Insights: Vulnerable Cancer Survivors & Caregivers Data. Mendeley Data; 2023. Available from: https://data.mendeley.com/datasets/69dcnv2gzd/1 19. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of NAACL-HLT 2019. 2019. p. 4171–86. 20. Lan Z, Chen M, Goodman S, et al. ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations. In: Proceedings of ICLR 2020. 2020. 21. Pedregosa F, Varoquaux G, Gramfort A. Scikit-learn: Machine Learning in Python. J Mach Learn Res. 2011;12:2825–30. [ Google Scholar ] 22. Ke G, Meng Q, Finley T, et al. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In: Advances in Neural Information Processing Systems. 2017. p. 3149–57. 23. Bird S, Klein E, Loper E. Natural Language Processing with Python. O’Reilly Media Inc.; 2009. [ Google Scholar ] 24. Cho K, van Merrienboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, et al. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014. p. 1724–34. 25. Frénay B, Verleysen M. Classification in the presence of label noise: a survey. IEEE Trans Neural Netw Learn Syst. 2014;25(5):845–69. doi: 10.1109/TNNLS.2013.2292894 [ DOI ] [ PubMed ] [ Google Scholar ] 26. Uma A, Fornaciari T, Muresan S, Plank B, Poesio M. Learning from disagreement in online discussions: A survey. J Artif Intell Res. 2021;72:1385–418. doi: 10.1613/jair.1.12752 [ DOI ] [ Google Scholar ] 27. Olteanu A, Castillo C, Diaz F, Kiciman E. Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries. Front Big Data. 2019;2:13. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 28. Hargittai E. Is Bigger Always Better? Potential Biases of Big Data Derived from Social Network Sites. Ann Am Acad Polit Soc Sci. 2015;659(1):63–76. doi: 10.1177/0002716215570866 [ DOI ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Availability Statement The original Mental Health Insights: Vulnerable Cancer Survivors & Caregivers dataset analyzed during the current study is available in the Mendeley Data repository (DOI: 10.17632/69dcnv2gzd.1 ). The annotated dataset produced in this study, which includes LLM-derived labels, structured variables, and train/validation/test split assignments, is publicly available on GitHub ( https://github.com/shxu7788/DigitalHealth ) and in the Mendeley Data repository (DOI: 10.17632/zftkdw3z7g.1 ). Articles from PLOS Digital Health are provided here courtesy of PLOS ACTIONS View on publisher site PDF (619.2 KB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• From suspicion to violent action: A multi-category framework of fear speech using qualitative content analysis and machine learning#341062
• AI-POWERED MENTAL HEALTH MOBILE APPLICATIONS: AN LLM-ASSISTED SENTIMENTAL ANALYSIS OF USERS REVIEWS#1097470
• Can Prompt Anonymity Protect Your Identity From LLM Providers?#1118976
• SelfTTS: cross-speaker style transfer through explicit embedding disentanglement and self-refinement using self-augmentation#1144774
• Semantic Flow Regularization: Teaching LLMs to Generate Diverse Yet Coherent Responses#798920