Perplexity and proximity: Large language model perplexity complements semantic distance metrics for the detection of incoherent speech.

Perplexity and proximity: Large language model perplexity complements semantic distance metrics for the detection of incoherent speech - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice J Biomed Inform . Author manuscript; available in PMC: 2026 May 31. Published in final edited form as: J Biomed Inform. 2025 Aug 21;170:104899. doi: 10.1016/j.jbi.2025.104899 Perplexity and proximity: Large language model perplexity complements semantic distance metrics for the detection of incoherent speech Weizhe Xu a, * , Serguei Pakhomov b , Patrick Heagerty c , Eric Horvitz d , Ellen R Bradley e , Josh Woolley e , Andrew Campbell f , Alex Cohen g , Dror Ben-Zeev h , Trevor Cohen a, h Author information Article notes Copyright and License information a Biomedical Informatics and Medical Education (BIME), University of Washington, Seattle, WA, USA b Pharmaceutical Care and Health Systems, University of Minnesota, Minneapolis, MN, USA c Department of Biostatistics, University of Washington, Seattle, WA, USA d Microsoft Research, Seattle, WA, USA e Department of Psychiatry, University of California, San Francisco, CA, USA f Department of Computer Science, Dartmouth College, Hanover, NH, USA g Department of Psychology, Louisiana State University, Baton Rouge, LA, USA h Behavioral Research in Technology (BRiTE) Center, Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA * Corresponding author. xuweizhe@uw.edu (W. Xu). Issue date 2025 Oct. This is an open access article under the CC BY-NC-ND license ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ). PMC Copyright notice PMCID: PMC13222020  NIHMSID: NIHMS2171786  PMID: 40849054 The publisher's version of this article is available at J Biomed Inform Abstract Objective: Semantic coherence in speech is characterized by a logical, connected flow of ideas. A lack of coherence in speech may reflect disorganized thinking, a core feature of psychosis in schizophrenia spectrum disorders (SSDs). Developing tools that could help with automated assessment of semantic coherence in language could facilitate early detection of SSDs and improved monitoring of symptoms, enabling more timely intervention. Large language models (LLMs) have demonstrated strong capabilities on numerous language-centric tasks and have shown promise for analyzing semantic coherence due to the natural fit between their innate measures of language perplexity and the surprising turns that incoherent narrative often takes. This study aims to develop a novel representation and associated measure of semantic coherence using LLM-based perplexity metrics and to compare this measure with traditional vector distance-based coherence metrics. Method: We evaluated “bag” and “chain” models based on LLM perplexities as measures of semantic coherence. Regression models were trained using both single and paired combinations of perplexity- and proximity-based features to predict human ratings of semantic coherence using standardized instruments. Performance was evaluated on held-out examples from a training set of speeches from individuals experiencing psychotic symptoms and a test set of clinical interviews with patients diagnosed with SSDs, both with labels from human assessments of disorganized thinking severity. Results: The best performance was achieved using a combination of perplexity and proximity features, yielding a Spearman correlation with human ratings of 0.61 (vs. 0.56 with proximity features alone) on leave-one-out cross-validation in the training set, and 0.54 (vs. 0.52 with proximity features alone) on the test set. Conclusion: We developed novel methods for assessing semantic coherence using LLM perplexities and found them complementary to proximity-based methods. Combined, these methods showed improved performance across two datasets, highlighting LLM’s potential in enhancing automated diagnosis and monitoring of SSDs. 1. Introduction Semantic coherence in speech is characterized by the orderly and interconnected flow of ideas. Incoherent speech can be a manifestation of thought disorder, which is a core feature of schizophrenia spectrum disorders (SSDs) [ 1 ]. An example of incoherent speech can be found in Elyn Sak’s autobiographical account of her own experience of a psychotic spectrum condition, The Center Cannot Hold : The memo materials have been infiltrated. They are jumping around. I used to be good at the broad jump, because I’m tall. I fall. People put things in and then say it’s my fault. I used to be God, but I got demoted. While the sentences are syntactically correct, it is hard to perceive the overall meaning because the topics transition drastically from sentence to sentence with little or no apparent connection. Different degrees of this type of incoherent speech are referred to as “derailment”, “loose associations”, or “flight of ideas” in psychiatry and can be measured using standard manual instruments [ 1 , 2 ]. The development of new computational systems has led to new methods for automated assessments of semantic coherence. One of the classic approaches, proposed by Foltz et al., measures semantic coherence through distributional similarity, estimating the semantic relatedness between successive text segments using Latent Semantic Analysis (LSA) [ 3 ]. This method’s effectiveness as a diagnostic tool was evaluated by Elveväg et al. [ 4 ], who found notable differences in automated coherence metrics when comparing patients with schizophrenia to healthy individuals, and among patients with varying levels of thought disorder. These coherence scores were used in a logistic regression model that differentiated between patients and controls with up to 80.4 % accuracy. A more recent study included LSA-based coherence metrics as features for a machine learning classifier that accurately forecasted psychosis onset in a small group of young individuals showing early symptoms, achieving perfect accuracy in leave-one-out cross-validation at the individual level [ 5 ]. An adapted version of this method maintained an 83 % prediction accuracy for psychosis onset in a larger, separate dataset [ 6 ]. More recently, neural word embeddings [ 7 ] – vector representations of words derived from the hidden states of neural networks trained to predict words given those in proximity to them – have been investigated as an LSA alternative for automated coherence analysis, yielding promising outcomes in predicting clinical ratings of thought disorder severity [ 8 , 9 ]. However, these semantic representations not context-specific. Despite promising results, little attention has been given to validating semantic coherence as a measure of specific linguistic anomalies (e.g., derailment, tangentiality) [ 10 ]. Most studies have evaluated model performance predicting diagnostic categories, which may be associated with complex factors other than coherence. However, the one study aside from our own prior work [ 11 , 12 ] (discussed subsequently) that specifically examined the derailment construct using static embeddings found the correlation to derailment score was only 0.36, leaving considerable room for improvement [ 13 ]. Recent research has identified differences in coherence between individuals with schizophrenia spectrum disorders and healthy controls using sentence embeddings from a pre-trained deep learning model known as BERT (Bidirectional Encoder Representations from Transformers) [ 14 , 15 ]. Though much smaller than many contemporary variants, BERT exemplifies the transformer architecture that has come to predominate in computational linguistics. Representations of individual tokens (words or smaller units) propagate through the network and are influenced by the representations of contextually co-located words/tokens, to generate representations that capture the meaning of each word in its current context (unlike LSA or neural word embeddings, where a universal representation of a word is used to generate sentence representations). In previous work, we developed and evaluated methods that assess global coherence [ 12 ], which is the relatedness between a word and the overall theme of a text (rather than between successive units), and methods that improve assessment of coherence in the context of automatic speech recognition extracting time series features from successive measurements and using these as features for machine learning [ 11 ]. Our best-performing model achieved a 0.58 correlation with derailment annotations with BERT derived embeddings, and this finding is consistent with another study that used BERT embeddings to measure coherence [ 10 ]. Despite these successes, there are reasons to believe that performance could be further improved through the use of GPT family language models. Notably, BERT’s bidirectional encoder is inconsistent with the unidirectional process of language generation during speech. In addition, both its size and volume of training data are considerably smaller than contemporary transformer models. In this work, we aim to examine semantic coherence through the lens of the current generation of large language models (LLMs – with “large” typically referring to models with a billion or more parameters at the time of this writing) such as the generative pretrained transformer family of models (GPT), that have demonstrated unprecedented capabilities in many NLP tasks [ 16 ]. Unlike BERT, which is a bidirectional model (that is, tokens on either side of a token of interest are used for prediction), the GPT family of models are autoregressive, which means they are trained to predict subsequent tokens only. While BERT-derived representations provide an excellent basis for estimating semantic relatedness, GPT models are specialized to generate coherent and fluent language as a sequence, a specialization that can potentially be exploited to develop alternative methods of coherence evaluation. Fradkin et al [ 17 ], employed a generative language model (GPT-2) to generate simulated thought disorder speech samples by perturbing the model’s temperature and memory span. Results indicate an association between these perturbations to GPTs output, and human ratings of the semantic coherence of this output. Additionally, work described in a preprint by Sharpe et al. [ 18 ] found that the speech of participants with SSDs was more difficult for LLMs to predict than that from healthy controls, with more marked differences when additional context was provided to the LLMs by increasing the size of the sliding window within which the perplexity was measured. These findings are consistent with prior research indicating difficulty in integrating distal context in SSD patients. Along similar lines, in our recent work we obtained promising results using a token-level sliding window perplexity approach[ 19 ]. However, these studies did not explore the potential of sentence-level LLM-based scores and focused primarily on token level scores. Research on proximity metrics [ 5 , 11 , 20 ] shows that sentence-level scores align well with human annotations, suggesting that perplexity methods at this level of granularity would be worth exploring also. Additionally, none of this prior work includes a comparison between LLM- and proximity-based measures of semantic coherence. In this work, we address this gap in literature by developing a novel sentence-level perplexity-based metric and applying perplexity-and proximity-based measures in concord. Our work is inspired by the work of Sap, Jafarpour and colleagues [ 21 , 22 ]. This work showed that recalled and imagined stories can be differentiated by a metric called linearity, which is derived from language model perplexity , which measures the extent to which a pre-trained model finds previously unseen text unpredictable. We hypothesized that incoherent speech could also be distinguished by an LLM’s perplexities because LLMs, trained on large amounts of presumably coherent language, should be “surprised” when presented with incoherent speech samples, resulting in higher perplexity measurements. We also hypothesized that LLM metrics ( perplexity metrics) would be complementary to the classic semantic distance metrics ( proximity metrics) because of their differences in conception. To evaluate this hypothesis, we used perplexity metrics derived from the LLaMA3 [ 23 ] LLM, which has 70B parameters, and proximity metrics from the comprehensive coherence calculator (CCC) 1 [ 11 ], which implements a range of proximity-based metrics using both global and contextual embeddings. We trained predictive models using proximity features, perplexity features and a combination of the two feature sets on a dataset collected from people experiencing auditory verbal hallucinations (AVH) (with annotated derailment severity) [ 24 ] and tested the resulting models on a dataset derived from clinical interviews with schizophrenia patients (with annotated thought disorder severity) [ 25 ]. 2. Statement of significance 2.1. Problem Methods to quantify semantic coherence in natural language to detect thought disorganization in psychosis have room for improvement, and the utility of contemporary large language models for this task is poorly understood. What is Already Known Prior work in this area has predominantly focused on methods based on word embedding distances (proximity methods). Recently, a few studies have considered language model perplexity as a potential metric for semantic coherence (perplexity methods). However, prior studies used relatively small models (under 1 billion parameters) and did not explore combining these approaches. What This Paper Adds. This study leverages a contemporary large language model and introduces a novel sentence-level perplexity feature, which is subsequently used in an integrated prediction model with both proximity and perplexity features. This combined approach improves performance over using either method alone. Who Would Benefit. Researchers studying automated semantic coherence metrics, linguistic biomarkers, and schizophrenia-spectrum disorders. 3. Methods 3.1. Datasets AVH dataset: The AVH dataset was obtained from a prior study [ 24 ] that collected speech samples using a smartphone application from n = 384 participants who experienced hallucinations. Participants experiencing AVH were recruited via both in-person and online means. A clinical diagnosis was not a requirement for participation, though some participants (n = 151) self-reported diagnoses of SSDs. Informed consent from participants was obtained through a rigorous procedure involving triple confirmations from a screening questionnaire. All participants were asked to install a mobile application, which had the capability of recording and uploading audio diaries, and were prompted to describe their experiences of AVH, as well as anything else they would like to share or think it would be helpful for the research team to know. Prompts for audio diaries followed the collection of other data, and the participants also had the option to record an entry directly on demand. Although no monetary incentives were offered for the audio diary component, most participants submitted their recorded audio diaries. We used data collected up to October 18th, 2019, consisting of 1868 recordings from 202 participants. From these recordings, we selected 310 transcripts that satisfy the following criteria: 1. They are longer than 30 s. 2. Each participant has at most 3 samples. 3. They have coherence labels from expert annotators. Two annotators labeled the transcripts for their degree of incoherence based on the Thought and Language Disorder (TALD) scale [ 2 ], using the construct of derailment which was selected on account of its fit with the open-ended nature of the audio diary task, where a coherent flow of topics is more relevant than adherence to an initial topic or question. The TALD score ranges from 0 to 4 and represents greater incoherence as the score increases. The inter-rater agreement between annotators was 0.71, as measured by quadratically-weighted Kappa. This set contains samples that have a mean TALD score of 1.21 with a standard deviation of 0.87. As in previous research [ 11 , 12 ], severe cases of thought disorganization were defined by scores of 3 or higher. This threshold was used for the purpose of computing the area under the receiver operating characteristic curve (ROC-AUC) in our experiments. We selected this dataset for this study because 1) people with AVH experiences may suffer from varying degrees of thought disorder, which ensures a diverse degree of semantic coherence in the sample pool, and 2) we have evaluated traditional proximity-based coherence measures using this set previously [ 11 , 12 ]. Clinical interview dataset: This dataset was obtained from clinical interviews of 39 male out-patients diagnosed with schizophrenia spectrum disorders (SSDs) participating in a study of oxytocin conducted at the University of California, San Francisco. (UCSF) 2 [ 25 ]. The interviews are semi-structured and the resulting clinical assessments by trained raters are provided in the form of a composite score combining the conceptual disorganization item (ranging from 1 to 7 with increasing severity) [ 26 ] from the Positive and Negative Syndrome Scale (PANSS) and the incoherent speech item (ranging from 0 to 5 with increasing severity) from the Comprehensive Assessment of Symptom and History (CASH) [ 27 ]. The composite score was derived as the sum of the two individual items. We collected 39 manually transcribed interviews with corresponding composite scores between 2 and 8 (in the range of 0–12), with a mean of 3.36 and standard deviation of 1.80. We included this dataset as an additional evaluation of the proposed method’s ability to detect incoherent speech because 1) it is structured differently (dialogue vs. monologue) and gathered from a different population, mediating evaluation of the model’s capacity for generalization; 2) it permits evaluation at degrees of thought disorganization typical of outpatients with SSDs, arguably the population in which automated coherence assessment would be of greatest clinical value as a means to anticipate relapse events; and 3) the scores reflect direct assessments of patients by clinicians, rather than judgments arising from the examination of transcribed speech, providing an indication of whether models trained on the latter assessments can generalize to the former. 3.2. Proximity metrics The proximity metrics were based on the best performing 3 coherence metrics from our previous work [ 12 ], namely the sentence-level sequential, static centroid, and cumulative centroid coherence metrics. These methods perform a semantic distance evaluation between consecutive sentence embeddings (sequential) or between each sentence embedding and a central vector of the transcript, derived as the vector average (or centroid) of the vector representations of the sentences it contains. With the static method, all sentences were used to estimate the centroid. With the cumulative method, only those that preceded the sentence under consideration were used. The sequential coherence metric estimates the local coherence whereas the centroid coherence metrics estimate global coherence. We implemented the 3 metrics with both static embeddings and contextual embeddings, bringing the total number of proximity metrics to 6. The static embeddings were trained using FastText [ 28 ], and publicly released as a set of 2-million-word vectors 3 trained on Common Crawl. 4 The contextual embeddings were obtained using Sentence-BERT [ 29 ], a model specifically trained on semantic search tasks to produce meaningful sentence embeddings for cosine similarity calculations. Specifically, we used the “all-MiniLM-L6-v2” Sentence-BERT model. 5 For simplicity, we show the best performing embedding version of the metric in the result section. These proximity metrics are well-established tools for evaluating semantic coherence as a proxy for organized thinking and provide strong baselines for this study. Because previous work has shown that neural word embeddings generally have better performance or greater utility in tasks that involve cosine similarity calculations than LSA embeddings [ 30 , 31 ], we did not include LSA derived word embeddings in this analysis. 3.3. Perplexity metrics We propose a new sentence-level perplexity-based estimate of thought disorganization (as reflected by diminished semantic coherence) based on the perplexity of LLMs, which measures the level of “surprise” of a LLM after seeing a sentence given its context. The perplexities were derived from the LLM’s probability output when predicting the next token. However, perplexity is intuitively inversely correlated with probability because the higher the probability the lower the level of surprise from the language model. Based on the work of Sap, Jafarpour et al [ 21 ], we used the negative log likelihood (NLL) as a surrogate for perplexity, which is monotonically associated with NLL. The NLL of a sentence S given context C can be calculated using the following equation: N L L ( S ∣ C ) = − 1 ∣ S ∣ ∑ i = 0 ∣ S ∣ log ( P ( T i ∣ C , T 0 ⋯ i − 1 ) ) where P is the probability output of the language model and T 0… i are the tokens in the sentence S . Following the work of Sap, Jafarpour et al [ 21 ], we adapted the “chain model”, where perplexities are used with a cumulative context, and the “bag model”, where perplexities are used with a central static context, as probabilistic analogues of semantic distance-based coherence measures. The chain and bag models are components of a measure of linearity developed in this work, which was used to quantify narrative flow to distinguish between real and imagined stories. In this work, we focus on the chain and bag model components’ utilities as LLM-based measures of sequential and global coherence, respectively. For the chain model, the perplexities of each of the subsequent sentences were computed by including all previous sentences and the initial context as the context C . For the bag model, the perplexities of each of the sentences were computed individually with a constant initial context (a summary of the text). Fig. 1 demonstrates how the chain and bag models handle contexts in perplexity calculation. The bag model is an estimate of global coherence as each sentence is assessed in the context of text representing a central theme, similar to its counterpart in proximity metrics (static centroid). The chain model, resembling the design of the cumulative centroid in proximity metrics, measures an evolving global coherence by including contextual information up until the point of measurement. We also included a traditional transcript-level perplexity score as a baseline for the bag and chain model methods. The transcript-level perplexity scores were derived from the average of the losses at each token. Fig. 1. Open in a new tab The difference between chain and bag models in choosing contexts for perplexity calculations at each sentence S1-Sn. Arrows indicate that the preceding segments inform the prediction of those that follow. For the AVH data, we used a summary from BART, a leading transformer-based summarization model [ 32 ], as the initial context. For the clinical interview data, we used the first sentence of the transcript as the initial context due to the difficulty of obtaining good summaries of data in long-form conversational formats. Because of the conversational nature of this dataset, we also used a different pre-processing strategy to represent sentences as semantic units: the conversations contain short and less meaningful phrases such as “Yes.” or “Of course.”, which are nonetheless categorized as independent sentences by the sentence tokenizer. After initial experiments revealed this limitation, we elected to count any sentence with 4 words or less as a part of the next sentence, to preserve the narrative without introducing semantic gaps by treating phrases with little semantic content as independent sentences (see Appendix Figure A for results without this preprocessing step). Following this procedure, we used concatenated participant speech for both the proximity and perplexity methods to avoid considering the interviewer’s speech. The LLM we used to obtain the perplexities was a locally hosted instruction-tuned 6 LLaMA-3.0 model (70 billion parameters), with 4-bit quantized weights obtained with Exllama2 7 , which we used for accelerated inference. After obtaining a series of perplexity values for the sentences in a transcript, we used the best performing aggregation function (mean, median, max, min, 10th and 90th percentiles) to represent the perplexity coherence feature for the input transcript for downstream analysis in the experiments. A detailed discussion of the process of choosing an aggregation function can be found in the Experiments sub-section of this manuscript. 3.4. Experiments 3.4.1. Overview Fig. 2 provides an overview of the experiments in this study. Fig. 2. Open in a new tab An overview of experimental design. LOOCV = leave one out cross-validation. 3.4.2. Aggregation, temperature, and model choice Both the perplexity and proximity metrics produce scores on a sentence-to-sentence scale. So, we examined several aggregation strategies to collect these sentence-level scores as a single feature for the transcript. These include minimum, maximum, median, mean, and the 10th and 90th percentiles. The temperature setting of an LLM is typically applied during text generation, where a higher temperature will encourage the model to sample more broadly from probable next tokens. However, it can also be applied at the point at which perplexity is measured, such that tokens further down the ranked list of predictions have higher probabilities as temperature rises. We evaluated temperature settings of LLaMA3 ranging from 0.1 to 2.1 with an increment of 0.1 and then from 2.1 to 9.1 with an increment of 1. To determine the best temperature, we examined the Spearman rank correlation of model loss (i.e. NLL for the transcript) with the annotations in the AVH training set and selected the best temperature based on the correlation score. Using this temperature, we tested different aggregation strategies for the series of NLL values generated from the bag and chain models. We also evaluated different regression models, specifically linear regression, support vector machine regression, and gradient boosting regression to probe for both linear and non-linear relationships between the features (sets of proximity-based and perplexity-based coherence measures) and human ratings of incoherence (TALD derailment or the composite measure). We used leave-one-out (LOO) cross validation to obtain predicted scores from the regression models and computed Spearman rank correlation with human annotations. We chose the best performing model on the training set for the following tests of the utility of different feature sets. Because the training set has a defined criterion for severe cases (TALD score >=3), we also included an evaluation of Area Under the Curve of the Receiver Operating Characteristic curve (ROC-AUC) for this dataset as an additional evaluation of performance. However, the selection of the best configuration is based on Spearman rank correlation because it is the metric used for the test set due to the lack of a defined threshold in the clinical evaluations accompanying this set. 3.4.3. Performance on training set To evaluate the perplexity and proximity features, we trained models using the following types of feature sets: single proximity features, single perplexity features, and dual-feature combinations of any single feature so that we can observe the effects of combined proximity and perplexity features. The model performance was evaluated by calculating the Spearman rank correlation between the LOO predictions for each transcript and the corresponding human annotations. Through this design, we can determine 1) how the proximity and perplexity feature set alone performs on this dataset and 2) if the combination feature sets including both perplexity and proximity-based coherence estimates outperform the single feature models, or the dual feature models trained using two proximity or perplexity features only. 3.4.4. Performance on test set We tested the performance of our proposed method on the clinical interview dataset by training LR models using the entirety of the AVH dataset. We followed the same feature configuration to test how each feature set performs on the test set. Because the scores used in the clinical interview dataset do not have a well-defined threshold for severe cases, we used the Spearman rank correlation as the evaluation score. 3.4.5. Adversarial paraphrase testing Adversarial paraphrase testing provides a way to test the robustness of a model’s interpretation of semantics. However, sentence-level paraphrasing may still introduce differences by perturbing the stylistic continuity of a passage. Therefore, we used adversarial paraphrasing to evaluate how each of the best-performing perplexity and proximity components behave under such perturbations. To do so, we paraphrased the AVH dataset using the FLAN-T5 paraphraser, 8 a model trained to generate paraphrase samples on a sentence-by-sentence basis. We then performed the same LOO evaluation on the paraphrased samples using single feature set regression models to observe how each feature reacts to the paraphrase. 4. Results 4.1. Aggregation method and temperature We found that a temperature of 1.1 produced the best correlations in the training set across aggregation techniques. Fig. 3 shows model NLLs’ correlation with human judgment under different temperature settings when used with median aggregation. Fig. 3. Open in a new tab Spearman rank correlation between model NLL and human annotation at different temperatures on the training set. Temperature 1.1 produces the best performance. With temperature 1.1, we also evaluated the Spearman rank correlation with the annotated score for each aggregation strategy as shown in Table 1 . We found that the minimum aggregation was the most effective aggregation strategy for the sequential proximity metrics, which is consistent with previous findings [ 5 , 12 ]. However, when prioritizing overall alignment (Spearman rank correlations) over the ability to identify severe cases (ROC-AUC), we found that the mean aggregation was better or maintained equal performance to minimum aggregation when using the centroid coherence metrics. Similarly, we found that the median aggregation performed best for the bag model and that the mean aggregation worked best for the chain model. The median aggregation function was proposed to address the disproportionately high perplexities observed in early sentences when little context was available and was chosen by selecting the aggregation function that produces the best Spearman rank correlation and ROC-AUC with the human annotated scores in the AVH training set. Of note, when considering the best aggregation function, the best-performing perplexity metric was the chain model (r = 0.47, perplexity), which is also the best performing method in terms of AUC (0.83) and the best-performing proximity metric was the cumulative centroid (r = −0.57, proximity). To share results from the full spectrum of contextual embeddings we tested, we also include the performance of other contextual embeddings in Appendix Table C . Table 1. Spearman rank correlation (ROC-AUC in parenthesis) of different aggregation strategies for each individual metric with the annotated scores in the AVH dataset (ROC-AUC calculated with the threshold of TALD >= 3). The best Spearman Rho value is highlighted in bold in each category. Note that the transcript level perplexity does not need to be aggregated so it is not shown in this table. Sequential (static embedding) Proximity metrics (higher values indicate more coherence) Cumulative Centroid (contextual embedding) Perplexity metrics (higher values indicate less coherence) Static Centroid (contextual embedding) Chain Model Bag Model Minimum −0.27 (0.81) − 0.51 (0.73) − 0.53 (0.77) 0.01 (0.51) − 0.01 (0.55) Mean 0.00 (0.66) −0.53 (0.79) −0.57 (0.80) 0.47 (0.83) 0.36 (0.73) Median − 0.03 (0.65) − 0.51 (0.78) − 0.53 (0.77) 0.42 (0.81) 0.39 (0.76) Maximum 0.23 (0.56) − 0.37 (0.76) 0.01 (0.56) 0.44 (0.79) 0.33 (0.63) 90th Percentile − 0.12 (0.52) − 0.44 (0.79) − 0.55 (0.80) 0.39 (0.76) 0.28 (0.67) 10th Percentile − 0.15 (0.76) − 0.50 (0.73) − 0.53 (0.77) 0.27 (0.66) 0.30 (0.66) Open in a new tab 4.2. Model selection We also evaluated the performance of linear and non-linear regression models on the training set. We included the model predictions’ correlation with annotated scores using each metric as a single feature training data as shown in Table 2 . The predictions were obtained using LOO cross validation, with a prediction for the held-out transcript from each LOO iteration. Our results indicate that linear regression is the best modeling approach for this task. Non-linear models did not offer improvement on the LOO cross validation performance, perhaps because they overfit to these single-feature inputs. Table 2. Spearman rank correlation between regression scores and human annotations (ROC-AUC in parenthesis) using single feature with the best aggregation across LR (linear regression), SVM (support vector machine), and GBR (gradient boosting regression). Proximity metrics Sequential (static embedding) Static Centroid (contextual embedding) Cumulative Centroid (contextual embedding) Perplexity metric Chain Model Bag Model Transcript-level Perplexity LR 0.23 (0.81) 0.51 (0.76) 0.56 (0.79) 0.45 (0.79) 0.36 (0.71) 0.31 (0.81) SVM 0.21 (0.76) 0.51 (0.76) 0.54 (0.79) 0.43 (0.78) 0.31 (0.72) 0.28 (0.77) GBR 0.09 (0.65) 0.40 (0.80) 0.45 (0.81) 0.35 (0.70) 0.25 (0.64) 0.21 (0.72) Open in a new tab 4.3. Training set LOO performance We used LOO cross validation to generate regression predictions using either 1 or 2 feature models. The performance was evaluated using Spearman rank correlation between the predicted scores and human annotations. The results are shown in Fig. 4 . We found that the dual feature combinations generally show improvements over the performance of individual feature models, especially when combining a perplexity metric and a proximity metric . The best performing feature set is a combination of transcript-level perplexity and cumulative centroid features, with a correlation of 0.61 to human annotated scores. This feature set also performed well for identification of severe cases, with an ROC-AUC of 0.88. However, on this task it was outperformed by the combination between the chain model (perplexity) and sequential measure (proximity), with an ROC-AUC of 0.89, both of which performed well (though not as well, with ROC-AUC of 0.79 and 0.81 respectively) for identification of severe cases as individual features. Fig. 4. Open in a new tab Training set cross-validation Spearman rank correlation (TOP) and ROC-AUC scores (BOTTOM) between model predictions and human annotations on the AVH dataset, including single feature models (in the diagonal of the heatmap) and 2 feature models of all combinations. Brighter shading indicates better scores. 4.4. Test set performance We applied the models trained on the AVH dataset to the clinical interview dataset for further evaluation. The results are shown in Fig. 5 . A similar pattern can be observed to that seen in the LOO cross validation result in the AVH dataset: combining a perplexity metric and proximity metric as a dual feature set produced the best results. In the test set, the sequential proximity model achieved a 0.52 correlation with the human annotations. However, this performance is further enhanced by the combination with the chain model feature resulting in the best performance of 0.54 correlation with human annotations. The linear regression model coefficients for the best performing model (sequential + chain) are −0.29 for the proximity feature and 0.41 for the perplexity feature, suggesting approximately equal contribution. The p-values for the correlations are provided in Table B of the appendix . Of note, all of the feature combinations that included proximity: sequential have correlations that are highly significant (p < 0.01) and all of those combinations that include proximity: static centroid are significant (p < 0.05) except the combination with cumulative centroid. Fig. 5. Open in a new tab Test set Spearman rank correlation between model predictions and human annotations, including single feature models (in diagonal of the heatmap) and dual feature models of all combinations. 4.5. Adversarial paraphrase test We performed the LOO evaluation on the paraphrased AHV dataset. The results are shown in Table 3 . The perplexity metrics are most affected by this perturbation, whereas the proximity metrics maintain performance on the paraphrased data. This was to some extent expected because the paraphrase was performed on individual sentences. While the semantics of each sentence might be similar to the original transcripts, the connectivity between each sentence could be lost in the process as the paraphrase model does not have access to previous text as context. We posit that perplexity features are more sensitive to such continuity within the transcripts, and disturbances in the flow between sentences likely obscure a key signal that the feature is responding, to, and lead to high perplexity in general. We provided examples of the original and paraphrased transcripts in the Appendix to demonstrate this disturbance. Table 3. LOO performance (Spearman rank correlation with human annotation) comparison between the original transcripts and paraphrased transcripts using single feature linear regression models. Original transcript performance Paraphrased transcript performance Perplexity 0.31 0.12 Perplexity: Bag Model 0.38 0.29 Perplexity: Chain Model 0.46 0.11 Proximity: Sequential 0.25 0.37 Proximity: Static Centroid 0.50 0.47 Proximity: Cumulative Centroid 0.56 0.56 Open in a new tab To further validate this hypothesis, we examined the feature values of the chain model. We observed that the paraphrased transcripts on average have higher perplexity scores. However, the scores from the proximity feature remain largely the same distribution across original and paraphrased transcripts (see Appendix , Figure B ). As a result, with the chain model the distinction between coherent and incoherent transcripts is less obvious in paraphrased samples, as can be observed in Figure C in the Appendix . 5. Discussion 5.1. Key findings In this study, we explored the utility of probabilistic approaches to automatically estimate deficits in semantic coherence using LLM perplexity, including a novel sentence-level approach. The scores produced by both transcript-level LLM perplexity and the new approach correlate with human judgments of disorganized thinking across two different datasets as an individual feature. While the best-performing proximity metrics from prior work exhibited stronger performance as individual features, combining perplexity and proximity metrics in a multi-variate model yielded the best overall performance in terms of correlation with human judgement on both datasets used in the study. This suggests that perplexity metrics and proximity metrics are complementary and can be combined into a composite feature set to improve the measurement of semantic coherence. 5.2. The unique contribution of perplexity From the results, we observe that the combination of LLaMA3-derived perplexity feature and FastText derived proximity features attain the highest correlation with human judgement in both training set LOO cross validation and test set evaluation scenarios. We initially attributed this high performance to LLaMA3′s better representation of text semantics. However, further experiments using LLaMA3-derived word embeddings to calculate proximity features showed that this is not the case. Our conclusion stems from two observations: 1). Using the training set, we found that the proximity metrics using LLaMA3-derived embeddings have higher correlation with other proximity metrics than the perplexity metrics, which are also derived from LLaMA3. For example, the values from the cumulative centroid using LLaMA3 embeddings have a correlation of 0.79 with the cumulative centroid using Sentence-BERT embeddings but it only has −0.31 and −0.49 correlations with the chain and bag models respectively. (For a complete list of correlations, refer to Table A in the Appendix .) This indicates that proximity metrics using LLaMA3-derived embeddings behave more similarly to proximity features derived from other embeddings than to LLaMA3-derived perplexity features. This observation suggests that the improved performance is due to LLM perplexity’s unique contributions rather than better representation of text in embeddings from an LLM. 5.3. The complementary nature of the two feature sets Like previously established distributional representations of language [ 33 , 34 ], neural word embeddings are vector representation of semantics derived such that words with similar meanings will have vectors oriented in a similar direction [ 7 ]. It follows from this that proximity features should be measuring the semantic similarity between two text units. In contrast, the perplexity of a LLM measures the level of “surprise” of the LLM given a prior context. As such, the perplexity feature reacts not only to semantic similarity, but also to continuity , which indicates how likely a sentence is to occur after a prior sentence regardless of their semantic similarity. This contributes additional, and apparently useful, information to the downstream regression model. This suggests that the perplexity features respond to both similarity and continuity, rather than selectively, to similarity alone. Both similarity and continuity are aspects of coherent speech. Therefore, adding the perplexity feature contributes additional information that is not available with the proximity feature alone. We have included transcript examples in the Appendix to demonstrate the differences in contributions of each feature type. Reflecting upon specific mechanisms thought to underlie the language differences in SSD, we theorize that the concepts of semantic similarity and continuity may correspond to differences in the subject’s semantic and working memory respectively. The theory of semantic memory assumes that in our mind, words and concepts are linked in a network based on their association [ 35 , 36 ]. In accordance with this proposal, language differences in SSD’s related to semantic memory may manifest as the “loosening of association” we observe in affected speech samples [ 35 ]. As the distance between word vectors (proximity metrics) represents semantic similarity, it fits the needs of an automated tool to assess differences related to semantic memory (the proposal that text-derived proximity metrics provide a computational model of semantic memory dates back at least as far back as Lund, 1995 [ 37 ]). On the other hand, working memory and executive functions can also be affected in SSDs [ 35 , 38 ]. The impairment of working memory can be reflected as the inability to recognize the constraints imposed by the context in previous text [ 39 ]. Sharpe et al [ 18 ] measured the NLL values of a GPT3 model with varying length of contexts and discovered that the difference between control and schizophrenia group widens as longer contexts are used in language models, suggesting that the impairment of working memory in the schizophrenia group might have caused deviation from distant context. As such, because the perplexity metrics are based on the next token probabilities of a LLM, we speculate that it may serve as an automated tool to assess aspects of incoherence related to working memory differences in SSDs. 6. Implications and limitations This work has further improved the performance of an automated method of estimating semantic coherence by incorporating features derived from an LLM. The models trained from the combined proximity and perplexity features performed well, even when the test data (conversational) was not identical to the training data’s format (diary), suggesting the 2 features together provide valid estimates of semantic coherence, regardless of the data format. This work also avoids some of current LLM research’s most prominent shortcomings, such as excessive prompt tuning and “hallucinations” (or fabrications) by not using the LLM’s generated content directly. Rather, our method uses model perplexity, a directly observable model output that makes the coherence estimation process more transparent. A limitation of this work is that only one construct, derailment, was used to annotate the training dataset. While this is the construct that is most closely related to semantic coherence, other manifestations of thought disorder such as tangentiality, logorrhoea and poverty of speech [ 2 ] were not explored in this study. Another limitation is that the clinical interview dataset is derived entirely from schizophrenia patients, which might offer limited test case diversity. However, we expect varied individual scores across participants for the severity of disorganized speech mitigates this lack of diversity to some extent. Additionally, we used the LLaMA 3 model instead of leading-edge commercial alternatives (such as the GPT-4), because LLaMA 3 has open-source weights. Consequently, we could host it locally on an in-house server where participant data are securely stored, which was necessary because we did not have approval to share these sensitive participant data with a third-party commercial AI service for analysis. The transcripts shared in this manuscript are partial transcripts and were anonymized to adhere to the data sharing policy. In addition, having access to model weights was convenient for the development of our customized perplexity metrics. Finally, the specific linguistic differences that perplexity and proximity features are measuring are not determined in this study. We interpreted our findings in relation to theories concerning the role of semantic and working memory in linguistic anomalies in SSD but we have not investigated these relations empirically. Further research may be necessary to determine the exact cause behind the complimentary nature of the two features. Specifically, a perturbation test in long-range context to simulate a working memory deficit could further support our theory. Unlike Kuperberg et al [ 18 ], who performed tests on varying sizes of context windows for the LLM to show that incoherence in SSD is more apparent when distal context is considered, the test could aim to change the content at the beginning of a document to assess whether proximity and perplexity features respond differently. 7. Conclusion In this paper, we leveraged features from an LLM to enhance the detection of incoherent speech. We found that features derived from LLM’s perplexity can complement features derived from the proximity of word embeddings. The improved correlation with human judgment exhibited with combined models in both training and test sets indicates an important role for LLM-derived metrics in the assessment of speech for indicators of thought disorganization, with the potential to improve automated diagnosis and monitoring of patients with schizophrenia-spectrum disorders. Acknowledgement This study was supported by National Institute of Health (NIH) (grant number U01MH135901) Appendix Examples of paraphrased transcripts. Original transcript: “I’m a really nice person and I don’t judge people at face value. There’s some truth that, physically, more symmetrical faces equate to better behavior in society and higher intelligence. Especially attractive people are more likely to behave well in society, and I am a very pretty girl, very pretty.” The corresponding paraphrased transcript is shown below: “I don’t judge people by their appearance. Symmetry is a good thing in life and it can help you to be more intelligent. A pretty girl is more likely to behave well in society and i have a pretty profile.” The paraphrased transcript reads as disjointed sentences that do not follow the natural flow of speech, and the second sentence distorts the meaning of the original transcript which does not suggest that symmetry can improve intelligence. However, this distinction is unlikely to be picked up by proximity models that solely rely on word emebedding similarities. Original transcript: “Hello.”. Paraphrased transcript: “Hello and welcome to the etsy store.” This differentiation is caused by the hallucination of the paraphrase model. This can result in unpredictable effects based on the hallucinated content but it is likely to disrupt the continuity between the sentences and thus cause inaccurate perplexity assessment. Examples of differences in feature estimates In this section, we show some specific snippets of transcripts to gain insight into what the perplexity and proximity features are responding to. We analyze these transcripts through the lens of the chain model (perplexity) and the sequential model (proximity) because they are easier to interpret with sentence-to-sentence transitions. For example, consider the following snippet of a transcript: “I don’t know if it’s real or somebody’s messing with me or something. I know I heard it. It’s not in my head.” In this example, because “I know I heard it.” and “It’s not in my head.” have low semantic relatedness, the proximity sequential method provides a coherence score of 0.58, which is relatively low for cosine similarity between sentence embeddings (mean value 0.70). However, the perplexity score for the last sentence is 0.91, which is also low for perplexities (mean value 2.07), suggesting a coherent transition from the previous context (which, in the case of the chain model, includes earlier sentences also). This illustrates how the perplexity feature can accurately capture normal transitions between sentences that have been misrepresented as incoherent by the proximity feature due to semantic dissimilarity. On the other hand, the perplexity feature may be insensitive to sudden transitions of topic that indicate the derailment of the train of thought. In an incoherent transcript: “Well, the pedophile finally got me up out of the out of the devil’s out of the devil’s trap. I’m way uptown now. I got away from out of Satan’s grasp. I’m up here by public XXX public market.” (location name redacted to protect privacy). In this example, the speaker shifts from describing a metaphorical escape from the devil to describing their location. The perplexity score of the last sentence is 0.40, which is very low and indicates a coherent transition. In contrast, the proximity feature is very sensitive to the semantic shift from the diabolical to a marketplace, so the score for the last sentence is 0.47, which indicates low coherence. Therefore, proximity features can sometimes detect incoherent transitions that may not be apparent to perplexity features. These examples illustrate the complementary nature of these approaches. Fig. A. Open in a new tab Performance heatmap without the preprocessing step of merging shorter utterrances in the clinical interview dataset. The sequential and static centroid methods are most affected by this because the minimum aggregation that works best for these in the training set can be significantly affected by less meaningful transitions from shorter utterrances. Fig. B. Open in a new tab Feature score comparison between original and paraphrased transcripts. The left (B1) shows the perplexity: chain model feature whereas the right (B2) shows the proximity: cumulative centroid feature. The paraphrase causes perplexity to rise in general but does not affect the proximity metric significantly. Fig. C. Open in a new tab Perplexity feature score comparison between original and paraphrased transcripts and between coherent (TALD < 3) and incoherent (TALD >=3) transcripts. The left panel (original transcripts) shows a more prominent distinction between the coherent and incoherent samples whereas the right panel (paraphrased transcripts) shows reduced distinction. Table A. Spearman rank correlation between proximity metrics using Llama embeddings and the corresponding metrics used in the main study. (i.e. Sequential (FastText embeddings), Static centroid (Sentence-BERT embeddings), Cumulative centroid (Sentence-BERT embeddings), Chain model, and Bag model). Note that the correlations between proximity metrics with Llama vs. Non-Llama embeddings are higher than the correlations between proximity and perplexity metrics that are both derived from Llama. This analysis is performed using the training FastText/BERT Embeddings Chain model (Llama) Bag model (Llama) Sequential (Llama Embeddings) 0.39 − 0.18 − 0.34 Static Centroid (Llama Embeddings) 0.72 − 0.30 − 0.46 Cumulative Centroid (Llama Embeddings) 0.79 − 0.31 − 0.49 Open in a new tab Table B. P-values for the Spearman rank correlation in the test set. Transcript-level Perplexity Bag Model Chain Model Sequential Static Centroid Cumulative Centroid Transcript-level Perplexity 0.08 Bag Model 0.10 0.25 Chain Model 0.06 0.22 0.06 Sequential <0.01 <0.01 <0.01 <0.01 Static Centroid 0.04 0.03 0.01 <0.01 0.01 Cumulative Centroid 0.06 0.13 0.05 <0.01 0.79 0.19 Open in a new tab Table C. Performance of different contextual embeddings in terms of Spearman rank correlation with human judgement. The best performing ones, which were used in the main experiment, are in bold . Sequential Static Centroid Cumulative Centroid Static embedding 0.25 0.44 0.43 BERT CLS token 0.21 0.38 0.45 BERT sum of last 2 layers 0.17 0.48 0.48 SentenceBERT 0.23 0.52 0.56 SimCSE 0.20 0.50 0.54 DiffCSE 0.24 0.51 0.53 Open in a new tab Footnotes CRediT authorship contribution statement Weizhe Xu: Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Serguei Pakhomov: Writing – review & editing, Validation, Supervision, Methodology. Patrick Heagerty: Writing – review & editing, Validation, Supervision, Methodology. Eric Horvitz: Writing – review & editing, Validation, Supervision, Methodology. Ellen R. Bradley: Writing – review & editing, Resources, Data curation. Josh Woolley: Writing – review & editing, Resources, Data curation. Andrew Campbell: Writing – review & editing, Validation, Supervision, Methodology, Data curation. Alex Cohen: Writing – review & editing, Validation, Supervision, Methodology. Dror Ben-Zeev: Writing – review & editing, Validation, Supervision, Resources, Funding acquisition, Data curation. Trevor Cohen: Writing – review & editing, Validation, Supervision, Resources, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. 1 https://github.com/LinguisticAnomalies/Coherence . 2 The referenced work used 37 participants because 2 of the participants did not complete all the tasks required. Because we only need the interview transcripts, we included all 39 in this work. 3 https://fasttext.cc/docs/en/english-vectors.html . 4 https://commoncrawl.org/ . 5 https://sbert.net/docs/sentence_transformer/pretrained_models.html . 6 While instruction-tuning is not necessary to estimate perplexity, we found it did not degrade performance in preliminary experiments and consequently elected to use the instruction-tuned variant of the model hosted internally as a shared service for use by members of the lab. 7 https://huggingface.co/turboderp/Llama-3-70B-exl2 . 8 https://huggingface.co/alykassem/FLAN-T5-Paraphraser . References [1]. Andreasen NC, Grove WM, Thought, language, and communication in schizophrenia: diagnosis and prognosis, Schizophr. Bull (1986), 10.1093/schbul/12.3.348. [ DOI ] [ PubMed ] [ Google Scholar ] [2]. Kircher T, et al. , A rating scale for the assessment of objective and subjective formal thought and language disorder (TALD), Schizophr. Res (2014), 10.1016/j.schres.2014.10.024. [ DOI ] [ PubMed ] [ Google Scholar ] [3]. Foltz PW, Kintsch W, Landauer TK, The measurement of textual coherence with latent semantic analysis, Discourse Process. (1998), 10.1080/01638539809545029. [ DOI ] [ Google Scholar ] [4]. Elvevag B, Foltz PW, Weinberger DR, Goldberg TE, Quantifying incoherence in speech: an automated methodology and novel application to schizophrenia, Schizophr. Res (2007), 10.1016/j.schres.2007.03.001. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [5]. Bedi G, et al. , Automated analysis of free speech predicts psychosis onset in high-risk youths, NPJ Schizophr. (2015), 10.1038/npjschz.2015.30. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [6]. Corcoran CM, et al. , Prediction of psychosis across protocols and risk cohorts using automated language analysis, World Psychiatry (2018), 10.1002/wps.20491. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [7]. Mikolov T, Chen K, Corrado G, and Dean J, “Efficient estimation of word representations in vector space,” in 1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings, 2013. [ Google Scholar ] [8]. Just S et al. , “Coherence models in schizophrenia,” 2019. doi: 10.18653/v1/w19-3015. [ DOI ] [ Google Scholar ] [9]. Just SA, et al. , Modeling Incoherent Discourse in Non-Affective Psychosis, Front. Psychiatry (2020), 10.3389/fpsyt.2020.00846. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [10]. Bilgrami ZR, et al. , Construct validity for computational linguistic metrics in individuals at clinical risk for psychosis: Associations with clinical ratings, Schizophr. Res 245 (Jul. 2022) 90–96, 10.1016/J.SCHRES.2022.01.019. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [11]. Xu W, et al. , Fully automated detection of formal thought disorder with Time-series Augmented Representations for Detection of Incoherent Speech (TARDIS), J. Biomed. Inform 126 (2022), 10.1016/j.jbi.2022.103998. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [12]. Xu W, Portanova J, Chander A, Ben-Zeev D, Cohen T, The Centroid cannot hold: Comparing Sequential and Global estimates of Coherence as Indicators of Formal Thought Disorder, AMIA Annu. Symp. Proc (2020). [ PMC free article ] [ PubMed ] [ Google Scholar ] [13]. Just SA, et al. , Validation of natural language processing methods capturing semantic incoherence in the speech of patients with non-affective psychosis, Front. Psychiatry 14 (Jul. 2023) 1208856, 10.3389/FPSYT.2023.1208856/BIBTEX. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [14]. Tang SX, et al. , Natural language processing methods are sensitive to sub-clinical linguistic differences in schizophrenia spectrum disorders, NPJ Schizophr. (2021), 10.1038/s41537-021-00154-3. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [15]. Devlin J, Chang M-W, Lee K, Google KT, Language AI, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Naacl-Hlt (2019, 2018.). [ Google Scholar ] [16]. Bubeck S et al. , “Sparks of Artificial General Intelligence: Early experiments with GPT-4,” 2023. [ Google Scholar ] [17]. Fradkin I, Nour MM, Dolan RJ, Theory-Driven Analysis of Natural Language Processing measures of Thought Disorder using Generative Language Modeling, Biol Psychiatry Cogn Neurosci Neuroimaging 8 (10) (Oct. 2023) 1013–1023, 10.1016/J.BPSC.2023.05.005. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [18]. Sharpe V, Mackinley M, Nour Eddine S, Wang L, Palaniyappan L, and Kuperberg G, “GPT-3 reveals selective insensitivity to global vs. local linguistic context in speech produced by treatment-naive patients with positive thought disorder,” Jan. 2024. doi: 10.1101/2024.07.08.602512. [ DOI ] [ PubMed ] [ Google Scholar ] [19]. Li C, Xu W, Pakhomov S, Bradley E, Ben-Zeev D, and Cohen T, “Bigger But Not Better: Small Neural Language Models Outperform Large Language Models in Detection of Thought Disorder,” Mar. 2025, Accessed: Jun. 08, 2025. [Online]. Available: https://arxiv.org/pdf/2503.20103 . [ DOI ] [ PMC free article ] [ PubMed ] [20]. Xu W, Portanova J, Chander A, Ben-Zeev D, and Cohen T, “The Centroid Cannot Hold: Comparing Sequential and Global Estimates of Coherence as Indicators of Formal Thought Disorder,” Sep. 2020, PsyArXiv. doi: 10.31234/osf.io/sfkqc. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [21]. Sap M, Horvitz E, Choi Y, Smith NA, Pennebaker JW, Recollection versus Imagination: Exploring Human memory and Cognition via Neural Language Models, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2020, pp. 1970–1978, 10.18653/V1/2020.ACL-MAIN.178. [ DOI ] [ Google Scholar ] [22]. Sap M, Jafarpour A, Choi Y, Smith NA, Pennebaker JW, and Horvitz E, “Quantifying the narrative flow of imagined versus autobiographical stories,” Proc Natl Acad Sci U S A, vol. 119, no. 45, 2022, doi: 10.1073/pnas.2211715119. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [23]. L. Team and A. @ Meta, “The Llama 3 Herd of Models,” 2024. [ Google Scholar ] [24]. Ben-Zeev D, et al. , Mobile RDoC: using Smartphones to Understand the Relationship between Auditory Verbal Hallucinations and need for Care, Schizophr Bull Open (2020), 10.1093/schizbullopen/sgaa060. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [25]. Bradley ER, et al. , Quantifying abnormal emotion processing: a novel computational assessment method and application in schizophrenia, Psychiatry Res. 336 (Jun. 2024) 115893, 10.1016/J.PSYCHRES.2024.115893. [ DOI ] [ PubMed ] [ Google Scholar ] [26]. Kay SR, Fiszbein A, and Opler LA, “The positive and negative syndrome scale (PANSS) for schizophrenia,” Schizophr Bull, vol. 13, no. 2, 1987, doi: 10.1093/schbul/13.2.261. [ DOI ] [ PubMed ] [ Google Scholar ] [27]. Andreasen NC, Flaum M, and Arndt S, “The Comprehensive Assessment of Symptoms and History (CASH): An Instrument for Assessing Diagnosis and Psychopathology,” Arch Gen Psychiatry, vol. 49, no. 8, 1992, doi: 10.1001/archpsyc.1992.01820080023004. [ DOI ] [ PubMed ] [ Google Scholar ] [28]. Bojanowski P, Grave E, Joulin A, Mikolov T, Enriching Word Vectors with Subword Information, Trans Assoc Comput Linguist 5 (Jul. 2016) 135–146, 10.1162/tacl_a_00051. [ DOI ] [ Google Scholar ] [29]. Reimers N, Gurevych I, Sentence-BERT: Sentence embeddings using siamese BERT-networks, in: In EMNLP-IJCNLP 2019–2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, 2020, 10.18653/v1/d19-1410. [ DOI ] [ Google Scholar ] [30]. Miani A, Hills T, Bangerter A, Interconnectedness and (in)coherence as a signature of conspiracy worldviews, Sci. Adv 8 (43) (Oct. 2022) 3668, 10.1126/SCIADV.ABQ3668/SUPPL_FILE/SCIADV.ABQ3668_SM.PDF. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [31]. Fang A, Macdonald C, Ounis I, and Habel P, “Using word embedding to evaluate the coherence of topics from twitter data,” SIGIR 2016 - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1057–1060, Jul. 2016, doi: 10.1145/2911451.2914729;WGROUP: STRING:ACM. [ DOI ] [ Google Scholar ] [32]. Lewis M et al. , “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,” Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880, Oct. 2019, doi: 10.18653/v1/2020.acl-main.703. [ DOI ] [ Google Scholar ] [33]. Turney PD, Pantel P, From frequency to meaning: Vector space models of semantics, J. Artif. Intell. Res (2010), 10.1613/jair.2934. [ DOI ] [ Google Scholar ] [34]. Cohen T, Widdows D, “Empirical distributional semantics, Methods and Biomedical Applications” (2009), 10.1016/j.jbi.2009.02.002. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [35]. Kuperberg GR, Language in schizophrenia Part 1: an Introduction, Lang Linguist Compass 4 (8) (2010) 576, 10.1111/J.1749-818X.2010.00216.X. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] [36]. Anderson JR, A spreading activation theory of memory, J Verbal Learning Verbal Behav 22 (3) (Jun. 1983) 261–295, 10.1016/S0022-5371(83)90201-3. [ DOI ] [ Google Scholar ] [37]. Lund A et al. , “UC Merced Proceedings of the Annual Meeting of the Cognitive Science Society Title Semantic and Associative Priming in High-Dimensional Semantic Space Publication Date Semantic and Associative Priming in High-Dimensional Semantic Space,” Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 17, no. 0, p. 17, 1995. [ Google Scholar ] [38]. Kerns JG, Verbal communication impairments and cognitive control components in people with schizophrenia, J. Abnorm. Psychol 116 (2) (2007) 279. [ DOI ] [ PubMed ] [ Google Scholar ] [39]. Kuperberg GR, Language in schizophrenia Part 2: what can psycholinguistics bring to the study of schizophrenia…and vice versa? Lang Linguist Compass 4 (8) (2010) 590, 10.1111/J.1749-818X.2010.00217.X. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] ACTIONS View on publisher site PDF (1.4 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Textual Acoustic Grounding for Generalizable LLM-Based Deepfake Voice Detection#818682
• Textual Acoustic Grounding for Generalizable LLM-Based Deepfake Voice Detection#808690
• What Are You Listening to? Temporal Music Grounding for Audio-to-Text Large Language Models#794262
• What Are You Listening to? Temporal Music Grounding for Audio-to-Text Large Language Models#790243
• Large Language models are stochastic measuring devices#82773