Large Language Models' Ability to Assess Main Concepts in Story Retelling: A Proof-of-Concept Comparison of Human Versus Machine Ratings.

Large Language Models' Ability to Assess Main Concepts in Story Retelling: A Proof-of-Concept Comparison of Human Versus Machine Ratings - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice Am J Speech Lang Pathol . 2025 Mar 31;34(6 Suppl):3636–3646. doi: 10.1044/2025_AJSLP-24-00400 Large Language Models' Ability to Assess Main Concepts in Story Retelling: A Proof-of-Concept Comparison of Human Versus Machine Ratings Jacquie Kurland a, ✉ , Vishnupriya Varadharaju b , Anna Liu c , Polly Stokes a , Ankita Gupta b , Marisa Hudspeth b , Brendan O'Connor b Author information Article notes Copyright and License information a Department of Speech, Language, and Hearing Sciences, University of Massachusetts Amherst b Manning College of Information & Computer Sciences, University of Massachusetts Amherst c Department of Mathematics and Statistics, University of Massachusetts Amherst Disclosure: The authors have declared that no competing financial or nonfinancial interests existed at the time of publication. ✉ Correspondence to Jacquie Kurland: jacquie@umass.edu Editor-in-Chief: Michael de Riesthal Editor: Nichol Castro ✉ Corresponding author. Received 2024 Sep 14; Revised 2025 Jan 14; Accepted 2025 Jan 21; Issue date 2025 Dec. Copyright © 2025 American Speech-Language-Hearing Association PMC Copyright notice PMCID: PMC12714372  PMID: 40163765 Abstract Purpose: Despite an abundance of manual, labor-intensive discourse analysis methods, there remains a dearth of clinically convenient, psychometrically robust instruments to measure change in real-world communication in aphasia. The Brief Assessment of Transactional Success (BATS) addresses this gap while developing automated methods for analyzing story retelling discourse. This study investigated automation of main concept (MC) analysis of stories by comparing scores from three large language models (LLMs) to those of human raters. Method: After watching/listening to each of the eight short video/audio BATS stimuli and retelling each story, 96 persons with aphasia (PWA; n = 48 female) engaged in topic-constrained conversations over Zoom with 94 familiar and 107 unfamiliar conversation partners (CPs). CPs then retold each story as co-constructed during their conversations with PWA. Audio files from the resulting 1,760 story retells were transcribed using Python and AssemblyAI's speech-to-text application programming interface. Each MC was first scored by human raters for presence, accuracy, and completeness. Raters used a semiautomated application, MainConcept. For each transcript, an MC composite ratio score was obtained. We evaluated three state-of-the-art LLMs: two proprietary models, GPT-4 and GPT-4o, and one open-source model, Llama-3-70B. The interrater reliability between each LLM versus human MC scoring was assessed via the Pearson correlation coefficient and reliability coefficients based on the generalizability theory (G-theory). Results: The Pearson correlation coefficients indicate strong positive linear relationships between LLM and human MC scores. G-theory reliability coefficients also indicate reliable scoring between LLM and human scoring across the spectrum of participants and conditions. Conclusions: This promising proof-of-concept study affirms the reliability of three LLMs in evaluating BATS story retell MCs and justifies ongoing investigation into their use. Providing clinicians and clinical researchers with automated tools for analyzing discourse without the need for prohibitively labor-intensive manual scoring could be a paradigm shift, potentially revolutionizing the aphasia intervention landscape. Models for aphasia intervention have moved from an impairment-based to a participation-based framework ( Brady et al., 2016 ). With this shift, a greater emphasis on communication has challenged the field to investigate ways of promoting aphasia recovery that can generalize to, if not focus on, language and communication skills for the real world ( Carragher et al., 2015 ; DeDe et al., 2019 ; Elman, 2007 ; Kagan, 1995 , 1998 ; McVicker et al., 2009 ; Wilkinson & Wielaert, 2012 ). Unfortunately, clinicians in the United States and elsewhere have described numerous barriers in time, skill, efficiency, and confidence in the acquisition and analysis of aphasic discourse data ( Bryant et al., 2017 ; Cruice et al., 2020 ; Rose et al., 2014 ). Moreover, despite an abundance of tools and procedures used in clinical research over the last 25 years to capture treatment effects in conversation ( Azios et al., 2022 ) and in other co-constructed communication ( Carragher et al., 2024 ), there remains a dearth of clinically convenient, psychometrically robust instruments that can reliably and validly measure change in real-world communication across the spectrum of aphasia severity. A new tool, the Brief Assessment of Transactional Success (BATS), in communication in aphasia ( Kurland & Stokes, 2021 ) aims to address this clinical–academic gap and to provide methods for automating the labor-intensive aspects of discourse analysis. Methods for automating speech to text and analyzing BATS story retells using natural language processing (NLP) tools demonstrated reliability and potential for overcoming the time-intensive traditional method of human-annotated main concept analysis (MCA; Kurland et al., 2025 ). The current study focuses on a proof of concept for leveraging large language models (LLMs) to assess main concepts (MCs) in story retelling. MCA ( Nicholas & Brookshire, 1995 ; Richardson & Dalton, 2016 , 2020 ) is a commonly used macrolinguistic measure of a person's ability to convey the gist of a picture, picture series, story, procedure, or other means of eliciting connected discourse. Traditionally, MCs have been developed through a labor-intensive iterative task of discovering what relevant concepts are produced by, for example, 50% of a sample of non-aphasic volunteers in response to a request to describe a picture, retell a story, and so forth. MCs are traditionally defined as an utterance that contains a subject, one main verb, and an optional object, with or without subordinate clauses (e.g., “A man gets up early”). MC checklists include a rating scale for judging the presence, accuracy, and completeness of each MC, for example, Accurate and Complete = 3; Accurate but Incomplete (one or more missing essential elements, e.g., “A young man wakes up”) = 2; Inaccurate but Complete (one or more inaccurate essential elements, e.g., “A woman wakes up early in the morning”) = 2; Inaccurate and Incomplete = 1; and not present = 0 ( Richardson & Dalton, 2016 , 2020 ). Checklists also typically include information such as the percentage of participants from the normative sample who produced each MC accurately and completely, as well as examples of acceptable alternative productions (e.g., “A guy gets out of bed at six o'clock in the morning”). Checklists have become a “clinician-friendly” means of overcoming the time- and resource-intensive burden of analyzing discourse in aphasia ( Kim et al., 2019 , 2021 ), rendering discourse analysis somewhat more clinically feasible. A recent tool for semiautomating the MCA process enhances clinical feasibility even further. The cloud-based MainConcept app ( Cavanaugh et al., 2021 , 2022 ) facilitates manual MCA by prompting a user to rate the presence, accuracy, and completeness at the utterance level and automatically scoring each MC by comparison to MC checklists ( Richardson & Dalton, 2016 , 2020 ). For the current study, human (“gold standard”) scoring of MCs was completed using an adaptation of the MainConcept app that included norm-generated MCs from recently developed MC checklists for all 16 BATS stimuli ( Kurland & Stokes, 2021 ). Reliability in scoring MCs, both manually (e.g., Boyle, 2014 ; Nicholas & Brookshire, 1995 ) and using the semiautomated app ( Cavanaugh et al., 2022 ), have been observed to be acceptable, although not perfect. In addition, although the app makes human MC scoring more efficient, it is still time consuming, which may render it clinically impractical. Thus, developing a fully automated tool for analyzing story gist, with comparable or better reliability to that of human raters, would enhance the clinical feasibility of discourse analysis in aphasia, and for that matter, in numerous populations of individuals with communication impairments. In this work, we explore the feasibility of using LLMs to automate MC scoring and evaluate them against human-annotated scores. LLMs have shown impressive performance in many language processing tasks ( Brown et al., 2020 ; Touvron et al., 2023 ), including tasks that require assessing the compatibility of two text inputs ( Viswanathan et al., 2024 ; Zhang et al., 2023 ), closely related to our MC scoring task that requires comparing each MC against the story retell. Traditionally, development of such automated text assessment methods (e.g., evaluating summaries, fact-checking, measuring semantic similarity) requires constructing data sets with human-annotated examples and training models on these examples. Recent advances in NLP, driven by LLMs, have led to a new modeling approach using specific key words or phrases as prompts to guide model responses ( Kojima et al., 2022 ; Wei et al., 2022 ), with little or no training data. This approach is especially promising for MC scoring tasks, where collecting large amounts of training data with aphasic patients could be quite challenging. Even the AphasiaBank, the world's only openly available data source for spoken language and communication in aphasia, has only acquired 450 transcripts from people with aphasia since 2007 ( MacWhinney & Fromm, 2024 ). While this is a very large number by aphasia research standards, it is a small number for training language models. Thus, we investigate the use of LLMs for MC scoring as it can enable the analysis of novel stimuli without requiring extensive annotations. Recently, LLMs have been employed to evaluate texts (both human and artificial intelligence [AI] generated) across various dimensions, including factual accuracy ( Min et al., 2023 ) and relevance ( Saad-Falcon et al., 2024 ). Moreover, LLMs have demonstrated higher agreement with human annotators compared to traditional methods like lexical overlap across a range of language generation tasks ( Fu et al., 2023 ; Lin et al., 2024 ), indicating their potential effectiveness in automated text scoring–related tasks. More recently, there have also been increasing efforts to use LLMs for automatic aphasic discourse analysis. For example, Salem et al. (2023) explored how LLMs could predict a speaker's intended word in discourse samples with paraphasias. Similarly, Cong et al. (2024) applied LLMs to measure language surprisals as a metric to capture word-level and structure-level impairments commonly found in individuals with aphasia. Purohit et al. (2023) investigated ChatGPT's ability to resolve word searches during circumlocution. In our work, we apply LLMs to score a given story retell on the presence, accuracy, and completeness of MCs. We then compare the scores obtained via an LLM to those of human raters and assess their correlation. We experiment with several different LLMs, including commercial LLMs (GPT4 and GPT4o, both provided by OpenAI company via API access) and an open alternative (Llama 3, which can be deployed on any information technology [IT] infrastructure of choice) and compare their performance on the MC scoring task. Finally, we compute correlations between human and LLM MC scores to assess the feasibility of harnessing LLMs in this context. We hypothesize that it is feasible for LLMs to assess MCs in story retells and expect that, at least for the moment, the commercial LLMs will surpass performance of the open-weight model, as compared to the “gold standard” human MC scoring. Method Participants Ninety-six persons with aphasia (PWA) of varying types and severity, 94 familiar conversation partners (FCPs), and 107 unfamiliar conversation partners (UCPs) participated in the current phase of BATS test development. Clinical and demographic information on participants is shown in Table 1 . Table 1. Clinical and demographic information on participants. Variable Aphasic participants Non-aphasic CP ( n = 96) ( n = 201) Age (years) 59 ± 10.9 47 ± 18.4 29–83 18–85 Sex 48 female 149 female, 2 NB Education 16.3 ± 2.8 16.7 ± 3.0 11–24 12–28 Race/ethnicity 76 Caucasian 172 Caucasian 10 African American 10 African American 5 Hispanic/Latino 11 Hispanic/Latino 4 Asian American 4 Asian American 1 mixed race 4 mixed race Familiarity with aphasic conversation partner 94 FCP, 107 UCP WAB-AQ 79.7 ± 16.8 26.8–98 WAB aphasia classification 16 anomic/NAPW 50 anomic 15 Broca's 9 conduction 1 transcortical motor 1 transcortical sensory 4 Wernicke's Open in a new tab Note. Only four participants scored 10 on Naming and Word Finding subtests of the WAB, although 16 participants scored above the 93.8 WAB cutoff, suggesting the “non-aphasic per WAB” (NAPW) label. CP = conversation partner; NB = nonbinary; FCP = familiar CP; UCP = unfamiliar CP; WAB = Western Aphasia Battery–Revised ( Kertesz, 2006 ); AQ = Aphasia Quotient. The institutional review board (IRB) of the University of Massachusetts Amherst approved the study (IRB Protocol 3129). Informed consent was obtained from all participants via phone or video conferencing software, with signatures obtained via Docusign. Data Acquisition In separate test sessions, each PWA was paired with an FCP and an UCP, as described in greater detail in a reliability study ( Kurland et al., 2025 ). PWA watched and/or listened to BATS video/audio stimuli and then were asked to retell the story in as much detail as they could remember. Immediately following each story retell, they engaged in timed 6-min conversations with a partner who was naïve to the story, with the goal of co-constructing the story using any means to aid communication. The partner then was instructed to retell the story in as much detail as possible. This was repeated for four stimuli per session, one from each of four stimulus types ( Kurland & Stokes, 2021 ). A total of 1,760 story retell samples were acquired and recorded via Zoom. Transcription and Preprocessing Story retell transcripts were generated from de-identified audio files using AssemblyAI's speech-to-text AI models ( https://www.assemblyai.com/products/speech-to-text ). Python scripts were used to interact with the model's API and generate the transcripts. Transcripts were checked for accuracy with very high interrater agreement on a random sample of 10% aphasic and 10% of non-aphasic story retells (94% and 96%, respectively), as described in Kurland et al. (2025) . Prior to performing analyses, preprocessing of transcripts included removal of punctuation, stop words, and irrelevant utterances that occurred mostly as opening and closing comments, for example, “Should I go now?” at the beginning, “Okay, I'm done” at the end, and so forth. Raw AI-generated transcripts were utilized for LLM MCA, in alignment with the goal of investigating the plausibility of leveraging LLMs to automate MCA. Human (“Semiautomated”) MCA Using MCs acquired from a normative sample of 96 non-aphasic story retellers ( Kurland & Stokes, 2021 ), we adapted the open-source web app MainConcept ( Cavanaugh et al., 2021 , 2022 ) to score the presence, accuracy, and completeness of MCs in PWA and CP story retellings of the 16 BATS stimuli. As each stimulus has a different number of MCs, an MC composite ratio was used to enable comparisons between stimuli. This ratio was calculated by summing the MC scores and dividing them by the number of MCs for each stimulus. LLM MCA Since human evaluation is costly, we next use LLMs to automatically score the presence, accuracy, and completeness of MCs in each story retell. LLMs have shown impressive performance on several NLP tasks such as textual entailment ( Brown et al., 2020 ) and fact verification ( Min et al., 2023 ), which are related to our task of scoring MCs. For instance, like the textual entailment and fact verification tasks, MC scoring requires the model to determine if an MC is present and accurately mentioned in a story retell. In particular, we provide an LLM with a story retell and expert-annotated MCs as input and ask it to score each MC on a 4-point scale (accurate and complete, accurate but incomplete or inaccurate but complete, inaccurate and incomplete, and absent). We also prompt the LLM to provide a brief explanation for each score prediction. Finally, we aggregate the LLM's individual MC scores for each stimulus to obtain a composite MC score, similar to the process followed for human analysis of MCs. For prompt development, we consider two common approaches for prompting LLMs: zero-shot prompting and in-context learning. In zero-shot prompting, the model is given only instructions to complete a task ( Brown et al., 2020 ). In contrast, in-context learning provides a few examples in the prompt in addition to the instructions ( Brown et al., 2020 ). We experimented informally with both methods and found anecdotally that zero-shot prompting performed better than in-context learning. Prior studies have shown mixed results regarding the benefits of in-context learning ( Sprague et al., 2024 ). Since our primary goal was to establish a proof of concept for using LLMs in MC scoring, we opted to go with zero-shot prompting. Zero-shot prompting also relies on shorter prompts, making it more cost efficient than in-context learning. 1 A detailed prompt, developed over multiple iterations, is provided in the Appendix . In recent years, several LLMs are being made available, including (a) commercial LLMs provided by a company via API access such as GPT (by OpenAI), Claude (by Anthropic), and PaLM (by Google) and (b) open alternatives, including Llama 3, Mistral, and Olmo, which can be deployed by any software service provider on their own (HIPAA compliant) IT infrastructure. Research in some NLP problems suggests that commercial LLMs sometimes outperform open LLMs for certain language understanding tasks; however, their performance on applications relevant to clinical practice needs further investigation. Furthermore, commercial LLMs also require submitting all narrative input and prompts to the service via a remote API controlled by the company, raising enormous concerns about institutional dependence and patient privacy. Thus, in this work, we experiment with two commercial and one open-weight language model to benchmark them on the MC scoring task. Commercial closed-source models, which currently achieve state-of-the-art performance on many language processing tasks ( Brown et al., 2020 ; OpenAI et al., 2023 ), are accessible only through public APIs without access to internal components like parameters, training data, or architecture. The proprietary nature of closed-source LLMs restricts access to their design, development, and data, raising significant concerns about the reproducibility and interpretability of their generated responses. In contrast, open-weight models can be downloaded and used locally, providing more flexibility for customization (e.g., adapting model parameters for specific tasks) and improved privacy, as they do not require sending sensitive data to external servers. Given the strengths and limitations of both model families, we explore both: proprietary models to establish an upper bound on the LLM performance for this task and open-weight models to assess performance in more practical and privacy demanding applications. For proprietary models, we consider GPT-4 (gpt-4-0314; released March 2023), a large multimodal model that accepts text or image inputs and outputs text, and GPT-4o (gpt-4o-2024-05-13; released May 2024), a newer version that provides comparable performance with lower costs and faster inference. Among open-weight models, we use Llama-3 (Meta-Llama-3-70B-Instruct; released April 2024), which is currently one of the best-performing open-weight models after the proprietary models. We access GPT-4 and GPT-4o through OpenAI's API 2 and Llama-3 via together.ai's public API. 3 A language model is trained to predict the next word given the previous context. When predicting the next word, the model computes a probability distribution over all the words in the model's vocabulary and selects the next word based on this probability distribution. Various hyperparameters are provided with public API access to control the selection of the next word based on the computed probability distribution. In particular, the temperature hyperparameter controls how deterministic the output is. The lower the temperature, the more deterministic the results in the sense that the highest probable next token is always selected. Increasing temperature could lead to more randomness, which encourages more diverse or creative outputs. To ensure reproducibility, we set temperature = 0 in all our experiments. For the remaining hyperparameters, we select the default values mentioned by the API providers. 4 Statistical Analysis The interrater reliability between each of the LLM versus human MC scoring was assessed via two methods: the Pearson correlation coefficient and the reliability coefficients based on the generalizability theory (G-theory; Brennan, 2001 ; Vangeneugden et al., 2005 ). In the classic test theory, it is assumed that the variance of a measurement is the sum of the true score variance and a simple error variance. The reliability of a measuring instrument is then defined as the ratio of the true score variance to the total score variance, which is the same as the Pearson correlation coefficient when measurements are parallel, that is, centered around the same true score with equal variances. On the other hand, G-theory is a conceptual framework in that it recognizes that there are multiple sources of error variance for measurement scales and reliability coefficients are defined by identifying the universal score variance and the appropriate error variance and taking the ratio of the universal score variance and the total variance. According to Brennan (2001) , the universal score of a subject is defined as the expected value of the mean scores, each of which is averaged over all the instances of the measurement procedure in the subject's universe of generalization. The variance of universe scores over all persons in the population is called universe score variance. It has conceptual similarities with true score variance in classical test theory but allows users to define the universe of generalization. G-theory generalizes classical test theory and encompass into one framework the various existing definitions of reliability such as interrater reliability, test–retest reliability, and internal consistency. For our data, each patient retold stories about multiple stimuli multiple times with different partners and the MCs were scored by GPT-4, GPT-4o, Llama-3-70B, and human. Therefore, there are multiple sources of measurement error in the scores: variation due to individuals including patients and CPs, the stimuli, their interactions, and the raters and their interactions. Thus, in the following we adopt a linear mixed model to tease apart the various variance components properly and calculate the G-theory generalizability coefficients accordingly. As a baseline analysis, we also calculated the Pearson correlation coefficients between each LLM rater and the human rater. In applying G-theory to evaluate the interrater reliability between each LLM versus human rater, we fit a linear mixed model to the scores based on the LLM and the human rater and identified sources of variability in MC scoring, which includes the variations due to individuals, σ 2 ( p ); the stimuli, σ 2 ( t ); their interactions, σ 2 ( pt ); the raters, σ 2 ( r ); and their interactions, σ 2 ( rt ), σ 2 ( pr ), and σ 2 ( prt ). The relative reliability coefficient, which is a measure of consistency between the LLM and human scoring, is calculated as (Equation 10 in Vangeneugden et al., 2005 ) in the following formula. Here, we considered the universe of generalization includes all the potential raters but only the 16 fixed stimuli; that is, we did not consider in our calculation the reliability of our instrument being generalized to potential other stimuli outside of the 16 we chose: σ 2 p + σ 2 pt σ 2 p + σ 2 pt + σ 2 pr + σ 2 prt . (1) The absolute reliability coefficient, which is a measure of agreement between the LLM and human scoring, is calculated as (Equation 13 in Vangeneugden et al., 2005 ): σ 2 p + σ 2 t + σ 2 pt σ 2 p + σ 2 t + σ 2 r + σ 2 pt + σ 2 pr + σ 2 prt . (2) The confidence intervals were generated based on the bootstrap technique of parametric methods with spherical random effects ( Jiang et al., 2022 ). The calculations were carried out using the statistical software R ( R Core Team, 2024 ) and the R package lme4 ( Bates et al., 2015 ). Results The Pearson correlation coefficients between the GPT-4, GPT-4o, and Llama-3-70B and the human scores are .90, .90, and .89, respectively. The relative reliability coefficient is .88 (95% confidence interval [CI; .86, .89]) for GPT-4, .87 (95% CI [.85, .89]) for GPT-4o, and .86 (95% CI [.83, .87]) for Llama-3-70B. The absolute reliability coefficient is .86 (95% CI [.74, .90]) for GPT-4, .86 (95% CI [.82, .90]) for GPT-4o, and .84 (95% CI [.75, .87]). The results indicate strong consistency and agreement between LLM and human MC scores. Figure 1 shows a representative scatterplot, the correlation between GPT-4 and human scores, and Table 2 shows the correlation matrix among the four scoring methods. Figure 1. Open in a new tab Correlation of human versus GPT-4 MC composite ratio scores. MC = main concept. Table 2. Correlation matrix among the three large language models and human main concept scoring methods. Rater GPT-4 GPT-4o Llama-3-70B Human GPT-4 1.00 .95 .94 .90 GPT-4o .95 1.00 .94 .90 Llama-3-70B .94 .94 1.00 .89 Human .90 .90 .89 1.00 Open in a new tab Note. Large language models (LLMs) include GPT-4 and GPT-4o ( OpenAI et al., 2023 ) and Llama-3-70B ( Touvron et al., 2023 ). The estimated variance components based on each linear mixed model between an LLM scoring method and the human scoring are presented in Table 3 . The variation of the individual conversation pairs, which is called the universe score variance in G-theory, is .0443, .0465, and .0459, respectively. The largest sources of variation are from stimuli and the interaction between individuals and stimuli. The different raters contribute a small portion of the source of variation, which led to the high interrater reliability coefficients as reported above. More detailed analysis of the sources of variation is currently in preparation in a separate article. Table 3. Estimated variance components based on each linear model between a large language model and human scoring method. Source of variation Estimated variance component GPT-4 GPT-4o Llama-3-70B PWA and CP (p) .0443 .0465 .0459 Rater (r) .0018 .0012 .0025 Stimuli (t) .0044 .0044 .0041 p × r .0018 .0026 .0025 p × t .0164 .0168 .0171 r × t .0003 .0004 .0002 p × r × t (error) .0068 .0066 .0080 Open in a new tab Note. PWA = person with aphasia; CP = conversation partner. Discussion The purpose of the current study was to explore the feasibility of automating MCA of story retells by comparing scores from three LLMs to those of the “gold standard” human raters. Our findings demonstrated the reliability and dependability of two proprietary LLMs (GPT-4 and GPT-4o) and one open-source LLM (Llama-3-70B) in assessing BATS story retell MCs by demonstrating a high degree of concordance with the human raters. This preliminary investigation suggests a promising future for employing AI in aphasia discourse analysis, particularly in tasks that involve co-construction of communication with a non-aphasic CP, wherein a non-aphasic discourse sample can be acquired and analyzed ( Carragher et al., 2015 ; Ramsberger & Rende, 2002 ). One advantage of analyzing the non-aphasic partner's production of a co-constructed communication task includes the obvious face validity inherent in a communication task that requires joint action to achieve transactional success. There is also the benefit of a discourse sample that is likely to be more accurately transcribed by a speech-to-text AI model ( Kurland et al., 2025 ), reducing the need for human intervention in the discourse analysis pipeline. In the current study, we utilized the raw AI-generated transcripts of all participants in keeping with our goal of demonstrating proof of concept that LLMs can be used to automate the process of story retell analysis. However, an important caveat for future investigations of story retells, when co-construction of stories is not the focus of analysis, is that a “human-in-the-loop” approach may be preferred, particularly in cases of aphasic participants' speech when it is sparse, severely dysarthric or apraxic, or when a nonnative English speaker's accent is pronounced. In such cases, human editing of transcripts prior to analysis will improve reliability. The positive results of this proof-of-concept study confirm our hypothesis, demonstrating feasibility of harnessing LLMs to assess MCs in this story retelling paradigm. The results also motivate further examination of methods for overcoming study limitations that might improve the current study results and justify investigating other applications in which LLMs might be used to automate discourse analysis in aphasia research. The current study used a data set of story retells of 16 stimuli that had already undergone the time-intensive “gold standard” process of first obtaining human-generated MCs ( Kurland & Stokes, 2021 ). Both proprietary and one open-source LLM proved to be adept at assessing the presence, accuracy, and completeness of those MCs in story retells, with a high degree of agreement with human raters. One limitation of the current study is that we only tested three LLMs, including only one open-source LLM, to benchmark them on the MC scoring task. Studies based on closed-source LLMs often suffer from reproducibility and stability issues. For instance, frequent model updates, whether through version changes or complete model replacements, and discontinuations of previous versions can make it difficult to reproduce findings. Open-source models offer better reproducibility and stability since they can be downloaded and repeatedly run locally over time. Additionally, since open-source LLMs provide access to their architectures and weights, they can be customized for specific use-cases through fine tuning (a process where model parameters are adapted for a specific task). For instance, fine tuning has been shown to improve a model's ability to better follow user instructions, thus improving the effectiveness and reliability of LLM responses ( Hewitt et al., 2024 ). Another challenge with closed-source LLMs is that they operate within proprietary frameworks that obscure their internal mechanisms ( Cottier et al., 2024 ). Limited access to training data sets, preprocessing pipelines, and model architecture constrains third-party audits and independent evaluations. This opacity compounds the challenge of identifying and addressing any biases in LLM responses as well as impedes interpretability studies. On the other hand, open-source LLMs promote transparency through access to their architectures, weights, details of model outputs (e.g., token probabilities), and training methodologies. This openness enables researchers and developers to conduct model examinations, auditing for biases, testing fairness metrics, and identifying model vulnerabilities ( Manchanda et al., 2024 ). As newer open-source LLMs become available, future work will continue to assess their feasibility against the current benchmarks. Another limitation of the current study is that it did not include rigorous testing of zero-shot versus few-shot prompts, an area for future investigation since current NLP research has found that LLMs exhibit significant output variability due to prompt choice ( Sclar et al., 2024 ). Since our primary goal in this work was to establish a proof of concept for using LLMs in MC scoring, we mainly focused on zero-shot prompting. We only anecdotally observed that zero-shot prompting outperformed in-context learning. Notably, prior studies have shown mixed results regarding the benefits of in-context learning ( Sprague et al., 2024 , Figure 8). For instance, tasks involving math and symbolic reasoning often benefit from alternate prompting strategies like in-context learning and chain-of-thought prompting. However, for tasks involving language understanding and reading comprehension—closely related to the MC scoring task in our study—the performance of in-context learning is often comparable to zero-shot prompting. Given these mixed results, there is significant potential for future work to explore the use of in-context examples for MC scoring more thoroughly. More generally, now that the feasibility of the method has been demonstrated, future work will include rigorously testing specific prompts and language processing pipelines to optimize the robustness and reliability of the MC scoring task. The findings of this feasibility study suggest promising implications for clinicians and clinical researchers whose work would benefit from tools that automate discourse analysis processing. The implications go beyond assessing story retelling narratives for MCs in aphasia, including, for example, the potential for LLM generation of MCs. Because LLMs perform well at both text summarization and comparison of two discourse samples, future models may be able to both generate and assess MCs, enabling the use of novel, personalized stimuli. Another potential application might include leveraging LLMs to evaluate other macrolinguistic measures besides MCs, for example, to assess story grammar. With the advent of such advancements, it is plausible that LLMs will have broad application across communication-impaired populations and settings, from examining “story goodness” in traumatic brain injury (e.g., Lê et al., 2012 ) to assessing pragmatic aspects of language such as question asking, a metric used in right-hemisphere disorders (e.g., Fannin et al., 2023 ), to measuring coherence in discourse, a metric used in dementia (e.g., Dijkstra et al., 2004 ). In the foreseeable future, it is not hyperbolic to suggest that leveraging LLMs to analyze real-world discourse could revolutionize life participation approaches to patient-centered language intervention across populations and settings across the lifespan. Data Availability Statement The data sets analyzed during the current study are available from the corresponding author on reasonable request. Where optional consent was obtained, some of the data sets analyzed during the current study are available in the Kurland-BATS corpus repository ( https://aphasia.talkbank.org/ ). Acknowledgments Research reported in this publication was supported by a UMass Amherst Provost's Office Interdisciplinary Research Grant and the National Institute on Deafness and Other Communication Disorders Grant R21DC020265. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We would like to thank our aphasic and non-aphasic study participants. We would also like to thank the research assistants who rated main concepts, notably Caroline Pare, Mia Tittmann, and Dominique Wander. Appendix LLM Prompt Details We use the following prompt for all three models: You are a system that analyzes the presence, accuracy, and completeness of the N main concepts in the given story retell, which describes a video about {stimulus}. The {stimulus} video contains {N} main concepts as follows: {Numbered list of main concepts for this stimulus} In a story retell, each main concept is judged as follows: -  ‘AC’ if it is present, accurate, and complete. -  ‘AI’ if all elements are true, but not all elements are present in the story retell. -  ‘IC’ if all elements are present, but at least one element is not true in the story retell. -  ‘II’ if not all elements are present, and at least one element is not true in the story retell. -  ‘AB’ if a story retell does not include any reference to a main concept. Please provide the analysis in a JSON format with the following structure: { “ID”: “####”, // Replace with input delimited in user message “MCScores”: { “1”: { “Score”: “AC/AI/IC/II/AB”, // Replace with the appropriate score “Explanation”: “Detailed explanation of why the score was assigned in under 30 words.” }, “2”: { “Score”: “AC/AI/IC/II/AB”, “Explanation”: “Detailed explanation of why the score was assigned in under 30 words.” }, // Repeat for all {N} main concepts } } The final response should only be a JSON structure with the format mentioned. ID: {retell ID} Story retell: {retell transcript} Publisher Note: This article is part of the Special Issue: Select Papers From the 53rd Clinical Aphasiology Conference. Funding Statement Research reported in this publication was supported by a UMass Amherst Provost's Office Interdisciplinary Research Grant and the National Institute on Deafness and Other Communication Disorders Grant R21DC020265. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Footnotes 1 LLMs are typically priced based on the number of input and output tokens, where a token represents a subword unit. Therefore, in-context prompting with examples, which requires longer input prompts, is more expensive compared to zero-shot prompting. 2 https://platform.openai.com/ 3 Although model parameters for Llama-3 are available for download, we use a public API access optimized for faster inference and the required hardware infrastructure. 4 We set top_p = 0.7, top_k = 50 and repetition_penalty = 1; see API documentations for details on these hyperparameters. References Azios, J. H., Archer, B., Simmons-Mackie, N., Raymer, A., Carragher, M., Shashikanth, S., & Gulick, E. (2022). Conversation as an outcome of aphasia treatment: A systematic scoping review. American Journal of Speech-Language Pathology, 31(6), 2920–2942. 10.1044/2022_AJSLP-22-00011 [ DOI ] [ PubMed ] [ Google Scholar ] Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. 10.18637/jss.v067.i01 [ DOI ] [ Google Scholar ] Boyle, M. (2014). Test–retest stability of word retrieval in aphasic discourse. Journal of Speech, Language, and Hearing Research, 57(3), 966–978. 10.1044/2014_JSLHR-L-13-0171 [ DOI ] [ PubMed ] [ Google Scholar ] Brady, M. C., Kelly, H., Godwin, J., Enderby, P., & Campbell, P. (2016). Speech and language therapy for aphasia following stroke. Cochrane Database of Systematic Reviews, (6), Article CD000425. 10.1002/14651858.CD000425.pub4 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Brennan, R. L. (2001). Generalizability theory. Springer-Verlag. 10.1007/978-1-4757-3456-0 [ DOI ] [ Google Scholar ] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf [PDF] [ Google Scholar ] Bryant, L., Spencer, E., & Ferguson, A. (2017). Clinical use of linguistic discourse analysis for the assessment of language in aphasia. Aphasiology, 31(10), 1105–1126. 10.1080/02687038.2016.1239013 [ DOI ] [ Google Scholar ] Carragher, M., Mok, Z., Steel, G., Conroy, P., Pettigrove, K., Rose, M. L., & Togher, L. (2024). Towards efficient, ecological assessment of interaction: A scoping review of co-constructed communication. International Journal of Language & Communication Disorders, 59(3), 831–875. 10.1111/1460-6984.12957 [ DOI ] [ PubMed ] [ Google Scholar ] Carragher, M., Sage, K., & Conroy, P. (2015). Preliminary analysis from a novel treatment targeting the exchange of new information within storytelling for people with nonfluent aphasia and their partners. Aphasiology, 29(11), 1383–1408. 10.1080/02687038.2014.988110 [ DOI ] [ Google Scholar ] Cavanaugh, R., Dalton, S., & Richardson, J. (2022, November). Novel web applications to facilitate discourse analysis in everyday clinical practice [Paper presentation]. American Speech-Language-Hearing Association Annual Convention, New Orleans, Louisiana. [ Google Scholar ] Cavanaugh, R., Richardson, J., & Dalton, S. G. (2021). MainConcept: An open-source web-app for scoring main concept analysis (R package Version 0.0.1.0000) [Computer software]. https://github.com/aphasia-apps/mainConcept Cong, Y., LaCroix, A. N., & Lee, J. (2024). Clinical efficacy of pre-trained large language models through the lens of aphasia. Nature Scientific Reports, 14(1), Article 15573. 10.1038/s41598-024-66576-y [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Cottier, B., You, J., Martemianova, N., & Owen, D. (2024). How far behind are open models? Epoch AI. Retrieved January 8, 2025, from https://epoch.ai/blog/open-models-report Cruice, M., Botting, N., Marshall, J., Boyle, M., Hersh, D., Pritchard, M., & Dipper, L. (2020). UK speech and language therapists' views and reported practices of discourse analysis in aphasia rehabilitation. International Journal of Language & Communication Disorders, 55(3), 417–442. 10.1111/1460-6984.12528 [ DOI ] [ PubMed ] [ Google Scholar ] DeDe, G., Hoover, E., & Maas, E. (2019). Two to tango or the more the merrier? A randomized controlled trial of the effects of group size in aphasia conversation treatment on standardized tests. Journal of Speech, Language, and Hearing Research, 62(5), 1437–1451. 10.1044/2019_JSLHR-L-18-0404 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Dijkstra, K., Bourgeois, M. S., Allen, R. S., & Burgio, L. D. (2004). Conversational coherence: Discourse analysis of older adults with and without dementia. Journal of Neurolinguistics, 17(4), 263–283. 10.1016/S0911-6044(03)00048-4 [ DOI ] [ Google Scholar ] Elman, R. J. (2007). The importance of aphasia group treatment for rebuilding community and health. In L. LaPointe (Ed.), Aphasia and related neurogenic language disorders (3rd ed., pp. 300–308). Thieme Medical. 10.1097/01.TLD.0000299884.31864.99 [ DOI ] [ Google Scholar ] Fannin, D. K., Elleby, J., Tackett, M., & Minga, J. (2023). Intersectionality of race and question-asking in women after right hemisphere brain damage. Journal of Speech, Language, and Hearing Research, 66(1), 314–324. 10.1044/2022_JSLHR-22-00327 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2023). GPTScore: Evaluate as you desire. In K. Duh, H. Gomez, & S. Bethard (Eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, pp. 6556–6576). Association for Computational Linguistics. 10.18653/v1/2024.naacl-long.365 [ DOI ] [ Google Scholar ] Hewitt, J., Liu, N. F., Liang, P., & Manning, C. D. (2024). Instruction following without instruction tuning. arXiv. 10.48550/arXiv.2409.14254 [ DOI ] Jiang, Z., Raymond, M., DiStefano, C., Shi, D., Liu, R., & Sun, J. (2022). A Monte Carlo study of confidence interval methods for generalizability coefficient. Educational and Psychological Measurement, 82(4), 705–718. 10.1177/00131644211033899 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Kagan, A. (1995). Revealing the competence of aphasic adults through conversation: A challenge to health professionals. Topics in Stroke Rehabilitation, 2(1), 15–28. 10.1080/10749357.1995.11754051 [ DOI ] [ PubMed ] [ Google Scholar ] Kagan, A. (1998). Supported conversation for adults with aphasia: Methods and resources for training conversation partners. Aphasiology, 12(9), 816–830. 10.1080/02687039808249575 [ DOI ] [ Google Scholar ] Kertesz, A. (2006). Western Aphasia Battery—Revised. Harcourt Assessment. [ Google Scholar ] Kim, H., Kintz, S., & Wright, H. H. (2021). Development of a measure of function word use in narrative discourse: Core lexicon analysis in aphasia. International Journal of Language & Communication Disorders, 56(1), 6–19. 10.1111/1460-6984.12567 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Kim, H., Kintz, S., Zelnosky, K., & Wright, H. H. (2019). Measuring word retrieval in narrative discourse: Core lexicon in aphasia. International Journal of Language & Communication Disorders, 54(1), 62–78. 10.1111/1460-6984.12432 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. arXiv. 10.48550/arXiv.2205.11916 [ DOI ] Kurland, J., Liu, A., Varadharaju, V., Stokes, P., & Cavanaugh, R. (2025). Reliability of the Brief Assessment of Transactional Success in communication in aphasia. Aphasiology, 39(3), 363–384. 10.1080/02687038.2024.2351029 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Kurland, J., & Stokes, P. (2021). Let's talk real talk: An argument to include conversation in a D-COS for aphasia research with an acknowledgment of the challenges ahead. Aphasiology, 32(4), 475–478. 10.1080/02687038.2017.1398808 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Lê, K., Coelho, C. A., Mozeiko, J., Krueger, F., & Grafman, J. (2012). Predicting story goodness performance from cognitive measures following traumatic brain injury. American Journal of Speech-Language Pathology, 21(2), S115–S125. 10.1044/1058-0360(2012/11-0114) [ DOI ] [ PubMed ] [ Google Scholar ] Lin, Y.-C., Neville, J., Stokes, J. W., Yang, L., Safavi, T., Wan, M., Counts, S., Suri, S., Andersen, R., Xu, X., Gupta, D., Jauhar, S. K., Song, X., Buscher, G., Tiwary, S., Hecht, B., & Teevan, J. (2024). Interpretable user satisfaction estimation for conversational systems with large language models. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers, pp. 11100–11115). Association for Computational Linguistics. 10.18653/v1/2024.acl-long.598 [ DOI ] [ Google Scholar ] MacWhinney, B., & Fromm, D. (2024). TalkBank methods for studying spoken discourse. In A. P.-H. Kong (Ed.), Spoken discourse impairments in the neurogenic populations: A state-of-the-art, contemporary approach (pp. 97–109). Springer. https://aphasia.talkbank.org/publications/2024/MacWhinney24.pdf [PDF] [ Google Scholar ] Manchanda, J., Boettcher, L., Westphalen, M., & Jasser, J. (2024). The open-source advantage in large language models (LLMs). 10.48550/arXiv.2412.12004 [ DOI ] McVicker, S., Parr, S., Pound, C., & Duchan, J. (2009). The communication partner scheme: A project to develop long‐term, low-cost access to conversation for people living with aphasia. Aphasiology, 23(1), 52–71. 10.1080/02687030701688783 [ DOI ] [ Google Scholar ] Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FActScore: Fine-grained atomic evaluation of factual precision in long form text generation . In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 12076–12100). Association for Computational Linguistics. 10.18653/v1/2023.emnlp-main.741 [ DOI ] [ Google Scholar ] Nicholas, L. E., & Brookshire, R. H. (1995). Presence, completeness, and accuracy of main concepts in the connected speech of non-brain-damaged adults and adults with aphasia. Journal of Speech and Hearing Research, 38(1), 145–156. 10.1044/jshr.3801.145 [ DOI ] [ PubMed ] [ Google Scholar ] OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., … Zoph, B. (2023). GPT-4 technical report. arXiv. 10.48550/arXiv.2303.08774 [ DOI ] Purohit, A. K., Upadhyaya, A., & Holzer, A. (2023). ChatGPT in healthcare: Exploring AI chatbot for spontaneous word retrieval in aphasia. In CSCW '23 Companion: Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing (pp. 1–5). Association for Computing Machinery. 10.1145/3584931.3606993 [ DOI ] [ Google Scholar ] Ramsberger, G., & Rende, B. (2002). Measuring transactional success in the conversation of people with aphasia. Aphasiology, 16(3), 337–353. 10.1080/02687040143000636 [ DOI ] [ Google Scholar ] R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/ [ Google Scholar ] Richardson, J. D., & Dalton, S. G. (2016). Main concepts for three different discourse tasks in a large non-clinical sample. Aphasiology, 30(1), 45–73. 10.1080/02687038.2015.1057891 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Richardson, J. D., & Dalton, S. G. H. (2020). Main concepts for two picture description tasks: An addition to Richardson and Dalton, 2016. Aphasiology, 34(1), 119–136. 10.1080/02687038.2018.1561417 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Rose, M., Ferguson, A., Power, E., Togher, L., & Worrall, L. (2014). Aphasia rehabilitation in Australia: Current practices, challenges and future directions. International Journal of Speech-Language Pathology, 16(2), 169–180. 10.3109/17549507.2013.794474 [ DOI ] [ PubMed ] [ Google Scholar ] Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. In K. Duh, H. Gomez, & S. Bethard (Eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1: Long Papers, pp. 338–354). Association for Computational Linguistics. 10.18653/v1/2024.naacl-long.20 [ DOI ] [ Google Scholar ] Salem, A. C., Gale, R. C., Fleegle, M., Fergadiotis, G., & Bedrick, S. (2023). Automating intended target identification for paraphasias in discourse using a large language model. Journal of Speech, Language, and Hearing Research, 66(12), 4949–4966. 10.1044/2023_JSLHR-23-00121 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2024). Quantifying language models' sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. arXiv. 10.48550/arXiv.2310.11324 [ DOI ] Sprague, Z., Yin, F., Rodriguez, J. D., Jiang, D., Wadhwa, M., Singhal, P., Zhao, X., Ye, X., Mahowald, K., & Durrett, G. (2024). To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. arXiv. 10.48550/arXiv.2409.12183 [ DOI ] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). LLaMA: Open and eficient foundation language models. arXiv. 10.48550/arXiv.2302.13971 [ DOI ] Vangeneugden, T., Laenen, A., Geys, H., Renard, D., & Molenberghs, G. (2005). Applying concepts of generalizability theory on clinical trial data to investigate sources of variation and their impact on reliability. Biometrics, 61(1), 295–304. 10.1111/j.0006-341X.2005.031040.x [ DOI ] [ PubMed ] [ Google Scholar ] Viswanathan, V., Gashteovski, K., Gashteovski, K., Lawrence, C., Wu, T., & Neubig, G. (2024). Large language models enable few-shot clustering. Transactions of the Association for Computational Linguistics, 12, 321–333. 10.1162/tacl_a_00648 [ DOI ] [ Google Scholar ] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E. H., Xia, F., Le, Q., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (pp. 1–14). Neural Information Processing Systems. https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html [ Google Scholar ] Wilkinson, R., & Wielaert, S. (2012). Rehabilitation targeted at everyday communication: Can we change the talk of people with aphasia and their significant others within conversation? Archives of Physical Medicine and Rehabilitation, 93(1), S70–S76. 10.1016/j.apmr.2011.07.206 [ DOI ] [ PubMed ] [ Google Scholar ] Zhang, Y., Wang, A., & Shang, J. (2023). ClusterLLM: Large language models as a guide for text clustering. In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 13903–13920). Association for Computational Linguistics. 10.18653/v1/2023.emnlp-main.858 [ DOI ] [ Google Scholar ] Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Data Availability Statement The data sets analyzed during the current study are available from the corresponding author on reasonable request. Where optional consent was obtained, some of the data sets analyzed during the current study are available in the Kurland-BATS corpus repository ( https://aphasia.talkbank.org/ ). Articles from American Journal of Speech-Language Pathology are provided here courtesy of American Speech-Language-Hearing Association ACTIONS View on publisher site PDF (444.6 KB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Validity of Large Language Model Comparative Judgment for Universal Writing Screening#613624
• Large Language Models Exhibit Human-Like Bayesian Hypocrisy#1130621
• Decomposing and Measuring Evaluation Awareness#1131543
• The role of large language models in the writing of surgical reviews: fact or fantasy?#342469
• Epistemic Warrant for LLM Recommendations: Characterizing the Basis for Reliance When Ground Truth Is Unavailable#926995