Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior Hanbo Xie1
Akshay K. Jagadish 3
Lan Pan 1
Robert C. Wilson1,2 1 School of Psychology, Georgia Institute of Technology Center of Excellence for Computational Cognition, Georgia Institute of Techonology 3 Princeton AI Lab, Princeton University [email protected], [email protected], [email protected], [email protected]
arXiv:2605.05091v1 [q-bio.NC] 6 May 2026
2
Abstract Computational cognitive models discovered using large language models have so far relied solely on behavioral data. However, it is well-known that models produced from the behavioral trajectory alone are typically underdetermined. In this work, we explore the use of Think Aloud traces as an additional form of data constraint during automated model discovery. When applied to the domain of risky decision-making, we find that the models discovered with think-aloud achieve significantly improved predictive performance on held-out data. Additionally, we find that the discovered models belong to different structural classes than those discovered from behavior alone for the majority of participants (69.4%), specifically, it shifts from Explicit comparator towards Integrated utility. These results suggest that process-level language data not only improve model fit, but also systematically reshape the structure of the discovered cognitive models, enabling the identification of mechanisms that are not recoverable from behavior alone.
Introduction Understanding human thought processes is a central goal of cognitive science. A common approach is to construct computational models that describe the mechanisms underlying human decisions and to select and validate these models based on behavioral data. However, behavioral observations alone often leave substantial ambiguity, as different computational mechanisms can produce similar patterns of choice (Wilson & Collins, 2019). One modality that has been shown to complement behavioral trajectories in cognitive science is think-aloud reasoning traces. The think-aloud protocol records process-level data in participants by capturing their intermediate reasoning in natural language, which potentially encodes structural information about decision processes (Ericsson & Simon, 1980). While early work raised concerns about their validity and scalability, recent advances in natural language processing and large language models (LLMs) have enabled systematic anal-
ysis of think-aloud traces, leading to renewed interest in linking verbal reports to the underlying cognitive process (Wurgaft et al., 2025; Xie et al., 2024, 2025; Zhang et al., 2025). However, prior work in this space has mostly focused on either validating the reliability of think-aloud data or developing methods to analyze it. In contrast, a more fundamental question remains largely unexplored: Do think-aloud reasoning traces facilitate the discovery of computational structures that cannot be identified from behavior alone? To address this question, we compare the models discovered from behavioral data alone with those derived using behavioral and thought-aloud data jointly, within the automatic model discovery framework (Rmus et al., 2025). We find that incorporating think-aloud leads to significant improvements in the quality of the models produced, in terms of predictive performance, while producing qualitatively different computational models, suggesting that process-level language data systemically reshapes the structure of the discovered models.
Results Think-aloud improves automated model discovery outcomes In this work, we use an automatic model discovery framework (Rmus et al., 2025), called GeCCo, in which an LLM (LLaMA-3.1-70B) iteratively generates candidate computational models as executable functions mapping task inputs to choices and evaluates them based on their fit to held-out data. At each step, the current best-performing model is provided as a reference, and the LLM is prompted to propose alternative model structures that improve upon it. We apply this framework to a risky decision-making dataset with think-aloud reports (N = 72), where participants verbalize their reasoning before making binary choices across 19 trials based on Kahneman and Tversky (1979).
A
B
C
Integrated Utility Option A
Integrate
𝑈
Option B
Integrate
𝑈
Choice
𝑈=
𝑤 𝑝 𝑢(𝑣)
Explicit Comparator 𝑈 𝑈
Comparison
Choice
∆𝑈 = 𝑈 − 𝑈
Rule-based Operator if gain If loss
Piecewise/ combine Choice conditional rule … 𝑢 𝑣 = …
Figure 1: Think-aloud improves model discovery and induces systematic shifts in discovered mechanisms. A, Trial-averaged held-out BIC for each participant’s best discovered model under the behavior-only and think-aloud conditions. Lower BIC indicates better out-of-sample fit. Each pair of points is connected within participant; larger points show the group mean ± 95% CI. B, Schematic definitions of the three main mechanism families identified from normalized computation graphs: Integrated utility, which transforms and integrates each option before comparison; Explicit comparator, which computes utilities and compares them directly (e.g., ΔU = UA − UB ); and Rule-based operator, which applies piecewise or conditional rules before combining information into a choice. C, Row-normalized transition matrix from the behavior-only best-model cluster to the think-aloud best-model cluster. Numbers indicate proportions (counts shown below). Off-diagonal mass indicates mechanism shifts, with 69.4% of participants transitioning to a different cluster. We compare two conditions: models discovered using behavioral data alone, and models discovered using both behavioral and think-aloud data as input to the LLM. In both cases, candidate models are evaluated on 10 held-out trials that were not included in the prompt used during model generation. For each participant, we repeat the discovery process 10 times with the same data splits and report the best-fitting model based on Bayesian Information Criterion (BIC) Watanabe, 2013. Among the 72 participants, 59.7% showed lower heldout BIC under the think-aloud condition. A paired t-test confirmed this difference, showing that models discovered with think-aloud data have significantly lower heldout BIC than those based on behavior alone (t(71) = −3.41, p = 0.001), indicating improved out-of-sample model fit (Figure 1A).
Think-aloud traces reshapes the structure of discovered models To characterize model structure, we convert each discovered program into a normalized computation graph, extract structural features, and cluster them using HDBSCAN (McInnes et al., 2017). This yields three major mechanism families—Integrated utility, Explicit com-
parator, and Rule-based operator—plus a small set of unassigned models (Figure 1B). We then assign each participant’s best model under each condition to a cluster and compute a row-normalized transition matrix (behavior → think-aloud). Upon clustering, we examine whether incorporating think-aloud data changes the structure of model identified by the discovery process. The resulting transition matrix (Figure 1C) shows substantial off-diagonal mass: 69.4% of participants are assigned to different mechanism clusters when models are discovered with versus without think-aloud data. These transitions are not uniform: for example, models in the Explicit comparator cluster frequently shift to Integrated utility (57%), while other clusters exhibit more distributed transition patterns. These transitions correspond to concrete changes in computational organization rather than superficial code variation. For example, some participants shift from models that explicitly compare option values to models that first transform and integrate gains, losses, or probabilities within each option before comparison; others show the reverse pattern. This indicates that think-aloud data can redirect discovery toward qualitatively different mechanism fam-
ilies (Figure 1B,C). Importantly, these two effects—improved model fit and shifts in mechanism clusters—are closely linked. Behavioral data alone often underdetermines the space of computational models, as multiple structures can produce similar choice patterns. The inclusion of thinkaloud data reduces this indeterminacy by providing additional constraints on the cognitive mechanism that is underlying the observed behavior and how it is structured, thereby favoring different classes of explanations. Together, these results show that incorporating thinkaloud data does not merely improve model fit but can systematically alter the computational structures identified by automated model discovery frameworks. This suggests that process-level language data plays a functional role in constraining the space of admissible models, enabling the identification of alternative mechanisms that are not recoverable from behavior alone.
References Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological review, 87(3), 215. Kahneman, D., & Tversky, A. (1979). 0prospect theory: An analysis of decision under risk, 1 econometrica. March, 47(2), 2635291. McInnes, L., Healy, J., Astels, S., et al. (2017). Hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11), 205. Rmus, M., Jagadish, A. K., Mathony, M., Ludwig, T., & Schulz, E. (2025). Generating computational cognitive models using large language models. arXiv preprint arXiv:2502.00879. Watanabe, S. (2013). A widely applicable bayesian information criterion. The Journal of Machine Learning Research, 14(1), 867–897. Wilson, R. C., & Collins, A. G. (2019). Ten simple rules for the computational modeling of behavioral data. Elife, 8, e49547. Wurgaft, D., Prystawski, B., Gandhi, K., Zhang, C. E., Tenenbaum, J. B., & Goodman, N. D. (2025). Scaling up the think-aloud method. arXiv preprint arXiv:2505.23931. Xie, H., Xiong, H.-D., & Wilson, R. (2024). From strategic narratives to code-like cognitive models: An llmbased approach in a sorting task. First conference on language modeling. Xie, H., Xiong, H.-D., & Wilson, R. C. (2025). Rethinking think-aloud in the age of language models. OSF Preprints. Zhang, Z., Xie, H., Baker, T. E., Peters, M., & Wilson, R. (2025). Linking strategies to think aloud in a
stochastic learning task. Proceedings of the Annual Meeting of the Cognitive Science Society, 47.