Locating acts of mechanistic reasoning in student team conversations with mechanistic machine learning Kaitlin Gili1 , Mainak Nistala, Kristen Wendell1 , and Michael C. Hughes1,♢ 1
arXiv:2604.21870v1 [physics.ed-ph] 23 Apr 2026
2
Department of Computer Science, Tufts University, Medford, MA, U.S.A. and Departments of Mechanical Engineering and Education, Tufts University, Medford, MA, U.S.A. (Dated: April 24, 2026)
STEM education researchers are often interested in identifying moments of students’ mechanistic reasoning for deeper analysis, but have limited capacity to search through many team conversation transcripts to find segments with a high concentration of such reasoning. We offer a solution in the form of an interpretable machine learning model that outputs time-varying probabilities that individual students are engaging in acts of mechanistic reasoning, leveraging evidence from their own utterances as well as contributions from the rest of the group. Using the toolkit of intentionallydesigned probabilistic models, we introduce a specific inductive bias that steers the probabilistic dynamics toward desired, domain-aligned behavior. Experiments compare trained models with and without the inductive bias components, investigating whether their presence improves the desired model behavior on transcripts involving never-before-seen students and a novel discussion context. Our results show that the inductive bias improves generalization – supporting the claim that interpretability is built into the model for this task rather than imposed post hoc. We conclude with practical recommendations for STEM education researchers seeking to adopt the tool and for ML researchers aiming to extend the model’s design. Overall, we hope this work encourages the development of mechanistically interpretable models that are understandable and controllable for both end users and model designers in STEM education research.
I.
Introduction
We are interested in analyzing students’ mechanistic reasoning (MR) in-moment, during collaborative problem-solving conversations. Such conversations are a key activity in the professional practice of science and engineering [1] and therefore a crucial element of science and engineering education [2–4]. To solve technical problems, science and engineering students often need to determine the relationships between different parameters of a natural or designed system. Mechanistic reasoning supports this work because it is a form of thinking focused on characterizing the entities and activities that cause a particular system performance [5, 6]. But to solve difficult technical problems collaboratively, as professional scientists and engineers do, students need not only to engage in mechanistic reasoning as individual thinkers; they must also express and revise their mechanistic reasoning to fellow learners in-the-moment, during conversations. The STEM education literature has established that students – even as young as first graders – can express mechanistic reasoning during teacher-facilitated and small-group conversations [7, 8], but these studies typically analyze at most a handful of conversations due to time constraints. In research settings where a large number of student conversations are recorded, it is often resource-intensive to manually identify the segments of text that contain the most evidence of mechanistic reasoning at the individual student level and the collective group level to analyze. Prior work has observed that evidence of mechanistic reasoning is often dynamic and unstable in student conversations. Students can shift in and out of a mechanistic reasoning frame on the order of minutes [9]. Thus, having an automated tool that can
pinpoint these locations in time-series data that span hours can save STEM education researchers a tremendous amount of time and mental energy in choosing which data segments to prioritize. Given the nature of the data task, it is natural to wonder whether advancements in machine learning (ML) can offer a solution. A first consideration might be to offload the task to the context-window of a large-language model (LLM) [10], which can take text as input and can be prompt-engineered by the user to output references to the most important text segments. While simple sounding, such an approach requires back-and-forth wrestling with a probabilistic token generator [11, 12] that lacks inherent interpretability [13, 14]. Different from a black-box model with a post-hoc explanation for its prediction [13], an inherently interpretable ML model is “constrained in model form so that it is either useful to someone, or obeys structural knowledge of the domain” [13]. In the latter case, predictions arise from inductive biases built in by design. Domain-aligned mathematical mechanisms are explicitly encoded to constrain and guide the model toward the intended behavior. Both model designers and users stand to benefit from this approach. Designers are able to have a clear understanding of the inductive biases built into their implementation, can obtain evidence of their value, and then can remove or refine them as necessary for model improvement. Users, in turn, can place greater confidence in a tool built around domain expertise and tailored to their specific task. Importantly, both designers and users are able to obtain a mechanistic account of the tool’s behavior, enabling more informed decisions about appropriate use. In this work, we explore an inherently interpretable ML design by leveraging switching-state models, which
2 represent the latent state dynamics and their probabilities that drive entity behavior over time [15–17]. Recent work [17] introduces an efficiently trainable ML model containing interpretable mechanisms for how entity observations (e.g student utterances) influence the probabilities of latent state changes (e.g evidence of mechanistic reasoning or not) for individual entities as well as the entire group. See Fig. 1 for an intuitive toy illustration of this desired probabilistic model behavior in our student discussion settings. The model in Ref. [17] offers a useful starting point for an interpretable ML approach to our data task and motivates the following research questions: 1. RQ1: How might the mechanistic ML model introduced in Ref. [17] be adapted with a specialized inductive bias to support the identification of mechanistic reasoning in group discussions of STEM learners? 2. RQ2: Does the specialized inductive bias improve the desired ML model behavior when generalizing to discussions involving a novel STEM problem and previously unseen students? And if so, by how much? The remainder of this article is organized as follows. Sec. II provides background with respect to prior work on mechanistic reasoning and presents examples of mechanistic reasoning evidence from our dataset, which we use to train and test our ML model. We also provide a brief overview of prior work incorporating ML tools to accelerate STEM education research. Sec. III describes the model from Ref. [17] and how we adapt it to identify regions of high mechanistic reasoning evidence in transcripts of student team conversations. We present our ML modeling decisions in technical detail. First, we outline our probabilistic modeling assumptions and discuss their benefits and limitations for our task. Second, we describe the specialized inductive bias that we bake into the model as this work’s contribution. Namely, we design a feedback mechanism that steers the individual and group latent state probabilities based on the evidence of mechanistic reasoning present in a student utterance. Our classifier-based feedback is trained with human annotation supervision, but can generalize to new unannotated transcripts. In Sec. IV, we introduce task-specific metrics for desired model behavior and our hypotheses for these metrics when the feedback mechanism components are present or absent from the model construction. We then examine experimental results for our hypotheses in two data settings. First, we evaluate predictions on data from previously unseen students discussing a STEM problemthat appears the training data. Second, we evaluate predictions on transcripts that include both unseen students and a new STEM problem. We interpret positive empirical support for each hypothesis as evidence that the model is interpretable by design for the target task. In Sec. V, we present a demonstration of the tool’s
output on a transcript segment and provide recommendations for STEM education researchers using our tool. In Sec. VI, we conclude with an outline of future research directions for ML-experts looking to design tools with inherent interpretability.
II. A.
Background
Analyzing students’ mechanistic reasoning in conversational data
Drawing from depictions of mechanistic reasoning in philosophy of science [5], Russ and colleagues developed a framework for characterizing its presence in classroom science conversations [8]. It puts forward seven hierarchical elements of mechanistic reasoning that may be evident in student talk. Those categories are (1) describe the target phenomenon, (2) identify the set-up conditions for the phenomenon, (3) identify the entities that play a role in producing the phenomenon, (4-6) identify the properties, activities, and organization of those entities that affect the outcome of the phenomenon, and (7) chain the current state of the entities backward to what happened previously or forward to what will happen next. By describing these categories as hierarchical, we mean both that higher categories represent more sophisticated reasoning and that higher categories generally require lower categories to have already been accomplished. Russ et al. also proposes two categories of mechanistic reasoning not included in the hierarchy. Reasoning with analogies involves making explicit comparisons to other phenomena. Reasoning with animated models involves simulating entities and activities through physical gestures or manipulation of objects. We omitted these two categories from our analysis, but they present interesting possibilities for future work. To see how the framework’s seven hierarchical categories build on each other, consider the following example from Russ and colleagues’ original study. First-grade students had observed that a crumpled sheet of paper and a heavy book reached the ground at the same time when dropped from the same height. This result surprised students because when the same sheet of paper had been smooth and flat, it had lagged far behind the falling book. Their teacher gathered the students and asked them to share their thinking about why the crumpled paper and book “tied”. One student pointed out, “If it’s balled up it’s still not heavy it’s the same size.” Here, her mechanistic reasoning included identifying the paper as a key entity (category #3), identifying the paper’s weight as a key property (category #4), and identifying the process of crumpling as a key activity (category #5). She also chained the activity to the property (category #7) by noting that while the crumpling changed the paper’s shape, it maintained its heaviness. It is important to note that Russ et al’s analytic framework does not consider the accuracy of students’ claims or the completeness of their explanations. Conversations may exhibit strong
3
j=2
o N R
“Pennsylvania. I didn’t going skiing for the rst time until like high school.”
0 R
j=1
M
t=2
t=2
N M o R
t=1
0
R
Probability
M
R
0
N M o R
1
1
Probability
“Whitetail Ski Resort. Yeah. It’s actually in Pennsylvania.” Probability
“I’m going to - we’re going to - go all out on the assumptions here.”
M
N M o R
R
0
t=1

j=1 j=2
M
1
M
Probability
“Is this a pure substance? Do we have to worry about that? The fact that it’s air and water mixed together?”
fi
FIG. 1: A toy illustration of our desired ML model behavior, using student utterances from the snowmaking problem. Top: When evidence of mechanistic reasoning (MR) is present at t = 1, the model sharply increases that student’s latent probability of MR at time t = 2 and modestly increases the other student’s probability. Bottom: When no MR is observed at t = 1, the model keeps both students’ latent probabilities low at t = 2. Comparison: When a student’s observed speech contains mechanistic reasoning evidence at t = 1, the latent probability of MR is closer to one (top left) than when no evidence is present (bottom left).
evidence of mechanistic reasoning without expressing the “correct” or “complete” canonically accepted mechanistic account of a phenomenon. Since its development, science and engineering education researchers have used Russ et al.’s framework to characterize the use of mechanistic reasoning by groups of learners. These studies show how mechanistic reasoning in-the-moment contributes to group progress in modeling or designing physical systems [18, 19]. For example, engineering students who engaged in higher levels of mechanistic reasoning as they discussed their physical prototypes tended to make iterations that improved the prototype’s performance [20]. B.
Evidence of mechanistic reasoning in our dataset
The data used in this work was collected by audio recording five homework problem solving sessions carried out by small groups of undergraduate engineering students in a thermal fluid systems course at Tufts university. Each student group was assigned two open-ended problems that involved analyzing ill-defined thermal-fluid systems related to real-world situations, such as making snow for a ski resort or keeping ice cream cold at a summer ice cream stand [21]. All groups received one problem that was the same about constructing a resistance heater to raise the temperature of the surrounding gas, and one problem designed around a scenario intended to interest at least one student in the group. For example, the group that received the ski resort snowmaking problem contained a student who was an avid skier. We take the snowmaking problem as a central example to show what evidence of mechanistic reasoning looks like in our dataset. The problem statement provided to students is written as:
Snowmaking problem Choose your favorite ski resort and the desired depth of snow for the best skiing, and use thermodynamics to determine how long it will take to cover the ski trails in that amount of snow. You may assume that one snow gun uses about 100 gallons of water per minute and that the compressor can produce 50 cubic feet per minute of air. Using the seven categories from Russ et al.’s framework, we determined what counts as evidence of each element of mechanistic reasoning within the specific problem space. Here we present Russ’ codes applied to the snowmaking problem with examples from our dataset that our coded positively: 1. Describe the target phenomenon: express the phenomenon that is trying to be explained (e.g. “covering the ski trail in snow”). Example: “Assume that’s the surface area [of the resort] that we need to cover [with snow].” 2. Identify setup conditions: recognize the initial organization of the environment that is required for the mechanism to run (e.g. “setting up snowguns around the trail”). Example: “Let’s say they have a snow gun - they have like ten on a trail.” 3. Identify entities: recognize the things in the environment that interact in cause-effect ways to produce the phenomenon (e.g. “air”, “water”, “snow”, “ice”). Example: “The fact that it’s air and water mixed together?” 4. Identify activities of entities: recognize the individual entity actions and mutual entity interactions in the mechanism that produce the phenomena (e.g. “mixing together”, “coming out of the nozzle”, “turning into ice”). Actions can include motions; interactions can include forces. Example: “Alright so we have a volumetric flow rate of water and air, which is gonna be.” 5. Identify properties of entities: recognize the entity characteristics or current states that are relevant for the mechanism to run. (e.g. “density”, “compressed”, “temperature”, “sparse”). Example: “So we could assume that it’s at zero degrees Celsius like just about to freeze, but it is entirely [liquid] water.” 6. Identify organization of entities: recognize the spatial organization of entities in the mechanism (e.g. “inside of the tube”, “different nozzles”). Example: “The fact that it’s air and water mixed together?”
4 7. Chain backward and forward: use knowledge about the causal structure of the world to make claims about what must have happened previously to bring about the current state of things (backward) or what will happen next given that certain entities or activities are present now (forward) (e.g. “expanding air and expanding water are turning into ice”). Example: “Outside of the tube, there’s expanding air and expanding water which is turning to ice.” The codes for all six problems in our dataset can be found in Appendix A. Our human analysis procedure for code creation and validation, including an inter-rater reliability (IRR) test to show evidence of reproducibility, can be found in Appendix B. In Sec. III, we present our ML method for automating mechanistic reasoning detection that is trained and tested on 10 total transcripts containing student discussion of these problems with humanannotated labels. C.
Using ML tools to accelerate STEM education research
There has been a rapid development of ML tools for the purpose of helping STEM education researchers speed up their data analysis procedures. Given that the majority of data studied in STEM education research is qualitative – such as written responses and transcripts from digital recordings – improvements in large language models (LLMs) [22–24] have enabled the large-scale automation of data annotation [10, 25–31]. Automated methods for data labeling include both in-context learning (ICL) and transfer learning with the possibility of foundation model supervised fine-tuning. In ICL, an engineered prompt for the task along with a few human labeled examples are fed directly into a chatbot window (e.g. ChatGPT) and the model then generates labels for the rest of the user provided data. The chatbot window is LLM-powered, but does not require any user interaction with the pre-trained LLM components, making it accessible to non-ML experts. Under the assumption that offthe-shelf LLMs can be guided with few data to perform well on a general set of language tasks, ICL is extremely data efficient. However, it also happens to be sensitive to the provided prompt [32, 33] and the selection and order of data shown [34]. Computing reliable statistics across models with different initializations (i.e. different prompts) is a challenge given that even the same exact input prompt may lead to different outputs due to the probabilistic nature of chatbots [35]. Lastly, ICL methods lack inherent interpretability for their predictions. Rather than feed text directly into a context-window, one can consider a transfer learning approach. In transfer learning, one leverages an LLM as an encoder that transforms text into numerical vectors (i.e. text embeddings) that can be used to train a downstream classification model for a particular task. Under the assumption that off-the-shelf text-embeddings are representationally meaningful, this approach removes the need to re-train
the foundation LLM model and reduces training costs only to the downstream model. To drop the assumption, one can fine-tune the pre-trained LLM (i.e. update the foundation model weights) to produce embeddings that are more aligned with the downstream classification task. In recent work, Gili et al. [29] show that for the downstream task of identifying students’ mechanistic reasoning in written explanations, fine-tuning yields only slightly better performance than using off-the-shelf LLM embeddings, while requiring nearly 23× more training time. Overall, transfer learning offers more inherent interpretability than in-context learning (ICL) because it produces text embeddings that can be directly analyzed and allows researchers to select an interpretable downstream classifier (e.g. a linear model). The majority of ICL and transfer learning methods explored for data annotation purposes in STEM education has been focused around annotating static data – isolated student responses that are assumed to be time independent. For example, Fussell et al. [28] provide an evaluation of different LLMs for identifying specific skills in students’ typed lab notes. In Ref. [36], Watts and her co-authors separate Russ’ original categories for identifying mechanistic reasoning into binary (present or not present) codes specific to organic chemistry problems. They then compare the performance differences between neural network model variants that are trained to classify each code on independent student explanations. In Ref. [29], the authors also adapt Russ’ original categories into binary codes, but specific to classical mechanics problems and with a PER-ML co-design approach that takes the ML model decisions into account when refining the qualitative coding framework. The authors train probabilistic classifiers with different LLM encoders, and evaluate the trade-off between performance and computational resource use. Our work is similar in that we initially adapt Russ’s original categories into binary codes, but we do so for time-varying conversational data from a thermal-fluid systems course and use the resulting signals to estimate the probability of mechanistic reasoning, rather than producing a binary yes/no label. The latter point separates our work from that of Ref. [37], which uses LLMs to classify talk moves in transcripts of tutoring sessions. They report classification results from two types of prompt-engineering ICL methods with GPT-3.5-turbo and transfer learning without finetuning. In two of the scenarios, they treat each student utterance in the transcript as independent, despite the underlying sequential nature of the data. In the third scenario, they prompt GPT-3.5-turbo to reproduce the entire transcript with annotations line-by-line, allowing for the context of all surrounding lines to influence the annotations. Our work focuses on constructing an ML model that is inherently interpretable so that the user can understand the context mechanism – how evidence present in a single student utterance influences their state and that of all others. This enables users to search for locations of high mechanistic evidence at the level of an
5 individual student or the entire group, using probabilities calibrated to the evidence present rather than a binary label. III.
RQ1: Specializing ML Inductive Biases to Locate Mechanistic Reasoning
In this section, we describe the design of an inherently interpretable ML model for the task of locating evidence of mechanistic reasoning in student conversations. Our model is a task-specific adaptation of the hierarchical switching-state recurrent dynamical model (HSRDM), a previously proposed [17] general framework for modeling interacting entities over time. We specialize this model in two ways: (1) to model text transcripts of conversations between students, and (2) so latent states at both team-level and student-level directly correspond to notations of mechanistic reasoning. The subsections below describe the core modeling framework as well as specialized components that provide desired inductive bias for our goals. More detail, especially about model training, can be found in Appendix F for the interested reader. A.
Data representation
We observe N sequences of observed data from J interacting entities. In our setting, each sequence, indexed by i ∈ {1, ..., N }, represents a separate conversation between students on a problem-solving team. Each of the J students, indexed by j ∈ {1, .., J}, is an “entity” 1 . In our data, some of the N conversations involve different unique students. We represent each turn in the conversation as a discrete timestep, indexed t ∈ {0, .., T }. The data sequence i,j i,j i,j xi,j 0:T = [x0 , x1 , . . . , xT ] denotes the text observed from the j-th student in conversation i over time. The turnby-turn nature of conversations means that each xi,j t represents either a contiguous utterance of speech if j is talking, or complete silence if j does not talk at turn t. The length of each conversation may vary, but to keep notation simple we denote length T as constant across i. Each actual utterance from the audio recording of the conversation is transcribed to written text by human researchers. We set xi,j t to the D-dim. vector embedding of that text via the recent open-weight EmbeddingGemma encoder-only model [38]. The EmbeddingGemma encoder is selected due to its state-of-the-art performance2 , while using relatively few parameters (308M parameters) compared to those of decoder-only models (e.g. Mistral7B [24]: 7.3 billion parameters). A primary benefit of the
1 Note that the usage of the term “entity” here is distinct from
the usage of the term in Sec. II when discussing the framework for mechanistic reasoning evidence. 2 EmbeddingGemma was the highest ranked on the on the Massive Multilingual Text Embedding Benchmark (MMTEB) at the time of conducting this work.
EmbeddingGemma encoder is that it allows a relatively compact embeddings size due to Matryoshka representation learning [39]. We use D=128-dimensional embeddings. This reduced dimensionality helps keep training runtime efficient. If turn t lacks an utterance for speaker j, we set the observed vector xi,j t to the vector embedding of the word “silence” via EmbeddingGemma. At each turn t, there is always just one speaker with a nonsilence embedding, whereas the rest of the group has this silence embedding. B.
Probabilistic model
As done in [17], we model each sequence i as an i.i.d. draw from a switching-state space model with two levels of latent variables. First, a shared “team-level” discrete state sequence si0:T represents the time-evolving state of reasoning for the entire group of J students. We assume L possible team-level states. Second, an “entity-level” disi,j crete state sequence z0:T describes student-specific state over time. We assume K possible states at each entity, indexed k ∈ {1, 2, . . . K}. Our model defines a joint distribution over all ran1:N,1:J , x1:N,1:J , θ). Throughout, dom variables: p(s1:N 0:T , z0:T 0:T we denote essential model parameters shared by all sequences as θ. We assume this joint distribution factorizes as a product over i.i.d. sequences. Each sequence i’s variables are generated by moving forward in time. First, for the very first timestep t=0, we draw initial values: si0 ∼ CatL (θs−ini ) for j in 1, 2, . . . J :
(1)
z0i,j ∼ CatK (θz−ini ) xi,j 0 ∼ ND (µx−ini , Σx−ini ) Then, each subsequent time t ∈ 1, 2, . . . T depends only on variables from the previous or current time: sit ∼ CatL ( G(sit−1 , xi,1:J t−1 , θss ) )
(2)
for j in 1, 2, . . . J : i,j zti,j ∼ CatK ( F (sit , zt−1 , xi,j t−1 , θse ) ) i,j i,j xi,j t |zt =k ∼ ND (Ak xt−1 + bk , Qk )
Here, CatL denotes a Categorical distribution over L possible discrete values, parameterized by an L-dim. vector of probabilities. Similarly, CatK denotes a Categorical distribution over K possible discrete values, parameterized by a K-dim. vector of probabilities. ND denotes a multivariate Normal which takes a mean vector in RD and a D × D covariance matrix. The key parameters here for initialization ({θs−ini , θz−ini , µx−ini , Σx−ini }), latent state transitions ({θss , θse }) and data emissions ({Ak , bk , Qk }K k=1 ) are all contained in θ. We encourage readers to first understand the dependencies between the variables, as visualized in Fig. 2 for adjacent times t − 1, t. Later bold-headed paragraphs provide more detail about each sampling step, such as
6


sti
zti,4
i,4 i,1 i,2 i,3 zt+1 zt+1 zt+1 zt+1
xti,1 xti,2 xti,3
xti,4
i,1 i,2 i,3 xt+1 xt+1 xt+1
i,4 xt+1
0.25
0.21
0.25
0.22
0.27
0.28
0.25
0.21
0.75
0.71
0.79
0.75
0.72
0.73
0.74
0.74
…
…
…
…
…
Observations
zti,3
…
Latent Entity States
i st+1
…
Latent System States
…
the form of functions F and G that produce L-dim. and K-dim. vectors of probabilities. These individual sampling steps are obtained from further factorizing the joint distribution over the random variables in each sequence i via statistical assumptions that enable computational tractability, efficiency, and domain alignment. A mathematical formulation of how these assumptions are incorporated into the joint distribution is provided in Appendix D for interested readers. Here, we present the four key modeling assumptions more descriptively: (1) the Markovian assumption that all variables at time t depend only on variables from time t − 1, (2) the conditional independence of per-entity variables given the shared team state st , (3) the presence of data-to-latent feedback that informs both team and entity-level variables, and (4) parameters controlling transitions and emissions are shared by all entities, not learned to be entity-specific. The Markovian assumption (1) makes computation tractable, but also aligns with domain knowledge. Each turn in conversation will most strongly depend on the previous turn. For example, if a student spoke at turn t − 1, by our construction their observation at turn t is going to be silence. Our generative model can account for this. More importantly, Markovian time dynamics can model the latent trends in mechanistic reasoning we expect to see over time. Since reasoning mechanistically is typically a method to make sense of a particular phenomenon [40], if a student–or the group–is already engaged in this practice, they are likely to continue until ambiguity has been resolved. While this is not always true, we consider this a reasonable assumption. The feedback mechanism can account for moments when mechanistic-reasoning evidence abruptly disappears from the conversation. Assumption (2), the conditional independence of entity states given system states, is primarily for efficiency. Because entity observations and latent states only influence one another through the system latent variable, the cost of generating data from the model or evaluating the probability of data under the model remains linear in J as the number of entities grows. While our present analysis looks at small student teams of just 2-4 students, this model could scale easily to classroom-wide discussions of 50 or more students. We do acknowledge that using a single variable to channel the aggregate influence from all entities may be limited in flexibility compared to direct pairwise modeling of entity–entity interactions. Assumption (3), using feedback from previous data to influence the next system-level state and entity-level state, is crucial for modeling our chosen student conversation data. Our conception is that students have their own internal states that drive distinct utterances, but their individual states can be influenced by the rest of the group. Feedback mechanisms allow an individual’s utterance to change both their own trajectory and the group’s. Finally, Assumption 4 assumes all students are governed by the same transition probabilities and emission
0.33
0.48
0.44
0.33
0.35
0.48
0.42
0.30
zti,1 zti,2
Time t
Time t + 1
FIG. 2: A graphical illustration of the HSRDM introduced in Ref. [17], which we adapt for our task of locating evidence of mechanistic reasoning at the individual and student level. This representation visualizes our first three primary probabilistic modeling assumptions: (1) each observation and latent variable is Markovian (indicated by the black dashed lines), (2) the entity observations and latent states only interact via a shared system state (solid black lines), and (3) the system and latent entity variables are influenced by the observations at the previous timestep through the feedback mechanisms (dashed colored lines).
probabilities in the generative model, rather than specialized parameters for individual students. This is a necessity given the limited overall number of sequences available in use case and the need to generalize at “test time” to students we have never seen before. Individualized parameters are well-suited for a context where one desires to model the same entities over many sequences, and when there is a large amount of data available for each entity. The original HSRDM work [17] showed how modeling the same 5 basketball players over a season with player-specific parameters could be beneficial. We don’t yet have enough student-specific data to build studentspecific models; even in longer conversations, some students barely speak. System-level states. Each sit is a latent system state indicator for sequence i at time t. Given our overall modeling goals, we consider just L = 2 possible states, denoted as 0 and 1 . State 1 indicates the team is in a collective mechanistic reasoning state l = 1, while 0 suggests the opposite. In our model, the system state is designed to mediate how entities influence one another. For example, if the probability of system state 1 is high, the desired model behavior is that each entity has a higher probability of producing text that indicates mechanistic reasoning. Entity-level states. Each zti,j is a latent entity state indicator for student j and sequence i at time t. By design, we define K=4 possible entity-level states, denoted as S0, S1, T0, or T1. We intend states with prefix S correspond to the emission of silence, whereas states with prefix T correspond to that entity talking at time t.
7 States with suffix 0 indicate no evidence of mechanistic reasoning in the t − 1 utterance, while suffix 1 indicates present evidence in the t − 1 utterance. The exact latent state for a given observation may not always be clear. For example, if there is a speaking observation with evidence of mechanistic reasoning, but no evidence was present in the previous speaker, should the entity latent be the state T0 or T1? We prioritize estimating a probability distribution over these states rather forcing the model to choose just one state. Thus, the desired model behavior in this instance would be to place the highest probability on T1, and second-highest on T0, with zero probability on any S states. System-level starting distribution. We assume a simple uniform probability over the 2 system states, so θs−ini = [0.5, 0.5] in Eq. (1). System-level transition distributions. To define the transitions between system-level latent states in Eq. (2), the function G needs to take in the previous state sit−1 and recent data from all J entities xi,1:J t−1 , and produce a probability vector defining a valid distribution over the L=2 system states. i,1:J T G(sit−1 , xi,1:J t−1 , θss ) = sm(Π̃ esit−1 + Λg(xt−1 ))
(3)
Here, sm is the softmax function, esit−1 is a one-hot vector of length L, and Π̃ is a L × L matrix where Π̃l,m gives a log prior probability of transition from l to m. The feedback function g produces features useful for determining transition probabilities, and Λ is a coefficient matrix that maps the features from g to a vector of size L. This feedback mechanism can aggregate from all J entities. The definition of g is one of our deliberate inductive bias components that will be discussed in detail in Sec. III D, alongside a fixed setting for Λ. Only Π̃ is a learnable parameter in θss . Entity-level starting distribution. We assume a uniform distribution over all K entity states at time t = 0, so θz−ini = [ 14 , 14 , 14 , 14 ] in Eq. (1). Entity-level transition distribution. The function F in Eq. (2) produces K-dim. probabilities defining the chance of the next entity-level state for student j. We construct F in a similar manner as G above, via the definition: i,j i,j T F (sit , zt−1 , xi,j t−1 , θes ) = sm(P̃si ez i,j + Ψsit f (xt−1 )) (4) t
t−1
Here, a key difference between F and the system function G is that only data feedback from the focal entity j is needed for F . No other entities are directly involved, as system state mediates all entity-to-entity interaction. The K ×K matrix P̃sit defines log transition probabilities specific to the current system state, while ek here denotes a K-dim. one-hot vector. The feedback feature function f and its coefficient matrix Ψsit are deliberately fixed as a way to control inductive bias. Details will be provided
in Sec. III D. Overall, the only learnable parameters for F are θse = {P̃ℓ }L ℓ=1 . Emission at t=0. For our initial observations xi,j 0 , we model the emission distributions for when k is T0 or T1 as multi-variate gaussian distributions with zero mean vectors and an identity covariance. Thus, µx−ini = 0D and Σx−ini = ID in Eq. (2). We model the emission distributions for when k is S0 or S1 as multi-variate gaussian distributions with fixed parameters: Ak = 0, Qk = 0.001ID , and bk to the embedding of “silence”. To further ensure a good starting point for each sequence, we penalize the density emissions of all non-k=0 states for the nonspeakers. As in, all non-speakers occupy the k=0 state given that there is no previous observation for evidence of mechanistic reasoning to occur. Emission for t≥1. As defined in Eq. (2), to generate each xi,j the current entity state indicator zti,j ∈ t {S0, S1, T0, T1} defines a state-specific autoregressive Gaussian emission model. The state ID as an integer k provides the state-specific coefficient matrix Ak ∈ RD , mean offset bk ∈ RD , and D × D covariance matrix Qk . For our task, we intend for state IDs with prefix S to generate only silent (non-talking) utterances. Text transcripts contain no interesting time-varying data for such turns t. Thus, instead of learning these parameters, we fix Ak = 0, fix Qk = 0.001ID , and set bk to the embedding of “silence” for k ∈ {S0, S1}. This leaves only the Ak , bk , Qk for talking states as learnable parameters in θee . We use Gaussian distributions for the emissions because they accommodate continuous data and have proven empirically versatile. Exploring alternative distributions better suited to modeling language embeddings, however, remains an important direction for future work. Priors on Parameters. Whereas the previous work [17] incorporated a sticky Dirichlet prior over the system transition parameters to make remaining in the same latent state more likely than transitioning, we do not include this bias in our work. Given that we lack knowledge around how many turns of talk we expect prior to a change of state, we use uninformative priors on all transition parameters – both at the system and entity level. We also use uninformative priors over the multi-variate gaussian parameters that govern student utterances. We highlight that incorporating informed priors to enhance the inductive bias of the modeling is an important part of future work. C.
Training procedure
To train our adapted HSRDM given a dataset of N sequences, we follow the fully unsupervised procedure introduced by Wojnowicz et al. [17]. Their algorithm simultaneously produces a maximum a-posteriori point estimate of the parameter vector θ and an estimated postei,1:J N rior distribution over all state sequences {si0:T , z0:T }i=1 . Practically, using these results can query the probability that sit = 1 at any time t, or the probability that
8 zti,j = T0 or zti,j = S1 at any time t. We should understand these as posterior probabilities, because they condition on the entire data sequence xi,1:J 0:T . The underlying algorithm uses Bayesian variational methods [41, 42] for posterior estimation, with structured approximations rather than naive “mean-field” independence assumptions. Further details for this algorithm are provided in App. F 1. The learned posterior distributions over the entity and system-level latent variables are our primary quantities of interest for identifying students’ mechanistic reasoning over time in a conversation. In the next section, we introduce the specialized inductive bias we use to steer these posterior probabilities based on evidence in the conversation. D.
Specialized inductive bias via data feedback
Our probabilistic model in Eq. (2) uses previous data to inform next state probabilities for both system and entity-level states, via the function g(xi,1:J t−1 ) in Eq. (3) and function f (xi,j ) in Eq. (4). Wojnowicz et al. [17] t−1 introduced such feedback functions in a generic fashion. In our work, we develop concrete custom functions as a primary inductive bias to guide our model behavior. Specifically, we train a classifier that takes EmbeddingGemma encoded utterance vectors xi ∈ R128 as input and predicts the maximum mechanistic-evidence class yi ∈ {0, 1, 2, ...7} under Russ’s taxonomy (Sec. II). The zeroth class is for no-evidence, whereas classes 1 − 7 quantify how much mechanistic reasoning evidence is present in the utterance. Denote the ultimate classifier as a function c that, given an xi,j t , produces a 8-dim. vector of probabilities for the 8 classes in Russ’ framework. Training labels for the classifier come from a subset of our human expert annotations of the data. We highlight that these annotations are utterance independent – meaning that each utterance is annotated as a standalone segment of text, rather than incorporating inferred meaning from the surrounding utterances. Further details on the human annotation analysis and evidence for code reproducibility can be found in Appendix B. Our use of “ground-truth” labels from a subset of data to inform the feedback mechanism makes our adapted-HSRDM a semi-supervised method, rather than fully-unsupervised. We are careful to make sure the data used to train this classifier is a subset of data used to train the adaptedHSDRM overall, and has no overlap with later “test” data used to evaluate ability to generalize. Classifier model. For the classifier c, we utilize a 2layer feed-forward neural network (NN) with a ReLu activation function and a hidden dimension size equal to the size of our text embeddings (D = 128). We made this choice after evaluating a range of pre-trained and train-from-scratch options. This choice reflects a favorable trade-off between task alignment and deployment cost (e.g. parameter storage for inference-time compute). We detail this rationale in the Appendix E 1.
Classifier training. We fit this classifier to all embedding-label pairs from N = 4 sequences of data, a subset of our training set. To avoid overfitting, we minimize a loss that includes a prediction-quality term (cross entropy) and an L2 penalty on the weights of the NN. This penalty is often called “ weight decay” in deep learning. Standard practice to select the strength hyperparameter for the weight decay requires a grid search, where at each candidate value we fit a model on one subset of the total dataset and then selected based on performance on a validation set–a separate subset of the total data. In our setting, we cannot afford a separate validation set. We have limited data and would like to reserve as much as possible for the overall HSRDM training; especially because we also need a separate held-out test for assessing generalization performance. Training the feedback classifier on the same exact dataset as the overall HSRDM training runs the risk of overfitting the HSRDM. To avoid needing a validation set for our classifier, we utilize a recent Bayesian model selection strategy from Harvey et al. [43]. This paper suggests a way to use the marginal likelihood of the training set, which has a well-known ability to avoid over-fitting [44], via a practical approximation in the case of deep neural networks where the marginal likelihood itself is not tractable. We provide technical details of the classifier training, including our training data selection, implementation decisions, and results in Appendix E. System-level feedback g. For the system recurrence i,j ∗ function, we set g(xi,1:J ) = OH(c(xt t )), where jt∗ indit cates the speaking entity at time t and OH produces a one-hot vector indicating the maximum value of its input vector. As other entities remain silent by construction, the speaker is the only entity whose speech can have evidence of mechanistic reasoning. We then set the coefficient matrix Λ, which is multiplied by the 8-dim. vector from g in Eq. (3), to be: 2 0 0 0 0 0 0 0 Λ = , 0 2 3 4 5 6 7 8 where the ordering of the rows corresponds to system state 0 and then 1 . Using these coefficients, as the classifier c predicts an increasing amount of mechanistic reasoning is present (mass moving rightwards), the probability of transitioning to 1 becomes higher. Entity-specific feedback f . The recurrence function f for entity j takes only the one-hot prediction from i,j that entity as input: f (xi,j t ) = OH(c(xt )). We utilize one-hot predictions rather than predictive probabilities from the classifier to avoid the extra step of analyzing probability-prediction calibration quality. Using the onehot predictions is a common practice in machine learning; however we acknowledge that more investigation could be done to understand which output is more suitable given the context.
9 We then set the system-state-specific coefficient matrix Ψ from Eq. (4) to one of two values: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 3 4 5 6 7 8 9 Ψ 0 = 1 , Ψ 1 = 0 0 0 0 0 0 0 0 . 0 0 0 0 0 0 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Here, the ordering of rows corresponds to entity states S0, S1, T0, T1. When sit = 0 , the left matrix above will increase the probability of the non-reasoning entity states, S0, T0, given that this matrix interacts with only one-hot vectors that have a 1 in the zeroth index. The integer shift is stronger for the S0 state than the T0, as only one student is able to transition to talk independent of the number of students in the group. These integer values are found empirically from implementing the model on the training data containing students of group size 4 and human-optimizing for the combination of integer values that provide quality training performance. We provide the model results on training data in Appendix G. When sit = 1 , interactions between the speaker onehot vector from the classifier and the right matrix above instead increase the probabilities of S1, with the strength of this increase determined by how much evidence is expressed in the utterance. Since the one-hot vectors for all silent observations have a 1 in the zeroth index, there is a 2 in the zeroth index for the k ∈ {S1, T1} states, so that silent observations also shift probability mass toward these mechanistic reasoning states, but more weakly than the speaker does. Together, the supervised classifier underlying the feedback functions f, g and the fixed settings of coefficients above implement a feedback mechanism that instills a desired inductive bias, pushing the model toward higher probabilities of mechanistic reasoning states when previous student utterances contain relevant evidence. E.
Informed initialization procedure
The training optimization problem for the HSRDM is non-convex. Thus, smart parameter initialization can substantially improve the quality of solutions delivered by our algorithm. Here we describe a way to initialize the parameters of the emission model, so that it can generate embedding vectors containing evidence of mechanistic reasoning when k = T1, and embedding vectors without evidence of such reasoning when k = T0. Recall these are the only states with learnable parameters in θ; Sec. III B describes how we fix Ak , bk , Qk for k ∈ S0, S1. Concretely, we use true reasoning labels y provided by our human annotators to aid initialization. We utilize the same labels as those used to train the classifier discussed in Sec.III D. From this data, we create i,j two subsets: S1 contains adjacent-time pairs xi,j t−1 , xt with reasoning present in yt−1 and j talking at t, and S0 contains pairs with reasoning absent yet j is talking.P We then estimate Ak , bk by minimizing squared error t∈Sk (xt − Ak xt−1 − bk )2 . The initial covariance Qk
is computed from the residuals, enforcing a variance floor of 0.0001 to prevent too small values along the diagonal. This mechanism, when combined with the recurrent feedback, enables the model to balance desired transition and emission probabilities. When the previous utterance contains mechanistic evidence, the feedback mechanism biases the next speaker toward the T1 state. If that speaker’s utterance subsequently contains no mechanistic evidence, the Gaussian emission instead pulls the latent state toward T0. The intent is for these competing influences to balance probability mass between the two states, so that neither state becomes overwhelmingly likely.
IV.
RQ2: How Inductive Biases Impact Generalization
In this section, we introduce our chosen metrics for model evaluation, and investigate if and how much the inductive bias via the data feedback (introduced in Sec.III D) impacts the performance of the adaptedHSRDM on unseen data. Our aim is to obtain evidence (or lack of) that the adapted-HSRDM is interpretable by design for the task of identifying evidence of student’s mechanistic reasoning in time-evolving conversational data. Given the relatively small amount of data we have, we stress this investigation is preliminary and exploratory. A.
Metrics and hypotheses
We introduce task-specific metrics tailored to desired model behavior in this setting. By no means do these metrics capture all desired model behavior; however, they offer a reasonable starting point for the primary behaviors we both desire and expect to observe empirically based on the theoretical inductive bias baked into the model. Thus, we frame our investigation around hypotheses of the following form: when the inductive bias is present, metric X should be Y relative to when it is absent or weaker. Positive support for these hypotheses is evidence that our adapted-HSRDM is inherently interpretable for the target task. Below, we define each hypothesis and provide the rationale for its inclusion in our exploratory evaluation. H(i): The correlation between a talker doing MR and being in S1 at the next turn should be stronger. We compute the Pearson correlation between the hand-coded mechanistic-reasoning strength of an entity’s observation at time t and the model’s posterior probability that the same entity occupies latent state S1 at time t+1. Pearson’s r captures the direction and magnitude of a linear association. We focus on the t → t+1 relationship because, in our data, an entity who speaks at time t always transitions to silence at the next time step. Thus, we evaluate correlations specifically for the subsequent silent state after a speaking turn. A large positive correlation in this setting signals that higher mechanistic evidence strength is associated with greater posterior
10 mass on S1 at the next turn. Thus, a stronger correlation when the inductive bias is present supports our hypothesis that the the mechanism is operating as intended.
main text as our test data only contains J=2 students (i.e no students are transitioning from silent to silent).
H(ii): The mean probability gap in being in S1 at the next turn between talkers who reason and those who do not should be larger. Across all time steps t, we examine the human annotation yt and assign to one of two sets: the times TY with present evidence of mechanistic reasoning, and the times with no evidence TN . Denote the talking entity at each time t as jt∗ ; we know that they will move to a silent state at time t + 1. Concretely, we’d like to compare the mean posterior probability of the target state across the two sets:
To explore the hypotheses put forth in Sec. IV A, we run variants of the model that either contain, weaken, or completely remove the inductive bias introduced in Sec. III D. To weaken or remove the bias, we ablate targeted components of the model while keeping the others intact. We describe these model variants below:
1 X 1 X i,j ∗ i,j ∗ q(zt+1t = S1) ≫ q(zt+1t = S1) |TY | |TN | t∈TY
(5)
t∈TN
This metric offers more context to the one above, as we compare how likely the speaker is to occupy the mechanistic reasoning state after their utterance contained evidence vs. when their previous utterance did not. We report standard deviation as well as mean to help readers understand variation. A larger mean probability gap when the inductive bias is present‘ supports our hypothesis that the the mechanism is operating as intended. H(iii): The mean probability gap in being in T1 at the next turn between non-talkers who hear MR and non-talkers who do not should be larger. Using the same TY and TN defined above, we now focus on those who did not talk at t but may have heard MR ∗ at that time. Denote jt+1 as the speaker at t + 1, who we know was silent at time t. Concretely, we’d like to compare the following: 1 X 1 X i,j ∗ i,j ∗ q(zt+1t+1 = T1) ≫ q(zt+1t+1 = T1) |TY | |TN | t∈TY
t∈TN
This metric informs how well mechanistic reasoning in the recent past influences the group member who talks next. Given that this metric provides information about a non-speaker at time t, we do not expect their increase in posterior mass at the next timestep to be as large as the speaker’s increase, so the delta from TY to TN will be smaller than H(ii). We report standard deviation as well as mean to help readers understand variation. A larger mean probability gap when the inductive bias is present‘ supports our hypothesis that the the mechanism is operating as intended. It is also important to separately measure how well mechanistic reasoning in the recent past influences the group members who remain silent. By tweaking the construction above, we compute the mean posterior probabilities of all entities who transition from silent to silent at all time steps in each of the two sets TY , TN . A larger likelihood gap when the inductive bias is present‘ supports our hypothesis that the the mechanism is operating as intended. We report this metric for our training data in the Appendix, but do not use it for the results in the
B.
Inductive bias ablations
Human feedback. As a “gold standard”, we consider replacing the classifier c with the true labels provided by humans for the 8 levels of MR evidence at each time t. This helps assess a ceiling for performance of the feedback mechanism. Naturally, such labels are rarely readily available at test time due to the expensive annotation effort required. Classifier feedback. We use the classifier c as the feedback mechanism described in Sec. III D. Thus, this model variant is the adapted-HSRDM with the specialized inductive bias for our task and serves as the out-of-the-box method proposed in this work. Weak classifier feedback (no system state). We consider a version of the model that removes the toplevel system state variables entirely of the adaptedHSRDM. Removing this while keeping the classifier feedback largely weakens the inductive bias introduced in Sec. III D as only the entity level feedback is utilized and the entity feedback matrix remains stationary as Ψ 0 . Additionally, ablating the system state provides evidence that the hierarchical group coordination structure in the model is inherently useful for the task. No feedback. We consider a version of the model that removed the feedback mechanism completely – removing the f term in Eq. (3) and the f term in Eq. (4). Thus, this model represents the complete removal of the inductive bias introduced in Sec. III D. C.
Experimental setup details
We divide our total dataset of thermo-fluids student conversations into a training set of N =8 transcripts and a test set of N =2 transcripts. Details about training and test data selection are provided in Appendix E 2. Each training sequence has J=4 students on a team. A subset of N =4 sequences from the training set was used to develop the classifier feedback mechanism. The test set contains two transcripts from previously unseen students discussing two different thermo-fluids problems. One of these problems also appears in the training data; we refer to it as the seen problem and to the other as the unseen problem. Importantly, the test transcripts are different from that of the training in that they only contain two students in the group rather than four. In a less data scarce setting, we would evaluate model performance on groups matching the number of students seen during training, as well as on smaller and larger groups.
11 Seen Problem S0 S1 T0
New Problem S1 T0
Feedback
Entity
Human
0 1
-0.069 0.548 -0.099 -0.062 -0.086 0.869 -0.247 -0.087 -0.100 0.780 -0.166 -0.087 -0.090 0.894 -0.283 -0.110
Classifier
0 1
-0.087 0.257 -0.097 -0.065 -0.019 0.484 -0.244 -0.089 -0.137 0.436 -0.160 -0.091 -0.005 0.467 -0.276 -0.120
No System
0 1
0.007 0.147 -0.100 -0.061 0.097 0.298 -0.255 -0.076 -0.004 0.233 -0.173 -0.078 0.058 0.349 -0.293 -0.100
None
0 1
0.140 0.135 -0.097 -0.064 0.288 0.286 -0.246 -0.088 0.221 0.212 -0.160 -0.091 0.340 0.334 -0.279 -0.117
T1
S0
T1
TABLE I: H(i): Pearson r correlation experiments on the seen (left) and unseen (right) test problems. For each model ablation, we examine turns when an entity speaks, and assess correlation between the posterior probability mass of the next state (columns) and the human-annotated evidence strength present in the utterance. For good models, the desired behavior is a strong positive correlation for state S1 (bold). We observe the correlation decrease as we ablate inductive bias components in the model architecture.
Focusing evaluation on J=2 student transcripts is a limitation of this study, which we discuss further in Sec. VI. To train the model parameters, we utilized closedform updates for the gaussian emission parameters and gradient-ascent on the ELBO objective for all others. We adopted the same low-level configuration settings as Ref. [17]: after initialization, each M-step used 50 gradient-ascent iterations; the initialization procedure consisted of a single E-step and M-step with 5 gradientascent iterations in the M-step. All gradient-ascent was computed with an Adam optimizer with a learning rate 0.01. We trained the model with 15 total CAVI iterations. We present the training ELBO plot for our best model in Appendix G. Rather than using the ELBO as the main guidepost to end model training, we attended to the correlation between the system level state occupying 1 and human-annotated evidence of mechanistic reasoning being present in the previous time step. We observed empirically that the correlation stabilized to two decimal places by iteration 15. Given the metrics we are interested in (see Sec. IV A), we believed this to be a more appropriate metric than monitoring the ELBO to determine convergence. We thus used the human annotations in the evaluation of the optimization procedure for all trained model feedback ablations. Each set of training parameters per ablation is used for that same ablation during test. We include all trained model parameters in our codebase [45] and the training results can be found in Appendix G. Additionally, the appendix includes training results for some model variants initialized with an uninformed, label-free k-means procedure rather than the informed procedure described in Sec. III E. These results stem from related experimentation on the initialization procedure and are deferred to the appendix because they are not central to our main research questions. No parameters in the adapted-HSRDM are randomly initialized and no randomness is introduced in the
gradient-ascent algorithm (each step uses all data, not a random minibatch). Thus, our training stability is not sensitive to any random seeds. As described in Sec. III, the adapted-HSRDM is completely deterministic given a fixed training set. D.
H(i) Results: Correlation
In Table I, we report metric results for all inductive bias ablations. Focusing on the seen problem setting, the column most relevant is S1, representing the correlation between the human-annotated evidence of mechanistic reasoning and the posterior probability mass of the S1 occupancy. We observe the highest average correlation values across both entities when there is human feedback. When there is classifier evidence, we observe the correlation decrease as expected, but still remain higher than when there is weak feedback and no feedback. Relative to no feedback, classifier feedback produces about a 2× increase in correlation strength. In non-S1 columns for both human and classifier evidence feedback, we observe small evidence of an anti-correlation, which is evidence of the feedback working as expected. Whereas, for no feedback we observe small evidence of a correlation in the S0 column. This is evidence that the inductive bias removes unwanted correlations. Overall, there is strong evidence to support the hypothesis that the classifier feedback is a useful inductive bias. In Table I (right), we report metric results for all ablations in the unseen-problem setting. As before, the column most relevant is S1, representing the correlation between the human-annotated evidence of mechanistic reasoning and the posterior probability mass of the S1 occupancy. We observe consistently higher correlation values across all the ablations relative to the seen-problem setting. The highest average correlation values across both entities are observed when there is human feedback. When classifier evidence is used, the model performs worse as expected. However, when the classifier evidence or the system state is removed, the correlation
12
Feedback
Seen Problem New Problem Evidence Mean Proba. Std. Dev. Mean Proba. Std. Dev.
Human
Yes No
0.791 0.151
0.105 0.167
0.649 0.096
0.270 0.124
Classifier
Yes No
0.973 0.543
0.017 0.292
0.769 0.476
0.230 0.280
No System
Yes No
0.964 0.926
0.011 0.052
0.963 0.937
0.047 0.039
None
Yes No
0.495 0.500
0.004 0.004
0.501 0.502
0.004 0.004
TABLE II: H(ii): Mean probability of entity state S1, for turns with and without evidence of MR, on the seen (center) and unseen (right) problems. For all model ablations, we report the mean and standard deviation of the previous speaker’s posterior probability mass assigned to state S1, conditioned on whether their utterance contains evidence of mechanistic reasoning. We expect for this state to have a larger probability when evidence is present than not, and we highlight in bold cases where the delta is at least 0.25.
Feedback
Seen Problem New Problem Evidence Mean Proba. Std. Dev. Mean Proba. Std. Dev.
Human
Yes No
0.143 0.309
0.350 0.450
0.199 0.164
0.373 0.360
Classifier
Yes No
0.143 0.335
0.350 0.461
0.230 0.179
0.403 0.371
No System
Yes No
0.143 0.282
0.350 0.442
0.148 0.134
0.342 0.326
None
Yes No
0.143 0.334
0.350 0.464
0.218 0.171
0.391 0.363
TABLE III: H(iii): Mean probability of entity state T1, for turns with and without evidence of MR, on the seen (center) and unseen (right) problems. For all model ablations, we report the mean and standard deviation of the current speaker’s posterior probability mass assigned to state T1, conditioned on whether the previous speaker’s utterance contains evidence of mechanistic reasoning. We expect for this state to have noticeably higher probability when evidence is present than not, and this, we highlight in bold cases where the delta is at least 0.05.
across both entities decreases further. Compared to the no-feedback setting, incorporating classifier feedback produces an approximately 1.5× stronger correlation. This supports the hypothesis that the classifier feedback induces a useful inductive bias. In the T0 and T1 columns, we observe consistent negative correlations. E.
H(ii) Results: MR Effect on Talker Next State
In Table II, we report metric results for all ablations on the seen problem. We observe the largest increase in the S1 state occupancy posterior mean – conditioned on mechanistic reasoning evidence – with the human feedback mechanism. As expected, the classifier feedback performs worse than the human feedback, but greatly outperforms removing the system state and the feedback altogether. When we ablate the system state from the model or take away the evidence feedback mechanism completely, we see there there is essentially no difference in the S1 state occupancy posterior mean. Compared to the no-feedback setting, incorporating classifier feedback produces an approximately 86× larger mean probability
gap. This is strong evidence to support the hypothesis that the classifier feedback induces a useful inductive bias. In Table II (right columns), we focus on the unseen problem. As before, we observe the largest increase in the S1 state occupancy posterior mean with the human feedback mechanism. Classifier evidence performs worse human evidence and performance decreases further when the system state is ablated or the evidence feedback mechanism is removed. Relative to when no feedback is present, incorporating classifier feedback produces an approximately 313× larger mean probability gap. These results are consistent with the hypothesis that classifier feedback induces a useful inductive bias. F.
H(iii) Results: MR Effect on Non-talker Next State
In Table III, we report metric results for all ablations for the seen problem. We observe all results run counter to our expectations: the average posterior mass on state T1 is higher when no mechanistic reasoning evidence is
13 present at the previous time step than when evidence is present. One possible reason for this poor generalization is that there are only two students in the group rather than four in our training data. However, we believe a stronger hypothesis is that in this particular transcript, there are very few examples of mechanistic reasoning. Given the small sample size (seven instances in this transcript), learning how the current speaker’s evidence of mechanistic reasoning should affect the next speaker’s posterior may be difficult, and likely harder than the previous two metrics. We include the number of positive label counts by problem in Appendix Table XVII. In Table III’s right columns, we report metric results for all ablations on the seen problem. For this metric, we do not expect that there be such large differences in the posterior means based on the conditional; rather, we expect that there be noticeable differences, which we do see here. Surprisingly, we observe the largest increase in the T1 state occupancy posterior mean with the classifier feedback mechanism. When we ablate the system state, performance drops significantly. It makes sense that ablating the system state would cause the most disturbance given that the metric validates the influence of an individual on the entire group. However, no evidence performs similarly to classifier evidence, suggesting that the feedback mechanism does not have a large affect on the T1 state occupancy posterior mean. The performance on the unseen-problem setting is better relative to the performance on the seen-problem setting which we hypothesize is because the unseen problem contains 75 instances of mechanistic reasoning, compared to only 7 instances in the seen problem. We include the number of positive label counts by problem in Appendix Table XVII. In summary, we have little evidence from this metric to conclude that the classifier feedback mechanism offers a useful inductive bias. However, we do have evidence to support that the system state is an integral inductive bias component.
V.
Tool demonstration and user recommendations
In this manuscript, we explore an inherently interpretable ML tool to automate the identification of students’ mechanistic reasoning in time-series data. In Fig. 3, we show the adapted-HSRDM’s output for a segment of the unseen test problem transcript. We took a moving average of human labels of mechanistic reasoning for 20 timesteps, found the max, and then selected those 20 timesteps as our region. The first row displays the posterior probabilities that each student is engaging in mechanistic reasoning throughout the segment of conversation. These probabilities are direct outputs from the adapted-HSRDM with classifier feedback. The second and third rows show the classifier feedback predictions and the human feedback annotations, respectively at aligned timesteps in the conversation. Our main motivation for including Fig. 3 is to demonstrate a visual
output of a high density region of mechanistic reasoning that a user can generate with our tool. Fig. 3 is not meant to represent a ground truth comparison of the posterior probabilities to the labels from the two feedback mechanisms. Recall that both the human and classifier feedback are labels of independent utterances that do not include information from surrounding utterances. Nonetheless, as a compliment to the evidence shown in Table I that our model outputs desired behavior, we see visually that the strength of the human feedback and the posterior probabilities of the speaker at the next timestep are strongly positively correlated. Additionally, the human feedback is a ground truth for the classifier feedback, and we can compare these predictions directly. We observe that classifier mistakes clearly influence the adapted-HSRDM’s behavior. For example, in step 89, where the classifier predicts no evidence of mechanistic reasoning and the posterior probabilities remain low. Overall, although the human and classifier annotations exhibit substantial variance in their exact numerical agreement, they align well on the binary distinction of whether evidence of mechanistic reasoning is present. The adapted-HSRDM with instructions for use can be found in the public Github repository [45], under the project name “Mech4Mech”. We provide clear guidelines for users to run the pre-trained classifier feedback model on their own data. We recommend using the classifier variant for new data without human feedback annotations, since it is most appropriate to deploy a model under the same information conditions available at test time as during training. That said, we provide the parameters for all model variants in the codebase, so users can easily switch to the human-feedback variant if they wish to test their data with it. Users can choose to generate a posterior probability plot over a specified time window for the region exhibiting the highest density of mechanistic reasoning (e.g., similar to the top row of Fig. 3). The highest density region is computed by taking a moving average across the system state probabilities for sit = 1 for a specified user time window. Users can also choose to output a csv file of the per-student max posterior probability with the associated state over time. To enable others’ explorations, we provide open-access embeddings of the raw student text 3 and an open version of the code. When using this model on new student transcripts, we ask the user to be cautious of the following contextual variables: (1) varying group sizes, (2) non open-ended problems, (3) what counts as evidence of mechanistic reasoning, and (4) small amounts of evidence present in the data. As for (1), our training set contained transcripts of
3 The raw student data used in this project is not able to made
publicly available as dictated by the original IRB documentation and authorization.
14
“So I guess we're gonna - do we make two separate systems for air and water or?”
“I think maybe?”
“We could assume that inside the pipe like the air and water are mixed and it's. Oh wait.”
FIG. 3: The tool’s output for a segment of the unseen test problem transcript. Top: the posterior probabilities that each student is engaging in mechanistic reasoning throughout the segment of conversation. These probabilities are direct outputs from the adapted-HSRDM with classifier feedback – selected as the region of 20 time steps that had the highest average human feedback. Middle: the classifier feedback predictions at aligned timesteps in the conversation. Bottom: the human feedback annotations at aligned timesteps in the conversation. Our main motivation for this figure is to demonstrate a visual output that a user can generate with our tool. We also observe visually that the human feedback strength and the posterior probabilities at the next timestep for a speaker are strongly positively correlated (as evidenced in our numerical results). We also observe that classifier mistakes clearly influence the adapted-HSRDM’s behavior. For example, in step 89, where the classifier predicts no evidence of mechanistic reasoning and the posterior probabilities remain low.
student groups of four and our test set contained transcripts of student groups of two. While the generalization results support that the model can generalize to student groups of new sizes, we do not have enough evidence to conclude that we would see good performance on groups of other sizes, or much larger sizes. With respect to (2), all of our transcripts contained student discourse about open-ended thermo-fluids problems, which biases our model’s capability towards dialogue that would be generated from such prompts. We caution a user against applying this tool on problems that would overly constrain a student’s thinking towards a specific answer. Although we have not validated the model beyond thermofluids contexts, the diversity of thermo-fluids problems used in training and evaluation gives us reasonable confidence that it may transfer to discussions of other openended science and engineering problems. Of course, we are most confident that the model can generalize to new student transcripts of the same problems from the training set. Given that questions which are too constraining may be less likely to lead to evidence of mechanistic reasoning, this point also aligns with (3). The generalization
results on a transcript containing very little evidence of mechanistic reasoning show that it is important for a user to consider whether their data might have enough evidence of mechanistic reasoning for the tool to be used properly. Without looking at the data, this judgment is best informed by the user’s familiarity with the problem prompts and the data-collection procedure. As a quick sanity check, we recommend for the user to annotate a small sequence of data and evaluate the model according to the metrics used in this work. Lastly, it is important for any user to determine whether their model of what counts as evidence of mechanistic reasoning aligns with that of the one used in this work. We encourage a user to review our process for human-annotation in the Appendix B and decide whether the labels taken as “ground truth” in this work align with their description of mechanistic reasoning evidence.
15 VI.
Research opportunities and limitations
We outline several future directions for ML experts aiming to build on this tool for STEM education research, especially through addressing its current limitations. An initial direction is to further characterize the variational posteriors at different time points, and how they change with respect to the inductive bias being present. For example, we did not investigate if and when the posteriors exhibit very extreme probability values (i.e. very close to zero or one) and how this might impact recommendations for tool usage. Given that our test experiments were conducted on only two transcripts, an important next step is to evaluate the model on a broader range of data that varies in problem type, student group size, and student populations. This would further inform probabilistic priors that we can introduce as additional inductive biases for the task. At present, one might reasonably ask why the classifier itself should not simply be used as a proxy method to identify text regions with the highest density of mechanistic reasoning. This question can be pursued systematically if provided ground truth human annotations that capture context, rather than assigning labels only at the level of acts of mechanistic reasoning in individual utterances. Generating context-sensitive human annotations poses a greater challenge when coding transcript data line by line using Russ’ framework. If obtained, these annotations would enable systematic comparisons between our model’s learned latent states, independent classifier predictions, and uninterpretable in-context learning outputs. Doing this human coding may also reveal new human-decision making elements that make useful inductive biases to incorporate into the modeling. As an alternative approach to more human annotation, all three methods could be used to identify regions of highest mechanistic reasoning density, which education researchers could then qualitatively evaluate for their usefulness. This would answer the question of which tool effectively sends education researchers to the most relevant transcript segments. Independent of the approach taken, we believe systematic comparisons are an important next step for understanding trade-offs among interpretability, usability, sustainability, and performance. As a separate direction, we encourage others to expand upon the adapted-HSRDM to introduce personal-
[1] D. Vinck and É. Blanco, Everyday Engineering: An Ethnography of Design and Innovation (The MIT Press, Cambridge, MA, 2003). [2] D. A. Dickerson, S. Masta, M. W. Ohland, and A. L. Pawley, Is carla grumpy? analysis of peer evaluations to explore microaggressions and other marginalizing behaviors in engineering student teams, Journal of Engineering Education 113, 603 (2024).
ized features for individual entities, inductive biases that allow for shuffled student groups, and multi-modal observations to include data beyond language. Especially with respect to the latter, we can include more informed evidence of mechanistic reasoning (e.g. student drawings, hand-gestures) that will strengthen the tool’s utility. We encourage the construction of similar models for identifying other disciplinary practices, behaviors, and emotional affects of students in conversation – such as engaging in confusion [46], analogical reasoning [47], and meta-cognition. Upon reflection, there are a set of smaller improvements that could be made to the adapted-HSRDM to enhance performance. For one, we could take advantage of the talk-silent knowledge in the data and add prior distributions to the system and entity transition probabilities reflecting this knowledge. Furthermore, we could generate a more mathematically principled silent vector than simply embedding the word “silence”. For example, we could generate an embedding that is maximally orthogonal to the mean of the talk embeddings. We hope that this work inspires an interpretable ML approach to the design and investigation of tools for STEM education research. While we acknowledge that interpretable methods do not always offer the largest short-term efficiency win, we believe that tools that can be reasoned about mechanistically provide longer-term advantages for both designers and users. Understanding the underlying mechanisms offers designers more principled control and evidence that can support or nullify a hypothesis, which in turn, offers users greater confidence in the tool and clearer guidance of when it should be applied. We emphasize that tools of this kind are especially valuable for fostering convergence between tool designers and end users, and we encourage deeper collaboration at the intersection of ML and STEM education. Acknowledgments
We are grateful for financial support from the U.S. National Science Foundation via awards GCR #2428640 and CAREER #2338962. We wish to acknowledge other members of the Hughes lab research group, namely Ethan Harvey for help training the feedback classifier with a data-emphasized variational ELBO approach. We also thank Erica Kemmerling, Katie Melsky, and Isabella Stuopis for their work creating the original data set.
[3] R. E. Scherr and D. Hammer, Student behavior and epistemological framing: Examples from collaborative activelearning activities in physics, Cognition and Instruction 27, 147 (2009). [4] M. D. Koretsky, E. J. Nefcy, S. B. Nolen, and A. B. Champagne, Connected epistemic practices in laboratory-based engineering design projects for largecourse instruction, Science Education 107, 510 (2023).
16 [5] P. Machamer, L. Darden, and C. F. Craver, Thinking about mechanisms, Philosophy of Science 67, 1 (2000). [6] D. van Eck, Mechanisms and engineering science, in The Routledge Handbook of Mechanisms and Mechanical Philosophy, edited by S. Glennan and P. Illari (Routledge, 2018) pp. 447–461. [7] V. de Andrade, Y. Shwartz, S. Freire, and M. Baptista, Students’ mechanistic reasoning in practice: Enabling functions of drawing, gestures and talk, Science Education 106, 199 (2022). [8] R. S. Russ, R. E. Scherr, D. Hammer, and J. Mikeska, Recognizing mechanistic reasoning in student scientific inquiry: A framework for discourse analysis developed from philosophy of science, Science Education 92, 499 (2008). [9] T. O. B. Odden and R. S. Russ, Sensemaking epistemic game: A model of student sensemaking processes in introductory physics, Phys. Rev. Phys. Educ. Res. 14, 020122 (2018). [10] Y. Long, H. Luo, and Y. Zhang, Evaluating large language models in analysing classroom dialogue., npj Sci. Learn. (2024). [11] Z. He, S. Naphade, and T.-H. K. Huang, Prompting in the dark: Assessing human performance in prompt engineering for data labeling when gold labels are absent, in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25 (Association for Computing Machinery, New York, NY, USA, 2025). [12] H. Subramonyam, R. Pea, C. Pondoc, M. Agrawala, and C. Seifert, Bridging the gulf of envisioning: Cognitive challenges in prompt based interactions with llms, in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24 (Association for Computing Machinery, New York, NY, USA, 2024). [13] C. Rudin, Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nature Machine Intelligence 1, 206 (2019), received 30 Dec 2018; accepted 26 Mar 2019; published 13 May 2019. [14] P. Zschech, S. Weinzierl, and M. Kraus, Inherently interpretable machine learning: A contrasting paradigm to post-hoc explainable ai, Business & Information Systems Engineering , 1 (2025), received 17 Dec 2024; accepted 24 Jul 2025; published 15 Sep 2025. [15] Z. Ghahramani and G. E. Hinton, Variational learning for switching state-space models, Neural computation 12 (2000). [16] A. Farnoosh, B. Azari, and S. Ostadabbas, Deep switching auto-regressive factorization: Application to time series forecasting, Proceedings of the AAAI Conference on Artificial Intelligence 35, 7394 (2021). [17] M. T. Wojnowicz, K. Gili, P. Rath, E. Miller, J. Miller, C. Hancock, M. O’Donovan, S. Elkin-Frankston, T. T. Brunyé, and M. C. Hughes, Discovering group dynamics in coordinated time series via hierarchical recurrent switching-state models (2024). [18] R. W. Bachtiar, R. F. G. Meulenbroeks, and W. R. van Joolingen, Stimulating mechanistic reasoning in physics using student-constructed stop-motion animations, Journal of Science Education and Technology 30, 777 (2021), published 28 Apr 2021; issue date Dec 2021. [19] M. H. Wilkerson-Jerde, B. E. Gravel, and C. A. Macrander, Exploring shifts in middle school learners’ modeling activity while generating drawings, animations, and com-
putational simulations of molecular diffusion, Journal of Science Education and Technology 24, 396 (2015). [20] K. Wendell, M. S. Topçu, and C. Andrews, Mechanistic reasoning: How cause-and-effect thinking supports engineering design problem-solving, Science and Children 62, 44 (2025). [21] K. Melsky, I. Stuopis, K. Wendell, and E. C. Kemmerling, Personalized problems and student discourse in thermal fluid transport courses, International Journal of Mechanical Engineering Education 52, 457 (2024). [22] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding (2019). [23] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023). [24] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, Mistral 7b (2023). [25] J. Campbell, K. Ansell, and T. Stelzer, Evaluating ibm’s watson natural language processing artificial intelligence as a short-answer categorization tool for physics education research, Phys. Rev. Phys. Educ. Res. 20, 010116 (2024). [26] J. Wilson, B. Pollard, J. M. Aiken, M. D. Caballero, and H. J. Lewandowski, Classification of open-ended responses to a research-based assessment using natural language processing, Phys. Rev. Phys. Educ. Res. 18, 010141 (2022). [27] D. Wang, D. Shan, Y. Zheng, K. Guo, G. Chen, and Y. Lu, Can chatgpt detect student talk moves in classroom discourse? a preliminary comparison with bert, in Proceedings of the 16th International Conference on Educational Data Mining, edited by M. Feng, T. Käser, and P. Talukdar (International Educational Data Mining Society, Bengaluru, India, 2023) pp. 515–519. [28] R. K. Fussell, M. Flynn, A. Damle, M. F. J. Fox, and N. G. Holmes, Comparing large language models for supervised analysis of students’ lab notes, Phys. Rev. Phys. Educ. Res. 21, 010128 (2025). [29] K. Gili, K. Heuton, A. Shah, D. Hammer, and M. C. Hughes, Combining physics education and machine learning research to measure evidence of students’ mechanistic sensemaking (2025), arXiv:2503.15638 [physics.ed-ph]. [30] T. Ullmann, Automated analysis of reflection in writing: Validating machine learning approaches, Int J Artif Intell Educ 29, 217–257 (2019). [31] H. Auby, N. Shivagunde, V. Deshpande, A. Rumshisky, and M. D. Koretsky, Analysis of student understanding in short-answer explanations to concept questions using a human-centered ai approach, Journal of Engineering Education 114, e70032 (2025), https://onlinelibrary.wiley.com/doi/pdf/10.1002/jee.70032. [32] J. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, and Q. Yang, Why johnny can’t prompt: How non-ai experts try (and fail) to design llm prompts (Association for Computing Machinery, New York, NY, USA, 2023). [33] Z. Jiang, F. F. Xu, J. Araki, and G. Neubig, How Can We Know What Language Models Know?, Transactions of the Association for Computational Linguistics 8, 423
17 (2020). [34] J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen, What Makes Good In-Context Examples for GPT-3?, in Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, edited by E. Agirre, M. Apidianaki, and I. Vulić (Association for Computational Linguistics, Dublin, Ireland and Online, 2022) pp. 100–114. [35] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, On the dangers of stochastic parrots: Can language models be too big?, in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21 (Association for Computing Machinery, New York, NY, USA, 2021) p. 610–623. [36] F. M. Watts, A. J. Dood, and G. V. Shultz, Developing machine learning models for automated analysis of organic chemistry students’ written descriptions of organic reaction mechanisms, in Student Reasoning in Organic Chemistry (The Royal Society of Chemistry, 2022). [37] B. Moreau-Pernet, Y. Tian, S. Sawaya, P. Foltz, J. Cao, B. Milne, and T. Christie, Classifying tutor discursive moves at scale in mathematics classrooms with large language models, in Proceedings of the Eleventh ACM Conference on Learning @ Scale (L@S ’24) (Association for Computing Machinery, Atlanta, GA, USA, 2024) pp. 361–365. [38] V. Schechter, H. Dua, S. Dua, B. Zhang, D. Salz, R. Mullins, S. R. Panyam, S. Smoot, I. Naim, J. Zou, F. Chen, D. Cer, A. Lisak, M. Choi, L. Gonzalez, O. Sanseviero, G. Cameron, I. Ballantyne, K. Black, K. Chen, W. Wang, Z. Li, G. Martins, J. Lee, M. Sherwood, J. Ji, R. Wu, J. Zheng, J. Singh, A. Sharma, D. Sreepat, A. Jain, A. Elarabawy, A. Co, A. Doumanoglou, B. Samari, B. Hora, B. Potetz, D. Kim, E. Alfonseca, F. Moiseev, F. P. Gomez, and G. Hernández, Embeddinggemma: Powerful and lightweight text representations (2025). [39] A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, and A. Farhadi, Matryoshka representation learning, in Advances in Neural Information Processing Systems (NeurIPS), Vol. 35 (2022) pp. 36005– 36018. [40] T. O. B. Odden and R. S. Russ, Defining sensemaking: Bringing clarity to a fragmented theoretical construct, Science Education 103, 187 (2019). [41] M. J. Beal, Variational algorithms for approximate Bayesian inference (University of London, University College London (United Kingdom), 2003). [42] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, Variational inference: A review for statisticians, Journal of the American Statistical Association 112, 859–877 (2017). [43] E. Harvey, M. Petrov, and M. C. Hughes, Learning hyperparameters via a data-emphasized variational objective (2025), arXiv:2502.01861 [cs.LG]. [44] D. J. C. MacKay, Bayesian interpolation, Neural Computation 4, 415 (1992), https://direct.mit.edu/neco/articlepdf/4/3/415/812340/neco.1992.4.3.415.pdf. [45] tufts-ml, Mech4mech (2026), gitHub repository. Accessed: 2026-04-03. [46] L. Z. J. Jennifer Radoff and D. Hammer, “it’s scary but it’s also exciting”: Evidence of meta-affective learning in science, Cognition and Instruction 37, 73 (2019).
[47] K. Bergeron, D. Pamuk Turner, and D. Hammer, Reasoning through uncertainty: expert chemists’ analogical thinking on a novel problem, Chemistry Education Research and Practice 26, 1031 (2025). [48] O. Weller, K. Ricci, M. Marone, A. Chaffin, D. Lawrie, and B. V. Durme, Seq vs seq: An open suite of paired encoders and decoders (2025), arXiv:2507.11412 [cs.CL]. [49] L. R. Rabiner, A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, Proc. of the IEEE 77, 257 (1989). [50] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12, 2825 (2011).
A.
Thermo-Fluid mechanics problems & codes
Below, we present all six thermo-fluid mechanics problems given to students; their responses comprise the dataset used to train the adapted-HSRDM models. The resistance heater problem was the only problem administered to all five student groups, so we denote it as problem B whereas all other problems are denoted by A and the group number (e.g. 1A). For each problem, we report the problem-specific codebook produced by applying Russ’ framework to that problem instance. Codes for which we found no evidence in students’ responses are marked “[Not mentioned by the students]”.
18 Problem 1A: Ice Cream Shop Problem: The building of Farfar’s Danish Ice Cream Shop in Duxbury, MA is somewhat old and thus does not seem to have a great cooling system. As a result, sometimes the ice cream gets a bit melty even when it’s still in the freezer. The temperature in the ice cream shop is to be maintained at 55°F. Estimate the dimensions of the building, use thermodynamics principles to determine the maximum heat loss the shop can have, and suggest a method for minimizing this heat loss. Target phenomena: Ice cream shop maintained at 55 degrees fahrenheit, where the ice cream does not melt. Set-up conditions: Dimensions of the ice cream shop, environmental conditions outside of the shop, refrigerator and/or air conditioner inside of the shop. Entities: Ice cream, air, refrigerant, air conditioner coil, compressor, condenser. Activities: Air moving through the coils, air mixing with the refrigerant, the compressor pressurizing and raising the temperature of the air. [Not mentioned by the students] Properties: Temperature of the air, temperature of the ice cream, mass of the ice cream, density of the air. [Not mentioned by the students] Organization: Hot air traveling from the inside to the outside of the building. [Not mentioned by the students] Chaining: Hot air travels through the coils and is absorbed by the refrigerant, and flows to the outside of the building, where it is compressed into a higher temperature and pressure gas before being absorbed by the surrounding air. [Not mentioned by the students]
Problem B: Resistance Heater Problem: Design an experiment complete with instrumentation to determine the specific heats of a gas using a resistance heater. Discuss how the experiment will be conducted, what measurements need to be taken, and how the specific heats will be determined. What are the sources of error in your system? How can you minimize the experimental error? Target phenomena: Gas inside the container is heated to a different measured temperature. [Not mentioned by the students] Set-up conditions: Setting up the container conditions, the gas, the resistor, the power sources, and the sources of measurement. Entities: electrons, atoms (inside of the resistor), molecules, atoms (inside of the gas). [Not mentioned by the students] Activities: Electrons moving through the wire, atoms colliding with electrons to resist their movement, gas atoms moving quickly. [Not mentioned by the students] Properties: Charges, masses of the electrons or atoms [Not mentioned by the students] Organization: Electrons in the resistor, atoms in the gas, spread of the atoms in the container. [Not mentioned by the students] Chaining: The electrons move through the wire and collide with atoms in the resistor, which causes the electrons to lose energy and the resistor atoms to gain energy/heat up. [Not mentioned by the students] Problem 2A: Snowmaking Problem: Most ski resorts in the U.S. use snow guns to make additional snow to supplement natural snow. These machines use water and compressed air. The air forces the water to form tiny droplets, which are then expelled from the nozzle and form ice crystals, which then fall to the ground as snow. Compressed air cools as it expands, which assists with converting the water droplets into snow. Choose your favorite ski resort and the desired depth of snow for the best skiing, and use thermodynamics to determine how long it will take to cover the ski trails in that amount of snow. You may assume that one snow gun uses about 100 gallons of water per minute and that the compressor can produce 50 cfm (cubic feet per minute) of air. Target phenomena: Covering the ski trail in snow. Set-up conditions: Setting up snowguns around the trail. Entities: Air, water, snow, ice. Activities: Mixing together, combing out of the novel, turning into ice. Properties: Density, compressed, temperature, sparse. Organization: Inside of the tube, different novels or systems. Chaining: expanding air and expanding water are turning into ice.
19 Problem 3A: Bus Problem: About a decade ago, Stanford University successfully tried using waste vegetable oil from the dining halls as fuel for campus shuttles (https : //news.stanf ord.edu/news/2006/january25/biodiesel− 012506.html). What if our campus tried to do this? Plan a useful bus route around our campus and specify the volume of fuel needed for the bus to travel this route without having to refuel. You may assume the energy density of vegetable oil is 42.20 MJ/kg or 30.53 MJ/L. Target phenomena: Bus travels a route on vegetable oil fuel without needing to re-fuel. Set-up conditions: Route distance, road type, bus, bus weight. Entities: Fuel, vegetable oil, air molecules,crankshaft (of piston), valves (of piston), camshaft (of piston). Activities: Fuel is released into the combustion chamber, fuel and air are compressed, crankshaft turns [Not mentioned by the students]. Properties: Amount of space taken up by the fuel or the air. [Not mentioned by the students] Organization: Fuel mixes with air, the crankshaft provides energy to the rest of the vehicle engine. [Not mentioned by the students]. Chaining: The extension rod of the piston provides a force on the crankshaft, such that it turns and powers the rest of the engine as well as the camshaft, which opens the valves that let fuel into the chamber for compression and combustion. [Not mentioned by the students].
Problem 4A: High-altitude plane Problem: Gas turbine engines used in airplanes consist of a fan followed by a compressor, diffuser, combustor, turbine, and sometimes an afterburner. You are designing the engine for a high- altitude airplane. Normally, commercial planes operate best around 35,000 ft above sea level, but your plane should operate optimally at around 100,000 ft. Because of the high altitude, there will be a lower concentration of oxygen than normal, and the air entering the engine will be colder. Design a protocol for getting the oxygen up to the appropriate temperature and pressure needed for combustion. Keep in mind your solution has to be relatively light. Target phenomena: Plane is flying at 100 thousand feet with high enough temperature and pressure oxygen to keep the plane in the air. Set-up conditions: Type of plane, atmospheric conditions at high altitude. Entities: Air, diffuser, compressor, combustor, turbine, afterburner, oxygen, fuel, exhaust, nozzle. Activities: Air slowing down, oxygen burning, turbine spinning. Properties: Air temperature, air pressure, air volume, compressor or diffuser material. Organization: air moving through diffuser and into compressor and into combustor, energy coming out of the combustor. Chaining: high energy gas leaving the combustor to spin the turbine, which manually controls the compressor.
20 Problem 5A: Rocket Problem: Hybrid rockets use a combination of solid and liquid or gaseous propellants. In hybrid rockets, a stable oxidizer is used with a solid fuel. In order to be used, the fuel needs to be vaporized. The primary difficulty with hybrids is with mixing the propellants during the combustion process. In a hybrid rocket, the mixing happens at the melting or evaporating surface. The mixing is not well-controlled and generally, a lot of propellant is left unburned, limiting the motor’s efficiency. On the other hand, liquid propellants are generally mixed with oxidizer by an injector at the top of the combustion chamber which directs many small streams of fuel and oxidizer into one another. Based on reasonable efficiencies of both liquid fuel and hybrid fuel processes, estimate the weight of fuel necessary to get a specific rocket of your choice to low Earth orbit if the fuel is liquid vs. hybrid. Target phenomena: Two rockets (one with liquid fuel and one with hybrid fuel) reach low Earth orbit. Set-up conditions: Two rockets (one with liquid fuel and the other with hybrid - solid and gas or solid and liquid) conditions, Earth’s atmosphere conditions. Entities: Fuel, propellants, molecules, pumps, valves, oxidizer. Activities: Fuel and oxidizer mixing, combined substance burning, molecules compressing, combined substance ejecting. Properties: State of matter (liquid, solid, gas) of fuel, amount of space taken up by the fuel, temperature or pressure of the molecules. Organization: Fuel and oxidizer starting out in separate containers, slowly combining, molecules coming out of the rocket. Chaining: Liquid fuel slowly mixes with the oxidizer, and combusts to release hot gas as thrust for the rocket to move. [Not mentioned by the students]
B.
Human annotation procedure and IRR analysis
Two human analysts collaborated to tailor Russ’ framework to a specific problem context – a professor of mechanical engineering education and a postdoctoral researcher working at the intersection of machine learning and education research. The two analysts first established a general coding strategy applicable across all problems, and then calibrated their problem-specific application of that strategy using the snowmaking problem as a reference case. To obtain evidence of annotation reproducibility with the developed code, the two analysts conducted an inter-rater reliability (IRR) procedure on student text from this problem. After this calibration phase, the remaining problem codes were developed by the postdoctoral researcher. Below, we describe the general coding strategy details, as well as the IRR implementation and results.
General coding strategy details. For a specific problem, the first step in coding the students’ talk for mechanistic reasoning is to clarify for ourselves the target phenomenon. We highlight the importance of this decision, as what is chosen to be the target phenomena defines the scale of objects we believe students will reason mechanistically about. For example, if the target phenomena is the amount of snow coverage as a result of the snowgun vs. the amount of snow coverage after a day in the sun. In the first scenario, we expect evidence of mechanistic reasoning to look like a discussion about the inter-workings of the snowgun. In the second scenario, we expect evidence to looks like a discussion about how sunlight interacts with snow. In this snowmaking problem, the students are tasked with determining how long it takes to cover a ski resort with snow from a snow gun. Thus, the target phenomenon is production of snow to cover ski trails. Next we do a first read-through of the entire conversation to identify the specific instantiations of the remaining mechanistic elements for this particular problem. This step enables us to be internally consistent - within a specific conversation - regarding which student language we consider to refer to activities versus properties versus organization of the different entities they have identified. Finally, we code sentence-by-sentence, considering whether each sentence presents evidence of one or more of the seven categories. We apply a code when we find an act of mechanistic reasoning in the conversational turn rather than only the linguistic features of mechanistic reasoning. For example, we code for entities when the student utterance is focused on identifying the entities that are relevant to the problem, rather than mentioning the entities to take the next identification step. For example, in the following utterance, the act is not identifying the entities of air and water (category #3) but rather mentioning them in order to accomplish the higher level act of identifying an activity (category #4). “If we have to have the air and the water come out of the same nozzle, we shouldn’t do - we shouldn’t split it. They just do that so it covers more area.” IRR implementation and results. The two analysis agreed on the general coding strategy detailed above, and then selected the snowmaking problem, due to its large number of utterances compared to other problems (372 utterances), to calibrate their strategy for developing a problem-specific code and applying it to the text. The analysts conducted a full read-through of the student conversation, developed an initial code for the problem together, and then coded the full problem individually (372 utterances). As evidence of initial inter-rater reproducibility, we report the Cohen’s κ and accuracy metrics for the individual coders across all categories in Table B. We report the Cohen’s κ and accuracy metrics for each category (e.g. target phenomenon). The same student utterance can have multiple categorical labels, and so we include a separate metric for the inter-rater agreement of the maximum (max) evidence category in the hi-
21 Label Target phenomena Set-up conditions Entities Activities Properties Organization Chaining No evidence Max evidence
Accuracy Cohen’s κ 98.59 96.89 67.80 88.70 81.64 95.76 98.87 81.92 72.03
0.6989 0.4073 0.0842 0.4024 0.2809 0.6716 0.4943 0.5723 0.4262
TABLE IV: Pre-discussion inter-rater reliability metrics for each code column and the maximum code evidence on the full snowmaking problem. Column Target phenomena Set-up conditions Entities Activities Properties Organization Chaining No evidence Max evidence
Accuracy Cohen’s κ 97.22 99.44 91.11 89.44 85.56 98.33 100.00 86.67 82.22
0.6032 0.8861 0.3814 0.4780 0.2439 0.9097 NaN 0.6837 0.6380
TABLE V: Post-discussion inter-rater reliability metrics for each code column and the maximum code evidence on the first 181 utterances of the snowmaking problem.
erarchy per utterance. This informs how often two raters observe the same max evidence in a given utterance. This validation is the most relevant to our usage of the data, as post the IRR procedure, decided to train the ML classifier feedback mechanism discussed in Sec. III D with only the max evidence labels. Throughout the main text and in the Appendix, when we refer to the number of positive labels for a category, we only refer to the max evidence. As in, a student utterance is labeled as its max category, and it is then a positive example for that category. We observe relatively high accuracies across categories except for entities. Through discussion, it was learned that both raters needed to resolve ambiguity around what counted as an act of reasoning about entities vs. the mere mentioning of entities. The Cohen’s κ values indicate mostly moderate agreement. We expect this to be lower than accuracy as κ values take into account how likely a rater is to mark 0/1 for a certain category based on their overall marking frequency. Post the IRR assessment, the analysts came together for a discussion to see whether they could resolve their disagreement and improve the language in the codebook. The analysts updated the code with the intention of making the code less ambiguous for a future user. After this update, the analysts re-coded nearly half (181 utterances) of the problem again separately. As evidence of reproducibility following codebook refinement, we report Cohen’s κ and accuracy across all categories for the two coders in Table B. We also include the inter-rater
Column Target phenomena Set-up conditions Entities Activities Properties Organization Chaining No evidence Max evidence
Accuracy Cohen’s κ 99.45 96.15 71.98 87.91 82.42 93.96 99.45 85.71 72.53
0.9063 0.3480 0.1749 0.4701 0.0125 0.6847 0.0000 0.6703 0.4577
TABLE VI: Pre-discussion inter-rater reliability metrics for each code column and the maximum code evidence on the first 181 utterances of the snowmaking problem.
agreement metrics on the first 181 utterances from the previous codings for an apples-to-apples comparison in Table B. We observe from comparing the values in Table B and Table B that all accuracies and κ values increase post the discussion, except for the category target phenomena. This is evidence to suggest that the raters did not fully resolve more ambiguity for this category as a result of the discussion. Despite this, agreement on the max evidence category increased significantly – from moderate to substantial– in κ value. We again emphasize this increase, as this is the annotation that we utilize in the ML portion of this work. Overall, the IRR assessment provides moderate to substantial evidence that annotations for the snowmaking problem using the code in Sec. A can be reproduced. However, stronger evidence of reproducibility would require a third rater, who had not been a part of the original discussions, to code the problem and compare with the first two rater’s results. We acknowledge this as a resource limitation. Furthermore, we recognize a key limitation of the overall IRR procedure is that all coding was done on the same problem. We utilized the time resources that we had to prioritize the fine-tuning of the codebook for one problem, so that learned considerations throughout the analyst’s discussions could then be applied across problems. We acknowledge that an improved reproducibility procedure would include the step of the analysts developing a code for a new problem together (or separately for even stronger evidence), coding that entire problem individually, and then conducting another round of IRR validation. This would provide stronger evidence that the strategy of developing a specific code for a problem and then applying that code to the student text is reproducible. Due to limited resources, the postdoctoral researcher developed the remaining problem-specific codes independently using learnings from the discussions on the snowmaking problem, and then shared the codes with the mechanical engineering education professor for final validation. No further updates were made to the codebook. The postdoctoral researcher coded the rest of the student text using these codes. Only the labels produced by the postdoctoral researcher are used to train the feed-
22 back classifier in Sec. III D. In future work, we believe it’s important to obtain more resources (e.g. coding time, more raters) to increase the evidence of annotation reproducibility. If we cannot agree on what evidence of mechanistic reasoning looks like, we lack a well-defined “ground truth” for the ML models.
C.
Data summary
See Tables VII - XVI for the number of total student utterances in each problem and the distribution of utterances across students in each problem. In the raw data for problem 1B, two additional students walked by the group and exchanged a two or three casual utterances. None of these utterances contain evidence of mechanistic reasoning. As we did not believe it made sense for these students to be represented as entities, we removed these lines during the initial processing of the raw data. Pseudonym Utterance counts Cameron Logan Sam Max Total
154 137 114 79 484
Pseudonym Utterance counts Jamie Morgan Total
55 54 109
TABLE X: Problem 2B
Pseudonym Utterance counts Drew Alex Charlie Blake Total
189 151 149 87 576
TABLE XI: Problem 3A
Pseudonym Utterance counts Drew Alex Charlie Blake Total
61 106 94 67 328
TABLE XII: Problem 3B
TABLE VII: Problem 1A
Pseudonym Utterance counts Pseudonym Utterance counts Cameron Logan Sam Max Total
69 61 54 53 237
Xian Taylor Andy Parker Total
285 270 43 10 608
TABLE XIII: Problem 4A
TABLE VIII: Problem 1B
Pseudonym Utterance counts Pseudonym Utterance counts Jamie Morgan Total
186 186 372
TABLE IX: Problem 2A
D.
Probabilistic model factorization details
As discussed in Sec. III B, our model defines a joint distribution over all random variables: 1:N,1:J p(s1:N , x1:N,1:J , θ). Under the assumption 0:T , z0:T 0:T that each generated sequence i is drawn independently,
Xian Taylor Andy Parker Total
178 176 19 18 391
TABLE XIV: Problem 4B
Pseudonym Utterance counts Palmer Golden Dylan Rowan Total
318 310 91 74 793
TABLE XV: Problem 5A
23 Pseudonym Utterance counts Palmer Golden Dylan Rowan Total
102 81 79 12 274
TABLE XVI: Problem 5B
we re-write the model in the following form: N Y
i,1:J i p(xi,1:J 0:T , z0:T , s0:T |θ)p(θ)
(D1)
i=1
=
N Y
i,1:J i p(si0:T |θ)p(xi,1:J 0:T , z0:T |s0:T , θ)p(θ)
i=1
From here, we incorporate the assumptions discussed in Sec. III B to obtain a further factorized form of the components above:
p(si0:T |θ) = p(si0 |θs−init )
T Y
p(sit |sit−1 , xi,1:J t−1 , θss ), (D2)
t=1 i,1:J i p(xi,1:J 0:T , z0:T |s0:T , θ) =
J Y
p(z0i,j |θz−init )
j=1
p(xi,j 0 |θx−init )
T Y
i,j , xi,j p(zti,j |sit , zt−1 t−1 , θes )
t=1 i,j i,j p(xt |zt , xi,j t−1 , θee )
Each of these factorized distributions correspond to different components introduced in Sec. III B: systemlevel starting distribution, system-level transition distributions defined by G, entity-level starting distribution, entity-level transition distributions defined by F , emission at t = 0, and emission for t ≥ 0.
E.
Inductive bias via data feedback details 1.
Classifier selection details.
We considered various pre-trained and need-to-train classifiers, and selected based on task alignment and the amount of computational resource requirements for deployment (e.g. the number of parameters stored and used during inference). Task alignment refers to the extent to which the pre-training objective is congruent with, or relevant to, our current task. Secondly, for the trained adapted-HSRDM to be able to run on one’s local device (e.g. M1 chip with 32GB RAM), we need to be able to download and run the model for the feedback mechanism locally while also running variational inference. Thus, we need to minimize the number of parameters stored at inference time.
Re-purpose from prior work. We first considered re-purposing an existing method from the literature. The authors in [29] fine-tune and validate the Bidirectional Encoder Representations from Transformers (BERT) [22] encoder-only language model for the binary multi-label classification task of measuring evidence of students’ mechanistic reasoning on self-contained responses to physics problems. The authors utilize domain labels that map to Russ’ framework minus the category target phenomena. Re-purposing this classifier for our task eliminates the need to extract training data for the feedback mechanism. However, this method is not useful due to task mis-alignment. There is a difference between our interpretation of a student identifying the items in Russ’ framework if the data is an individual summary explanation vs. a sequence of in-moment dialogue. In the former, we have no information about the process leading up to the explanation generation, and so we can reasonably make the assumption that the text represents a self-contained account of all of the student’s intellectual work (either conducted in the moment of writing the explanation or in moments prior). Therefore, if a student mentions one of the items (e.g. an entity, activity, property), the annotator assumes this is identifying one of these items. In contrast, when we know that students are engaging in dialogue to solve a problem, we can reasonably assume that the intellectual work is spread across different moments in time, and mentioning an item in one moment does not necessarily mean that a student is doing the intellectual work in that moment to identify that item; it could be referencing the item from intellectual work that was done in previous moments. As a more concrete example, consider the following two utterances from the snowmaking problem: J: “Alright. Also I’m concerned. Is this a pure substance? Do we have to worry about that? The fact that it’s air and water mixed together?” M: “I don’t know the density of water in a snow gun.” In the first utterance, the student identifies that the substance under consideration is composed of the entities air and water. In the second utterance, the student uses the word “water”, but is not using it to identify water as an entity. Rather, in this moment, the student is reasoning about a property of that entity - the density. As the intellectual work of identifying “water” was done in the first utterance, we would want to mark a 1 for the entity domain. As this does not appear to be the intellectual work done in the second utterance, we would want to mark this a 0 despite the term “water” being used. Given that the classifier in [29] was never trained on student responses that mentioned items without doing the intellectual work (i.e. the response should receive a label 0 for that item rather than a 1), we concluded that this classifier would do poorly on our data, which includes these characteristics in a large portion of the responses.
24 Transformer decoder for in-context learning. We considered using a pre-trained decoder-only language model, where one can explicitly prompt the model to return the labels associated with the provided utterance. These labels can then be mapped to a binary vector. Recent work [48] provides empirical evidence that using decoder-only language models with more parameters for classification tasks performs worse than using encoder-only models for these tasks with fewer parameters. Choosing such a method requires storing a ∼ billion parameter model and running this model at each timestep in the adapted-HSRDM. We prioritized a feedback method that keeps the number of model parameters low rather than needing to download a ∼ billion parameter model. Transformer encoder for transfer learning. Lastly, we considered training a foundation language encoderonly model for our specific downstream classification task. In an attempt to not retrain foundation model weights, which demands more computational resources – especially when optimizing over various hyperparameters, we decided to freeze the weights of the EmbeddingGemma encoder and train a small neural network architecture to classify each generated embedding. Unlike decoder-only models, the classification outputs are deterministic for each embedding, and the number of parameters that need to be stored locally and used at each timestep is significantly fewer (millions instead of billions). 2.
Training data selection.
We first selected 2 problems (2A and 2B) as a test set for the adapted-HSRDM, which left 8 problems for training the supervised feedback mechanism classifier and the CAVI training. As we did not want our feedback mechanism to be trained on the exact same data as the CAVI training, we used the smallest sub-set believed possible for training the classifier (4 problems). To once again prevent data leakage across students, we chose 4 problems from two student groups. As we only have one problem outside of our test set that contains examples of chaining (problem 4A), we included the data from problems 4A and 4B in our training. To select the second student group, we picked the pair of problems that had the fewest number of positive labels in the hierarchy past the entity domain. The reason for this is that problems 4A and 4B contain the most positive labels in this subsection (entity–chaining) of the hierarchy out of all of the training data, and we would like to ensure that there is enough non-overlapping positive labels in this sub-section between the supervised training and the unsupervised training loss. Thus, we picked problems 1A and 1B. See Table XVII for all positive label counts present in the dataset and in the train/test splits. 3.
Method details.
Method justification. To avoid needing a validation set, we utilized a recent Bayesian variational inference
approach put forth in [43] for model selection, where one can minimize a negative data-emphasized ELBO (DEELBO) as a method of implicitly choosing the model with the optimal prior parameters over neural network parameters (i.e. the weight decay hyper-parameter), as well as the model that most likely generates the training data. By definition, minimizing the negative DE-ELBO mitigates over-fitting behavior if the assumed prior and likelihood distributions for the data are reasonably specified. The data-emphasized term simply adjusts the amount of importance that is placed on the data relative to the prior term in the ELBO, which may be necessary when the number of model parameters np is greater than the amount of data N for training (np =17544 > N =1721 in our situation). The demonstrations in Ref. [43] provide evidence that this method achieves just as high or better performance on test data for multi-label text classification tasks. In utilizing this method, we were able to use a 4 problem training set for the classifier, rather than 6 problems (the extra two problems would have served as our validation set). Training details. We trained a classifier that takes EmbeddingGemma encoded utterance vectors xi ∈ R128 as input and predicts the maximum mechanistic-evidence class yi ∈ {0, 2, ...7} under Russ’ taxonomy (Sec. II). The zeroth class is for no-evidence, whereas classes 1−7 quantify how much mechanistic reasoning evidence is present in the utterance. We utilized a 2-layer neural network (NN) with a ReLu activation function as our model, and assumed an isotropic gaussian distribution for our prior over these parameters θ with a learnable weight decay λ. We set the hidden layer of the NN to our embedding dimension size D = 128. We assumed a categorical distribution for our probabilistic likelihood function, which is computed as the softmax over per-example class logits zi ∈ R↚ from the output of the 2-layer NN. We define our probabilistic likelihood p(yi |xi , θ, M ) and prior p(θ) density in the following form: p(yi |xi , θ) = Cat(yi |softmax(zi )), p(θ) = N (θ|0, λ
−1
I)
(E1) (E2)
For our approximate posterior, we also assumed a parameterized ϕ isotropic gaussian distribution, with a mean based on the current parameter mean θ̄ and a trainable parameter for the variance ρ2 . We define our approximate posterior in the following form: qϕ (θ) = N (θ|θ̄, ρ2 I)
(E3)
The approximate posterior is trained to match the true posterior, such that qϕ (θ) ≈ p(θ|x1:N , y1:N ). The parameters of the true distribution and the approximate posterior can be learned via an optimization procedure
25 Target Set-up Entities Activities Properties Organization Chaining phenomena conditions
Problem
No evidence
1A 1B 2A 2B 3A 3B 4A 4B 5A 5B Total
415 209 298 102 513 267 505 323 761 243 3636
3 1 6 0 1 7 1 0 3 0 22
60 17 8 6 51 36 6 68 10 29 291
5 9 5 1 10 12 39 0 2 1 84
0 1 13 0 0 4 18 0 5 0 41
2 0 21 0 1 0 27 0 11 1 63
0 0 21 0 0 2 10 0 1 0 34
0 0 1 0 0 0 2 0 0 0 3
Supervised train Unsupervised train Semi-supervised test
1452 3338 400
5 16 6
151 277 14
53 78 6
19 28 13
29 42 21
10 13 21
2 2 1
TABLE XVII: Positive label counts by problem
that minimizes the per-datapoint negative (DE-ELBO): N
−ELBO = −
1 X np [ Eθ∼q(θ) [log p(yi |xi , θ)] N i=1 N
(E4)
−KL(q(θ)||p(θ)] Note that the negative-log-likelihood (NLL) term (left) prioritizes fitting the training data well and the Kullback–Leibler (KL) divergence term (right) prioritizes obtaining a posterior distribution over parameters that ren mains close to the prior. The Np term allows one to up-weight the data-fit term when a large amount of parameters might cause the prior term to dominate. The data-fit term will be maximized when the correct label yi is generated from the model. Due to the class imbalance over our K=8 classes, we weighted the importance N of each class in the training by 8×n , where nk is the k number of examples belonging to the class k. Training data selection details can be found in Appendix Sec. E 2. Results. The weight decay hyper-parameter λ was learned implicitly via the variational approach, and we conducted a grid search to optimize for the learning rate (LR) α over different parameter initializations that produces the lowest negative ELBO over 1000 training epochs. See Table ?? for all final negative ELBO results over five different parameter initialization seeds and fourteen different learning rates {α}. The best model is in bold (seed=7, α=2.9). See Table E 3 for the per-class accuracy scores computed from the model’s predictions on the training data. We conducted three additional checks, outlined below, to verify that our model choice is reasonable. Negative ELBO, NLL, and KL behavior. We checked whether the individual negative ELBO, NLL and KL terms in Eq. E4 computed over 1000 training epochs also descend and converge to minimum values. The NLL term represents the fit of our model to our training data
and the KL terms represents the regularization of our approximate posterior distribution to our prior distribution. While we expect the negative ELBO to be the primary guidepost, if the NLL term is low and the KL term is high, this is a signal of overfitting. Thus, for our best model (seed=7, α=2.9), we ensured that both terms computed over epochs show reasonable behavior (see Fig. 4). Learning rate behavior. In our initial training procedure, we only included learning rates up to α = 0.5, and found the model associated with this rate to be optimal. Thus, we extended our grid search to values that iteratively +0.4 up to α=3.3. We then found that α=2.9 led to the optimal model. However, we noticed that the lowest negative ELBO value was similar among all high learning rates, indicating similar convergence paths towards local minima. Thus, to do a final check that model training ran as expected, we zoomed in on the first 20 steps of training for each learning rate and looked for similar descents happening in fewer iterations. We observed that increasing the LR led to earlier descents. Parameter initialization stability. Oftentimes, parameter initialization can impact model performance, and different initializations then require different learning rates to do well. We checked the training stability to observe whether the initialization parameters played a critical role in the model selection. The results in Table ?? show that model performance is stable across initializations for different learning rates. We used this as evidence to conclude that searching over more starting points would not necessarily lead to a better model. F. 1.
ML training details
Training Algorithm Overview
the HSRDM is trained in a fully unsupervised fashion via Bayesian variational inference techniques [41, 42]. One i,j can utilize a structured distribution qϕi (z0:T , si0:T ) parameterized by ϕi that can be trained to approximate
26 105
100
103
10 1
10 1
kl
nll
Negative ELBO
100
101 10 1
10 2
10 2 0
200
400
600
Number of Iterations
800
1000
(a) Negative ELBO
0
200
400
600
Number of Iterations
800
1000
0
(b) Negative-log likelihood
200
400
600
Number of Iterations
800
1000
(c) Kullback–Leibler divergence
FIG. 4: Log-scale convergence behavior for the optimal model (seed: 7, LR: 2.9). Left: full negative ELBO. Middle: the negative log-likelihood or data term in the ELBO. Right: the KL term in the ELBO between the prior and approximate posterior.
TABLE XVIII: Final losses by initialization and LR (Part 1).
TABLE XIX: Final losses by initialization and LR (Part 2). Init seed
Init seed 7 7 7 7 7 7 7 7 7 7 7 7 7 7 123 123 123 123 123 123 123 123 123 123 123 123 123 123 213 213 213 213 213 213 213 213 213 213 213 213 213 213
LR
Final loss
3.3 0.0096656819805502 2.9 0.009665491990745 2.5 0.0096656940877437 2.1 0.0096656968817114 1.7 0.0096655311062932 1.3 0.0096656577661633 0.9 0.0096659399569034 0.5 0.0096686063334345 0.1 0.0131261078640818 0.01 0.0179602280259132 0.001 0.0886219218373298 0.0001 5.237014293670654 1e-05 6.147874355316162 1e-06 6.2400665283203125 3.3 0.0096749262884259 2.9 0.0096757467836141 2.5 0.0096768550574779 2.1 0.0096779707819223 1.7 0.0096803484484553 1.3 0.0096833296120166 0.9 0.0096903471276164 0.5 0.0097132893279194 0.1 0.014538419432938 0.01 0.0212347786873579 0.001 0.0887199118733406 0.0001 5.237014293670654 1e-05 6.147874355316162 1e-06 6.240066051483154 3.3 0.0096758734434843 2.9 0.0096771800890564 2.5 0.0096781142055988 2.1 0.0096795102581381 1.7 0.0096811978146433 1.3 0.0096835847944021 0.9 0.0096878539770841 0.5 0.0097013833001255 0.1 0.0130045237019658 0.01 0.0172159876674413 0.001 0.0886680483818054 0.0001 5.237014293670654 1e-05 6.147874355316162 1e-06 6.240066051483154
512 512 512 512 512 512 512 512 512 512 512 512 512 512 637 637 637 637 637 637 637 637 637 637 637 637 637 637
LR
Final loss
3.3 0.0096786580979824 2.9 0.0096797160804271 2.5 0.009681187570095 2.1 0.0096823880448937 1.7 0.0096843177452683 1.3 0.0096874348819255 0.9 0.0096928542479872 0.5 0.009710619226098 0.1 0.0133584877476096 0.01 0.0178864225745201 0.001 0.0886841490864753 0.0001 5.2370147705078125 1e-05 6.14787483215332 1e-06 6.2400665283203125 3.3 0.0096724024042487 2.9 0.0096732759848237 2.5 0.0096739986911416 2.1 0.0096749421209096 1.7 0.0096763456240296 1.3 0.0096787083894014 0.9 0.0096830110996961 0.5 0.0096974587067961 0.1 0.0137728871777653 0.01 0.0196864567697048 0.001 0.0886529237031936 0.0001 5.237014293670654 1e-05 6.147874355316162 1e-06 6.2400665283203125
i,1:J i the true posterior distribution p(z0:T , s0:T |xi,1:J 0:T , θ) over latent variables in sequence i. This can be done by maximizing the per-datapoint Evidence Lower Bound (ELBO): N
ELBO =
1 X i,1:J i Eq log p(xi,1:J 0:T , z0:T , s0:T |θ) N i=1 ϕi
(F1)
i,1:J i −Eqϕi log qϕi (z0:T , s0:T )
For the interested reader, see Appendix Sec. F 2, for a detailed derivation of the ELBO from our joint distribution and factorization assumptions in Sec.III B, which offers a theoretical explanation as to how maximizing the
27 {s0:T }N i=1 , given our parameters θ, we derive the Evidence Lower Bound (ELBO) for unsupervised training of the model. First, we factorize the joint into the following form:
Class Accuracy 0 1 2 3 4 5 6 7
0.9074 1.0000 0.8409 0.8099 0.9603 0.8047 0.9885 1.0000
N Y i=1
=
N Y
TABLE XX: The per-class accuracy scores of the classifier predictions on the training data.
i=1
ELBO allows one to approximate the assumed true posterior. As done in Ref. [17], to maximize the ELBO in practice, we make a mean field assumption to faci,j i,j torize qϕi (z0:T , si0:T ) = qϕi1 (z0:T ), qϕi2 (si0:T ), an additional independence assumption that allows the parameters ϕi = {ϕi1 , ϕi2 } of each distribution to be computed and updated separately. We implement coordinate ascent variational inference (CAVI) [42] to train all model parameters, which includes three separate steps:
=
Variational expectation system (VES)
(F3)
i,1:J i,1:J i qϕ∗ i1 (z0:T ) ∝ exp Eqϕi [log p(xi,1:J 0:T , z0:T , s0:T |θ)] 2
Maximization step (M-step) θ∗ = argmaxθ
N X
(F4)
i,1:J i Eqϕi log p(xi,1:J 0:T , z0:T , s0:T |θ)
i,1:J i i,1:J i p(xi,1:J 0:T | z0:T , s0:T , θ) p(z0:T , s0:T , θ)
N Y
i,1:J i i,1:J p(z0:T , s0:T | xi,1:J 0:T , θ) p(x0:T | θ)
i=1
From here, we factorize and re-position some terms to obtain the latent variable posterior: N Y
i,1:J i p(z0:T , s0:T |xi,1:J 0:T , θ)
(F5)
i=1 QN i,1:J i,1:J i p(xi,1:J 0:T |z0:T , s0:T , θ)p(s0:T , z0:T |θ) = i=1 QN i,1:J i=1 p(x0:T |θ)
(F2)
i,1:J i qϕ∗ i2 (si0:T ) ∝ exp Eqϕi [log p(xi,1:J 0:T , z0:T , s0:T |θ)] 1
Variational expectation entity (VEZ)
i,1:J i p(xi,1:J 0:T , z0:T , s0:T | θ)
Rather than learning the latent variables and model parameters θ that maximize the posterior distribution, we aim to learn the posterior distribution that maximizes QN the evidence term i=1 p(xi,1:J 0:T |θ). However to maximize this evidence term, we would have to compute a highly intractable integral – marginalizing over all hidden variables for each entity and across time points. Thus, instead, we maximize the lower bound of the log evidence as a proxy with an approximate distribution for the posterior. For example, the evidence can be written as:
i
In the VES step, the optimal parameters of each approximate system posterior distribution qϕ∗ i are computed via 2 a forwards-backwards dynamic programming algorithm [49], given a fixed θ and the current approximate entity posterior distribution qϕ∗ i . In the VEZ step, the 1 same procedure is conducted, but the system posterior parameters are fixed along with θ. Both VES and VEZ steps combined form the E-step. In the M-step, given the current optimal system and entity approximate posterior distribution parameters, ϕi1 , ϕi2 , the other model parameters (i.e. those of the true distributions) θ are maximized via gradient-ascent or a closed-form solution when available. These steps offer an efficient algorithm to update all of the parameters in the model to maximize the ELBO in Eq. F 1. For the interested reader, see Appendix Sec. F 3 for a detailed derivation of the CAVI updates for our problem and see Appendix Sec. F 4 for details regarding the algorithmic implementations for these updates, including the forwards-backwards inference procedure. 2.
ELBO derivation
Starting from our joint distribution over our observed i,1:J N N data {xi,1:J 0:T }i=1 and our latent variables {z0:T }i=1 ,
N Y
p(xi,1:J 0:T |θ) =
i=1
QN
i,1:J i,1:J i,1:J i i i=1 p(x0:T |z0:T , s0:T , θ)p(s0:T , z0:T |θ) QN i,1:J i i,1:J i=1 p(z0:T , s0:T |x0:T , θ)
(F6) The denominator and numerator are combined so that we have: N Y
p(xi,1:J 0:T |θ)
(F7)
i=1
=
N X Y
i,1:J i i,1:J i p(xi,1:J 0:T |z0:T , s0:T , θ)p(z0:T , s0:T |θ)
i=1 z,s
By taking the log of both sides, we obtain: N X
log p(xi,1:J 0:T |θ)
(F8)
i=1
=
N X i=1
log
X
i,1:J i i,1:J i p(xi,1:J 0:T |z0:T , s0:T , θ)p(z0:T , s0:T |θ)
z,s
We assume some structured auxiliary probability distrii,1:J i bution over our latent variables qϕi (z0:T , s0:T ) that we
28 are going to learn the ϕ parameters of alongside our parameters θ for all N sequences. (z i,1:J ,si
q
)
0:T We add a qϕi (z0:T to the right side of the equai,1:J i ,s ) ϕi
tion: N X
0:T
0:T
log p(xi,1:J 0:T |θ) =
N X
i=1
i=1
log
X
i,1:J i p(xi,1:J 0:T |z0:T , s0:T , θ)
z,s
3.
CAVI update rule derivations
To utilize the CAVI updates proposed in [42], we make a mean field approximation, meaning that i,1:J i i,1:J qϕi (z0:T , s0:T ) = qϕi1 (z0:T )qϕi2 (si0:T ). This allows us to take derivatives of the ELBO with respect to i,1:J qϕi1 (z0:T ) and qϕi2 (si0:T ) separately. First, we disentangle these two posterior terms in the ELBO. The first ELBO term is written as:
(F9) i,1:J i Eqϕi log p(xi,1:J 0:T , z0:T , s0:T | θ)
qϕ (z i,1:J , si0:T ) i,1:J i p(z0:T , s0:T |θ) i 0:T i,1:J i qϕi (z0:T , s0:T )
=
We combine the terms on top and transform the sum over hidden variables into the expectation value with respect to latent variables sampled from our distribution i,1:J i qϕ (z0:T , s0:T ): N X
N X
i,1:J i p(xi,1:J 0:T , z0:T , s0:T |θ) ] log p(xi,1:J log Eqϕ [ 0:T |θ) = i,1:J i , s0:T ) qϕi (z0:T i=1 i=1
(F10) By using Jenson’s Inequality log E[f (x)] >= E[log f (x)], we obtain:
XX qϕi qϕi 1
=
2
2
i,1:J i Eqϕi log qϕi (z0:T , s0:T ) (F15) h i XX i,1:J i,1:J i = log qϕi1 (z0:T )qϕi2 (s0:T ) qϕi1 (z0:T )qϕi2 (si0:T ) qϕi qϕi
i=1
1:J i p(xi,1:J 0:T , z0:T , s0:T |θ) i,1:J i qϕi (z0:T , s0:T )
XXh 2
>=
i=1
i
1
i,1:J qϕi1 (z0:T )qϕi2 (si0:T ) i h XX i,1:J i,1:J = log qϕi1 (z0:T )qϕi2 (si0:T ) )qϕi1 (z0:T
]
qϕi qϕi 2
+
1
XX
i,1:J log qϕi2 (si0:T ) qϕi1 (z0:T )qϕi2 (si0:T )
qϕi qϕi 2
=
1
X
i,1:J i,1:J ) log qϕi1 (z0:T ) qϕi1 (z0:T
qϕi
log p(xi,1:J 0:T |θ)
1
(F12)
+
i=1 N X
i,1:J log qϕi1 (z0:T ) + log qϕi2 (s0:T )i
qϕi qϕi
We separate the log in the right-most expression into a two term subtraction to end up with a joint term and an entropy term. N X
1
=
(F11) Eqϕi log[
i i,1:J i i,1:J log p(xi,1:J 0:T , z0:T , s0:T | θ) qϕi1 (z0:T )
We split the second term into two parts – one part that depends on qϕi2 (si0:T ) and another that depends on i,1:J qϕi2 (z0:T ).
2
>=
Eqϕi
1
i,1:J i p(xi,1:J 0:T , z0:T , s0:T |θ) log Eqϕi [ log p(xi,1:J ] 0:T |θ) = i,1:J i qϕ (z0:T , s0:T ) i=1 i=1 N X
h
X qϕi
N X
N X
(F14)
i,1:J i i,1:J i log p(xi,1:J 0:T , z0:T , s0:T | θ) q(z0:T ) qϕi2 (s0:T )
X
qϕi2 (si0:T ) log qϕi2 (si0:T )
qϕi
2
i,1:J i i,1:J i Eqϕi log p(xi,1:J ) 0:T , z0:T , s0:T |θ) − Eqϕi log qϕi (z0:T , s0:TWe
The term to the right is our per datapoint variational ELBO: N X
ELBO(qϕi , θ)
(F13)
i=1
=
N X
i,1:J i Eqϕi log p(xi,1:J 0:T , z0:T , s0:T |θ)−
i=1 i,1:J i Eqϕi log qϕi (z0:T , s0:T )
This is the training loss for the HSRDM. This loss is maximized via Coordinate Ascent Variational Inference (CAVI) updates [42], which we derive in the following section.
obtain the final result by using the rules that P he operator is linear and the summation over a probability density is constrained to be one P i e.g. (si ) qϕi2 (s0:T ) = 1. Now, we take the deriva0:T
i,1:J tive with respect to an individual qϕi2 (si0:T ) or qϕi1 (z0:T ) and set equal to zero to obtain the optimal value for each. We show the full example for the entity level upi,1:J date qϕi1 (z0:T ). First, we take the derivative with respect to
29 i,1:J qϕi1 (z0:T ):
4.
( N X
d ELBO(qϕi , θ)
d
= (F16) i,1:J i,1:J dqϕi1 (z0:T ) dqϕi1 (z0:T ) i=1 h i X i,1:J i i,1:J Eqϕi log p(xi,1:J 0:T , z0:T , s0:T | θ) qϕi1 (z0:T ) 2
CAVI update rule algorithms
VEZ and VES forward-backwards algorithm. Recall that we have some optimal qϕ∗ i functions that we can compute directly. i,1:J i,1:J i qϕ∗ i1 (z0:T ) ∝ exp Eqϕi [log p(xi,1:J 0:T , z0:T , s0:T |θ)] (F21) 2
qϕi
1
−
d
X
i,1:J dqϕi1 (z0:T ) qϕ i
i,1:J i,1:J qϕi1 (z0:T ) log qϕi1 (z0:T )
i,1:J i qϕ∗ i2 (si0:T ) ∝ exp Eqϕi [log p(xi,1:J 0:T , z0:T , s0:T |θ)] (F22) 1
1
) d X i i + qϕi2 (s0:T ) log qϕi2 (s0:T ) dqϕi1 q ϕi 2
=0 The last term ends up going to zero. For the other two terms, we need to take the derivative for each i,1:J component z = z ′ in the distribution qϕi1 (z0:T = z ′ ), z ′ ∈ RT ×J×K . The first term ends up being i,1:J i,1:J i ′ Eqϕi [log p(xi,1:J 0:T , z0:T , s0:T |θ)] for each qϕi1 (z0:T = z ).
To compute each function, we hold the other distribution fixed. We start with holding qϕ∗ i (si0:T ) fixed and take 2 the expectation with respect to that distribution. Our θ values are also held fixed. Then, we write the expression in the factorized form from above: i,1:J i exp Eqϕi log p(xi,1:J , z0:T , s0:T | θ) (F23) 0:T 2 ( " T Y = exp Eqϕi log p(si0 | θ) p(sit | sit−1 , xi,1:J t−1 , θ) 2
t=1
2
i,1:J The second term ends up being −(log(qϕi1 (z0:T ))+1) i,1:J ′ for all qϕi1 (z0:T = z ). We add in a third term as a Lagrange multiplier that imposes a normalization conP i,1:J straint on the objective: qϕ1 qϕi1 (z0:T ) = 1. We add P i,1:J zero to the function by adding λ( qϕ qϕi1 (z0:T ) − 1), 1 and the derivative of this is simply a constant λ. As such, we obtain the following proportion: i,1:J i,1:J i log qϕ∗ i1 (z0:T ) ∝ Eqϕi [log p(xi,1:J 0:T , z0:T , s0:T |θ)] (F17) 2
As such, we get the following expression for the optimal i,1:J qϕ∗ i (z0:T ): 1
×
J Y
i,j p(z0i,j | si0 , θ) p(xi,j 0 | z0 , θ)
j=1
×
T Y
!#) i,j i,j i,j i,j p(zti,j | sit , zt−1 , xi,j t−1 , θ) p(xt | zt , xt−1 , θ)
t=1 i,1:J To compute qϕ∗ i (z0:T ), we drop all of the terms that do 1
i,1:J ) as these will just be absorbed not depend on qϕ∗ i (z0:T 1 into the constant (hence the ∝ term). Now what we have left is:
exp Eqϕi {log[
J Y
2
i,1:J i,1:J i qϕ∗ i1 (z0:T ) ∝ exp Eqϕi [log p(xi,1:J 0:T , z0:T , s0:T |θ)] (F18)
i,j p(z0i,j |si0 , θ)p(xi,j 0 |z0 , θ)
(F24)
j=1
2
T Y
By the same set of rules, we obtain the expression for the optimal qϕ∗ i (si0:T ):
i,j i,j i,j i,j p(zti,j |sit , zt−1 , xi,j t−1 , θ)p(xt |zt , xt−1 , θ)]}
t=1
2
i,1:J i qϕ∗ i2 (si0:T ) ∝ exp Eqϕi [log p(xi,1:J 0:T , z0:T , s0:T |θ)] (F19) 1
These are the CAVI updates for the VEZ and VES steps in the three-step training algorithm. As for the M-step, we take the derivative of the ELBO with respect to θ. By doing this, we see that the second term of the ELBO, which does not depend on θ goes to zero, and only the first term is maximized. θ∗ = argmaxθ
N X
We move the exp Eqϕi log operations throughout the ex1 pression and obtain: J Y j=1 T Y t=1
i,1:J i Eqϕi log p(xi,1:J 0:T , z0:T , s0:T |θ)
(F20)
i=1
Each of these CAVI updates is computed separately. The VEZ and VES updates are computed via the forwardbackward inference algorithm outline in the following section. The M-step update is implemented either with gradient-ascent or with a closed-form solution if the probability distributions are assumed gaussian, also outlined in the following section.
i,j p(xi,j 0 |z0 , θ)
i,j i,j p(xi,j t |zt , xt−1 , θ)
L Y
i
p(z0i,j |si0 , θ)(s0 =l)
(F25)
l=1 L Y
i
i,j (st =l) p(zti,j |sit , zt−1 , xi,j t−1 , θ)
l=1
This expression comes from a few rules: (1) The expectation does not change a function that does not depend on qϕi2 , so we just have a exp log which does not change anything. (2) We can use the properties PN QN exp = and exp(x log(y)) = y x to obtain the QL i terms like l=1 p(z0i,j |si0 , θ)(s0 =l) . This functional form takes that of a discrete latent variable Hidden Markov Model for J independent entities, where we can use a forwards-backwards dynamic programming method to
30 compute the marginals and pairwise marginals for each i,j qϕ∗ i (z0:T ), which is all we need for downstream compu1 tations due to our Markov assumptions. So for each entity j, we have the following expression: i,j p(xi,j 0 |z0 , θ)
L Y
p(z0i,j |si0 , θ)
(si0 =l)
(F26)
l=1 T Y
i,j i,j p(xi,j t |zt , xt−1 , θ)
t=1
L Y
i
i,j (st =l) , xi,j p(zti,j |sit , zt−1 t−1 , θ)
i,j Now, βt (kj ) = p(xi,j = kj , st , θ). To obtain t+1:T |zt the marginal probability, we compute the following:
αt (kj )βt (kj ) p(zti,j = kj | xi,j 0:T , st , θ) = PK kj =1 αt (kj )βt (kj )
i,j p(xi,j | zti,j = kj , st , θ) p(xi,j 0:t , zt = kj | st , θ) = PK t+1:Ti,j i,j i,j i,j kj =1 p(xt+1:T | zt = kj , st , θ) p(x0:t , zt = kj | st , θ)
l=1
=
We break this into initial, transition, and emission probability distributions, where for each term where we QL compute l=1 , we can normalize to ensure that our distributions sum to one. For example: Initial:
(F28)
i,j (sit =l) , xi,j p(zti,j |sit , zt−1 t−1 , θ) Ψ̃t = PK Ql=1 L i,j i,j i i,j (sit =l) kj =1 l=1 p(zt = kj |st , zt−1 , xt−1 , θ)
QL
Here, kj indicates the current state p(zti,j = kj ) at time t, i,j and ki indicates the previous state p(zt−1 = ki ) at time K t − 1. π̃ ∈ (0, 1) represents the initial probability distribution of the latent variable, whereas Ψ̃t ∈ (0, 1)K×K represents the transition probability distribution for a current latent state given the previous latent state. The i,j i,j term ϕt = p(xi,j t |zt , xt−1 , θ) represents each emission probability density for an observation given the current latent state. Forwards-backwards inference allows us to compute the following forward probability via recursion for each state kj :
αt (kj ) = ϕt (zti,j = kj )
K X
i,j αt−1 (zt−1 = ki )
αt−1 (ki )Ψ̃t (ki , kj )ϕt (kj )βt (kj ) ki ,kj αt−1 (ki )Ψ̃t (ki , kj )ϕt (kj )βt (kj )
(F29)
We do these computations for all K states to end up with the optimal probability distributions q ∗ (zti,j ) and i,j q ∗ (zt−1 , zti,j ) for each entity, which is all we need for computing the other EM steps. Similar to this step, for computing q ∗ (sit ) and q ∗ (sit−1 , sit ), we only need q ∗ (zti,j ). As we show below, for the maximization step, we will only need the pairwise marginals for both. To compute the full forward-backward inference to obtain the optimal distributions for all entities, it takes O(K 2 T LJ) steps (K multiplications for T time steps per K states for each of the J entities). M-step optimization. Given the optimal variational distributions, we compute the following: N X
∗
θ = argmaxθ
We factor the data likelihood and spread out the terms in the log function: ( N X θ∗ = argmaxθ Eqϕi log p(si0 | θs−init ) (F36) i=1
(F30)
i,j Ψ̃t (zt−1 = ki , zti,j = kj )
+
T X
+
K X
ϕt (zti,j = kj )
kj =1 i,j Ψ̃t (zt−1 = ki , zti,j = kj )βt (zti,j = kj )
J h X
log p(z0i,j | si0 , θz−init )
j=1 i,j + log p(xi,j 0 | z0 , θx−init )
+ (F31)
log p(sit | sit−1 , xi,1:J t−1 , θss )
t=1
i,j Now, αt (kj ) = p(xi,j = kj |st , θ). To obtain the la0:t , zt tent marginal and pairwise marginals, we implement the backward step.
βt−1 (ki ) =
i,1:J i Eqϕi [log p(xi,1:J 0:T , z0:T , s0:T |θ)] (F35)
i=1
ki =1
βT (kj ) = 1
(F34)
=P
p(z0i,j |si0 , θ)(s0 =l) π̃0 = PK Ql=1 L i,j i (si0 =l) kj =1 l=1 p(z0 = kj |s0 , θ)
α0 (kj ) = π̃0 (z0i,j = kj )ϕ0 (z0i,j = kj )
p(xi,j 0:T | st , θ)
i,j = ki , zti,j = kj |xi,j p(zt−1 0:T , st , θ)
(F27)
Transition:
i,j p(xi,j 0:T , zt = kj | st , θ)
Now, for the pairwise marginal probability, we compute:
i
QL
(F33)
T X
i,j log p(zti,j | sit , zt−1 , xi,j t−1 , θes )
t=1 i,j i,j + log p(xi,j t | zt , xt−1 , θee )
) i
(F32) We take the derivative with respect to each distinct θ independently for each distribution. When we assume gaussian distributions, we obtain closed-form solutions. Otherwise, we implement gradient-ascent.
31 G. 1.
ML training results
Seed Entity
S1
T0
T1
17
0 1 2 3
-0.157 0.323 -0.049 -0.028 -0.130 0.295 -0.055 -0.019 -0.138 0.409 -0.107 -0.051 -0.211 0.478 -0.108 -0.041
41
0 1 2 3
0.182 -0.093 -0.028 -0.049 0.172 -0.087 -0.019 -0.055 0.165 -0.021 -0.051 -0.107 0.200 -0.055 -0.041 -0.108
76
0 1 2 3
0.010 0.075 -0.041 -0.037 0.028 0.060 -0.039 -0.041 0.007 0.214 -0.081 -0.081 -0.030 0.217 -0.075 -0.082
126
0 1 2 3
-0.157 0.323 -0.049 -0.028 -0.130 0.295 -0.055 -0.019 -0.138 0.409 -0.107 -0.051 -0.211 0.478 -0.108 -0.041
166
0 1 2 3
0.182 -0.093 -0.028 -0.049 0.172 -0.087 -0.019 -0.055 0.165 -0.021 -0.051 -0.107 0.200 -0.055 -0.041 -0.108
K-means initializations
We include the adapted-HSRDM training results across different random k-means initializations. We used the k-means algorithm as implemented in scikit-learn [50] to cluster the EmbeddingGemma vectors into K=4 groups. Then, using only the data assigned to each cluster, we fit an autoregressive linear model over time to estimate the Ak , bk , Qk parameters for that cluster. Immediately following this initialization, we clamp the parameters related to silent entity states S0 or S1 to a silent embedding mean and a 0.001 variance. Thus, the difference between the informed initialization and k-means is in the parameters for states T0, T1. Only the informed settings contain prior knowledge of which states align with utterances containing evidence of mechanistic reasoning. K-means does rely on a random seed, so for that procedure we keep the best run of 5 seeds. We define “best” as the highest speaker correlation metric results on the training set. While not relevant for model selection, we report all metric results for the model’s performance on training data with the informed initialization under various ablations (e.g. removing the classifier feedback mechanism). We include the results across seeds for investigating all four hypotheses regarding the desired model behavior: H(i) results in Table XXI, H(ii) results in Table XXII, H(iii) non-speaker to non-speaker results in Table XXIII, and H(iii) non-speaker to speaker results XXIV. We selected best model for test based on the seed with the highest speaker correlation between the human annotations (seed 17) and the S1 state (the bold column in Table).
S0
TABLE XXI: H(i): Pearson r correlation experiments on the training data with k-means initializations. For each k-means random initialization, we examine turns when an entity speaks, and assess correlation between the posterior probability mass of the next state (columns) and the humanannotated evidence strength present in the utterance. For good models, the desired behavior is a strong positive correlation for state S1 (bold). We observe the correlation decrease as we ablate inductive bias components in the model architecture.
Mean Std. Dev Seed Evidence Posterior Posterior 17
Yes No
0.796 0.438
0.159 0.224
41
Yes No
0.213 0.095
0.214 0.144
76
Yes No
0.430 0.166
0.194 0.173
126
Yes No
0.796 0.438
0.159 0.224
166
Yes No
0.213 0.095
0.214 0.144
TABLE XXII: H(ii): Mean probability of entity state S1, for previous speaker turns with and without evidence of MR on the training data with k-means. For all k-mean initializations, we report the mean and standard deviation of the previous speaker’s posterior probability mass assigned to state S1, conditioned on whether their utterance contains evidence of mechanistic reasoning. We expect for this state to have a larger probability when evidence is present than not.
32 Mean Std. Dev Seed Evidence Posterior Posterior 17
Yes No
0.362 0.363
0.038 0.038
41
Yes No
0.846 0.832
0.215 0.225
76
Yes No
0.633 0.626
0.123 0.127
126
Yes No
0.362 0.363
0.038 0.038
166
Yes No
0.846 0.832
0.215 0.225
TABLE XXIII: H(iii): Mean probability of entity state S1, for silent-to-silent turns with and without evidence of MR, on the training data with k-means. For all k-mean initializations, we report the mean and standard deviation of the current non-speaker’s posterior probability mass assigned to state S1, conditioned on whether the speaker’s previous utterance contains evidence of mechanistic reasoning. We expect for this state to have noticeably higher probability when evidence is present than not. Mean Std. Dev Seed Evidence Posterior Posterior 17
Yes No
0.255 0.163
0.435 0.369
41
Yes No
0.747 0.837
0.434 0.369
76
Yes No
0.522 0.539
0.494 0.495
126
Yes No
0.255 0.163
0.435 0.369
166
Yes No
0.747 0.837
0.434 0.369
TABLE XXIV: H(iii): Mean probability of entity state T1, for silent-to-speaker turns with and without evidence of MR, on the training data with k-means. For all k-mean initializations, we report the mean and standard deviation of the current speaker’s posterior probability mass assigned to state T1, conditioned on whether the speaker’s previous utterance contains evidence of mechanistic reasoning. We expect for this state to have noticeably higher probability when evidence is present than not. 2.
Model ablation study with training data
We report adapted-HSRDM training results across multiple ablations using the informed initialization from Sec. III E, and compare them to the best-performing k-means model. We include the following metrics: H(i) results in Table XXV, H(ii) results in Table XXVI, H(iii) silent to silent results in Table XXVII, and H(iii) silent to speaker results XXVIII.
Init + Feedback
Entity
S0
S1
T0
T1
K-means + human
0 1 2 3
0.197 -0.126 -0.042 -0.036 0.177 -0.109 -0.048 -0.030 0.194 -0.076 -0.098 -0.062 0.267 -0.142 -0.098 -0.056
Informed + Human
0 1 2 3
-0.188 0.477 -0.049 -0.027 -0.137 0.414 -0.056 -0.018 -0.145 0.618 -0.115 -0.038 -0.235 0.654 -0.101 -0.053
K-means + Classifier
0 1 2 3
-0.157 0.323 -0.049 -0.028 -0.130 0.295 -0.055 -0.019 -0.138 0.409 -0.107 -0.051 -0.211 0.478 -0.108 -0.041
Informed + Classifier
0 1 2 3
-0.180 0.346 -0.049 -0.027 -0.139 0.310 -0.056 -0.018 -0.151 0.419 -0.115 -0.040 -0.232 0.491 -0.100 -0.053
Informed + No System
0 1 2 3
-0.125 0.255 -0.050 -0.026 -0.108 0.222 -0.056 -0.017 -0.090 0.249 -0.117 -0.036 -0.165 0.348 -0.102 -0.051
Informed + None
0 1 2 3
0.059 0.060 0.131 0.123
0.057 -0.049 -0.027 0.060 -0.056 -0.018 0.124 -0.115 -0.039 0.117 -0.100 -0.053
TABLE XXV: H(i): Pearson r correlation experiments on the training data. For all model ablations on training data, we examine turns when an entity speaks, and assess correlation between the posterior probability mass of the next state (columns) and the human-annotated evidence strength present in the utterance. For good models, the desired behavior is a strong positive correlation for state S1 (bold). We observe the correlation decrease as we ablate inductive bias components in the model architecture.
33
Init + Feedback
Mean Std. Dev Evidence Posterior Posterior
Mean Std. Dev Evidence Posterior Posterior
Init + Feedback
K-means + Human
Yes No
0.043 0.009
0.068 0.016
K-means + Human
Yes No
0.386 0.295
0.485 0.455
Informed + Human
Yes No
0.745 0.092
0.224 0.103
Informed + Human
Yes No
0.340 0.154
0.473 0.359
K-means + Classifier
Yes No
0.796 0.438
0.159 0.224
K-means + Classifier
Yes No
0.255 0.163
0.435 0.369
Informed + Classifier
Yes No
0.881 0.395
0.210 0.257
Informed + Classifier
Yes No
0.340 0.160
0.472 0.364
Informed + No System
Yes No
0.832 0.761
0.220 0.262
Informed + No System
Yes No
0.328 0.141
0.468 0.346
Informed + None
Yes No
0.496 0.501
0.024 0.022
Informed + None
Yes No
0.334 0.159
0.470 0.364
Init + Feedback
Mean Std. Dev Evidence Posterior Posterior
K-means + Human
Yes No
0.767 0.747
0.292 0.303
Informed + Human
Yes No
0.357 0.340
0.074 0.071
K-means + Classifier
Yes No
0.362 0.363
0.038 0.038
Informed + Classifier
Yes No
0.366 0.356
0.083 0.061
Informed + No System
Yes No
0.082 0.092
0.181 0.189
Informed + None
Yes No
0.498 0.498
0.023 0.021
TABLE XXVII: H(iii): Mean probability of entity state S1, for silent-to-silent turns with and without evidence of MR, on the training data. For all model ablations, we report the mean and standard deviation of the current non-speaker’s posterior probability mass assigned to state S1, conditioned on whether the speaker’s previous utterance contains evidence of mechanistic reasoning. We expect for this state to have noticeably higher probability when evidence is present than not.
TABLE XXVIII: H(iii): Mean probability of entity state T1, for silent-to-speaker turns with and without evidence of MR, on the training data For all model ablations, we report the mean and standard deviation of the current speaker’s posterior probability mass assigned to state T1, conditioned on whether the previous speaker’s utterance contains evidence of mechanistic reasoning. We expect for this state to have noticeably higher probability when evidence is present than not. 3.
ELBO results
We include the ELBO training plot in Fig. 5 for the best seed 17 based on the speaker correlation metric for the k-means initialization. We observe the ELBO increase over the 15 CAVI iterations.
35000
+3.6874e9
ELBO Over Iterations
0
4
30000
ELBO
TABLE XXVI: H(ii): Mean probability of entity state S1, for turns with and without evidence of MR, on training data. For all model ablations, we report the mean and standard deviation of the previous speaker’s posterior probability mass assigned to state S1, conditioned on whether their utterance contains evidence of mechanistic reasoning. We expect for this state to have a larger probability when evidence is present than not.
25000 20000 15000 10000 2
6
8
Iteration
10
12
14
FIG. 5: The ELBO training curve over 15 CAVI iterations.