Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory Peng Cui∗1 , Heejin Do∗2 , Mrinmaya Sachan1 ETH Zürich Department of Computer Science1 , ETH AI Center2 {peng.cui, mrinmaya.sachan}@inf.ethz.ch1 [email protected]
Conventional
Abstract
arXiv:2609.05245v1 [cs.AI] 4 Sep 2026
Human knowledge is inherently structured and interdependent: mastery of a concept requires prior mastery of its prerequisites, a principle formalized by Knowledge Space Theory (KST). While LLMs achieve strong performance on complex reasoning tasks, it remains unclear whether they exhibit coherent, human-like knowledge structure. We introduce a KST-grounded framework for evaluating LLM knowledge structure in mathematical reasoning, using it as a normative framework to analyze whether LLM behavior adheres to principled knowledge dependencies. Evaluating eight open- and closed-source LLMs against real human learners, we find that (1) LLMs do not adhere to human knowledge structure—they frequently violate knowledge dependencies and fail to leverage related knowledge provided in context to improve performance on dependent questions; (2) LLMs do not share a consistent knowledge structure among themselves, as reflected by low overlap in their knowledge distributions. Furthermore, these structural deficiencies remain largely invisible to accuracy-based and LLM-as-judge evaluations. Together, our results provide behavioral evidence that current LLMs knowledge does not follow a human-like structure. https://github.com/pengcuix/LLM-KST
1
Introduction
Recent advances in large language models (LLMs) have led to remarkable performance on a wide range of reasoning benchmarks. Yet high accuracy alone does not imply genuine understanding. A growing body of evidence shows that models can arrive at correct answers through flawed, short-cutbased, or unfaithful reasoning processes (Lanham et al., 2023; Turpin et al., 2023). In response, recent work has shifted from evaluating final answers *: Equal contribution
✓
✗
✓
Humans / LLMs
✓
✗
Average accuracy=0.6
KST-grounded
X …
Advanced knowledge
✓
✓
Entail?
Test questions
✗
✓
✗
Scaffold?
Prerequisite knowledge
Figure 1: Conventional evaluation measures average accuracy, treating knowledge as a flat unstructured collection. We propose a KST-based framework that models structured dependencies in knowledge, examining whether LLM exhibits coherent knowledge behavior.
to evaluating reasoning trajectories themselves, introducing process-based benchmarks (Lightman et al., 2023; Uesato et al., 2022; Xia et al., 2025; Do et al., 2025). However, these evaluations remain inherently local: they assess each problem in isolation and cannot reveal whether a model’s success and failure patterns are globally consistent with the structure of the knowledge being tested. In contrast, established theories of human learning emphasize that knowledge is inherently structured. Knowledge Space Theory (KST) (Doignon and Falmagne, 2012) formalizes this principle by modeling a knowledge domain as a set of latent concepts connected through prerequisite dependencies. Under this framework, mastery of a concept presupposes mastery of its prerequisites — for instance, solving quadratic equations requires prior mastery of linear equations and algebraic manipulation. These dependency relations constrain what constitutes a valid knowledge state (i.e., which concepts a learner has mastered) and define coherent learning pathways through the domain. In this work, we adopt KST as a normative framework for evaluating the behavioral consistency of
LLMs. We ground our study in mathematics, where prerequisite relations are well-defined and extensively documented through expert-curated educational standards. As illustrated in Figure 1, rather than the conventional approach of treating knowledge as a flat collection of independent items and measuring individual or aggregate accuracy, our framework analyzes whether the knowledge structures underlying observed LLM response patterns adhere to principled dependencies. Specifically, we aim to answer two research questions: • RQ1: Do LLMs adhere to human knowledge dependencies? • RQ2: If not, do LLMs share a consistent and coherent knowledge structure among themselves? We evaluate a broad range of open- and closedsource LLMs alongside human learners on a dataset with real student response records. Our results show that: (1) despite a moderate accuracy of 79.6%, human learners satisfy prerequisite dependencies for the majority (72.7%) of their correct answers. LLMs also fail to leverage prerequisite knowledge to scaffold dependent questions, further suggesting a lack of human-like knowledge structure. (2) Among human learners, stronger learners’ knowledge states consistently subsume those of weaker ones, reflecting ordered knowledge growth. LLMs, in contrast, show significantly lower subsumption across performance levels, suggesting that LLMs do not share a consistent and coherent knowledge structure, and that their knowledge acquisition is more likely flat than structured. In summary, our contributions are as follows: • We propose a novel KST-grounded analytical framework for evaluating the coherence of LLM knowledge, providing a principled complement to accuracy-based evaluation. • Our analysis reveals systematic incoherence in LLM mathematical knowledge structure, offering a new perspective and empirical evidence that current LLMs may not engage in genuine formal reasoning. • We construct a dataset with concept and dependency annotations, providing a resource for future research on LLM knowledge structure.
2
Related Work
Knowledge Space Theory (KST) is a theoretical framework for modeling the structure of knowledge and learning introduced by Doignon and Falmagne (1985, 2012) in the 1980s. The central idea of KST is that learners’ knowledge is not an arbitrary collection of isolated facts, but rather forms a structured space constrained by prerequisite relations among concepts or skills. In this framework, each learner is associated with a knowledge state representing the subset of problems or concepts they have mastered, while the set of all feasible states forms a knowledge space. KST further models learning as transitions between knowledge states, thereby providing a principled representation of hierarchical and cumulative learning processes. Subsequent work extended KST from the deterministic framework to probabilistic models to account for response noise (Falmagne and Doignon, 1988; De Chiusole et al., 2024), introducing parameters such as lucky guesses and careless errors to model the stochastic nature of real-world assessment data, enabling KST to be applied to largescale empirical settings. Over the past decades, KST has been widely applied in educational assessment (Falmagne et al., 2013), intelligent tutoring systems (Nkambou et al., 2010), and adaptive learning environments (Falmagne and Doignon, 2010). Early systems such as ALEKS demonstrated the practical utility of KST for personalized assessment and curriculum sequencing (Cosyn et al., 2021; Cui and Sachan, 2023). LLM Reasoning Evaluation Recent efforts to evaluate the capabilities of LLMs have shifted from simple answer-correctness metrics to the scrutiny of reasoning trajectories that lead to the answer. Driven by the observation that correct final answers can emerge from flawed or unfaithful reasoning chains (Lanham et al., 2023; Turpin et al., 2023), the focus has moved toward assessing the validity of intermediate steps through process-based reward models (PRMs) (Lightman et al., 2023; Uesato et al., 2022) and targeted reasoning benchmarks (Cobbe et al., 2021; Zeng et al., 2024). Further attempts have dissected reasoning quality into multiple dimensions (Xia et al., 2025; Do et al., 2025), providing granular signals for model improvement. However, these trajectory-based evaluations remain inherently limited by their focus on local, isolated problem-solving; as such, they fail to capture whether a model’s behavior is consistent
with the latent hierarchical structure of knowledge. In human cognition, mastery is not a collection of isolated successes but a consistent epistemic state where complex concepts are built upon foundational prerequisites (Doignon and Falmagne, 2012). By shifting the evaluative focus from superficial texts to structural dependencies, this work introduces a principled framework to detect these hidden inconsistencies, offering a more global and robust measure of model reliability.
states forms a knowledge space K. In this work, we focus on mathematical reasoning, where knowledge is highly structured and interdependent, making it an ideal testbed for our framework. Many expert-defined mathematical standards with explicit prerequisite dependencies exist, such as the Common Core State Standards (Association et al., 2010). We use the New York State Mathematics Learning Standards1 as our C, along with their defined dependency relations among concepts.
3
3.2
Framework
Overview. In this section, we propose a normative framework grounded in Knowledge Space Theory (KST) to analyze whether the knowledge structure of LLMs is coherent. We begin by introducing the foundational definitions and concepts of KST (§3.1), followed by a description of how we adapt this framework to LLMs under the mathematical reasoning setting (§3.2). Building on this, we define a list of normative behaviors characterizing the ideal knowledge state and dynamics in LLMs, along with metrics to quantitatively measure the degree to which LLMs conform to these behaviors (§3.3). Together, these provide a principled basis for identifying incoherent aspects of LLM knowledge structure that would otherwise remain obscured by standard accuracy-based evaluation. See Figure 2 for an overview. 3.1
Background of KST
KST is a mathematical framework for modeling the structure of human knowledge. It represents the domain of knowledge as a finite set of knowledge concepts C = {c1 , . . . , cm }, where each concept is a fundamental unit of knowledge, serving as a building block for understanding and reasoning within a domain, for example, quadratic equations in math. A central assumption of KST is that these concepts are not independent, i.e., mastery of some concepts is required before others can be acquired. This is formalized as a prerequisite relation, a partial order ⪯ on C, where ci ⪯ cj denotes that ci is a prerequisite of cj . A knowledge state K ⊆ C is the set of concepts a learner has mastered at a given point. Note that a valid knowledge state must be closed under prerequisites: if cj ∈ K and ci ⪯ cj , then ci ∈ K. This closure property ensures that a learner cannot have mastered a concept without having mastered its prerequisites. The collection of all valid knowledge
Knowledge Space Construction
Since a learner’s mastery of knowledge concepts is difficult to estimate, we operationalize KST at the question level, where performance on each question can be directly observed and evaluated. Let Q = {q1 , q2 , . . . , qn } be a set of mathematical questions, where each question q ∈ Q is associated with one or more concepts in C, denoted as C(q). However, the dependencies among questions Q are unknown. Therefore, we first infer question dependencies from their associated concepts. LLM-based concept annotation Specifically, we first prompt an LLM to identify the relevant concepts from C for each question. Prior work has demonstrated that advanced LLMs can accurately identify required skills (equivalent to concepts) from question text, particularly in mathematical domains (Didolkar et al., 2024; Li et al., 2024; Shah et al., 2024). Unlike prior approaches, which rely on free-form concept generation, we prompt the LLM with the question together with a predefined concept list — the NYS standards — and instruct it to select relevant concepts only from this list. The NYS standards are organized hierarchically into 63 domains, 148 clusters, and 480 concepts, where each domain is a group of related clusters, and each cluster is a group of concepts. Supplying all 480 concepts and their descriptions in a single prompt would result in an unnecessarily long context, which increases cost and makes accurate selection more difficult. We therefore adopt a two-stage prompting procedure: the LLM first selects the relevant clusters, and is then prompted to choose the final concepts from those belonging to the selected clusters. Our prompt template for concept annotation is in Prompt 2. Deriving question dependencies Grounded on concept-level dependencies, we define the prerequi1
NYS Mathematics Learning Standards
(a) Concept Annotation
(b) Dependency Induction Concept Graph
Concept set C Solve:1/3 + 1/4 = ?
3.NF.A.1 4.NF.A.1 5.NF.A.1
C(q) ⊆ C 5.NF.A.1
: Direct : Indirect
4.NF.A.1
4.NF.B.3
Equivalent fractions
Add like fractions
3.NF.A.1
3.NF.A.2
Understand a/b
Fraction on line
4.NF.A.1
prerequisites
Add like fractions
3.NF.A.1 Understand a/b
Examples
q1
Equivalent fractions
q2
q3
q4
q5
q1: 1/3 + 1/4 = ? q2: 1/3 = ?/12 q3: 2/5 + 1/5 = ? q4: What is 3/4? q5: Place 1/2 on number line
Violation
q1
q2 Low
Question Graph
Add unlike fractions
4.NF.B.3
Incorrect
High
…
Concept Tagger (LLM)
Target q
Correct
5.NF.A.1 Add unlike fractions
ANALYSIS
Question q
(c) Behavioral Analysis of LLMs
q4
q3 q5
Knowledge Coherence Knowledge Scaffolding Knowledge Progression
Figure 2: (a) Concept Annotation: For each question, we prompt an advanced LLM to identify the relevant mathematical concepts from an expert-curated concept map. (b) Dependency Induction: We use the questionconcept associations together with expert-defined concept dependencies to infer prerequisite relations between questions. (c) Behavioral Analysis of LLMs: We assess whether the knowledge structure and dynamics of LLMs are coherent with respect to the constructed knowledge space through a list of normative behaviors.
site relation between questions as follows: question qi is a prerequisite of question qj if and only if ∀ ca ∈ C(qi ), ∃ cb ∈ C(qj ) such that ca ⪯ cb . (1) We intentionally adopt this strict definition, rather than weaker alternatives such as requiring only a single concept pair to satisfy the prerequisite relation. This ensures that the induced question-level dependencies are precise and conservative, reducing the risk of spurious prerequisites that could confound our analysis. For each question q, we denote its prerequisite questions as Pre(q), which is a subset of Q. For a learner l (either an LLM or a human learner), we define their knowledge state Kl as a subset of questions Q that they answer correctly. 3.3
Behavioral Analysis of LLMs via KST
In this section, we use the KST framework to analyze whether LLMs follow a coherent knowledge structure. Specifically, we examine (RQ1) whether LLMs adhere to human knowledge dependencies, and (RQ2) if not, whether LLMs share a consistent knowledge structure among themselves. We investigate these two questions through three normative behaviors (NBs) that a reliable reasoning model should exhibit. 3.3.1 Do LLMs Adhere to Human Knowledge Dependencies? (RQ1) Should human knowledge structure apply to LLMs? Although the prerequisite dependencies
among concepts in C are derived from human knowledge standards, we argue that these dependencies should, in principle, apply to any reliable reasoning system. Mathematical reasoning follows strict logical procedures that are intrinsic to mathematics itself, not artifacts of human cognition. For example, an LLM that fails at 3 + 3 (Addition) yet succeeds at 3 × 3 (Multiplication) is unlikely to arrive at the latter through genuine reasoning, and instead may rely on memorization or superficial pattern matching. Therefore, LLMs’ conformity to knowledge dependencies could serve as a signal of genuine logical reasoning. NB1: Mastery of a question q should entail mastery of its prerequisite questions Pre(q).
NB1 is grounded in the foundational axiom of KST: a valid knowledge state must be closed under prerequisite relations, that is, if a learner has mastered an item, they must have also mastered all of its prerequisites. We measure the extent to which an LLM follows NB1 using the Prerequisite Satisfaction Ratio (PSR). Given a learner’s knowledge state Kl , the PSR for a single question q is defined as: PSR(q, Kl ) =
|Pre(q) ∩ Kl | , |Pre(q)|
(2)
which is the proportion of q’s prerequisite questions (Pre(q)) that the learner has also answered correctly. Note that PSR is computed only on correctly answered questions.
Under NB1, this metric should be ≈ 1 for all correctly answered questions. In practice, however, exceptions can occur. For human learners, this may be attributed to a lucky guess on the target question, a careless slip on one or more prerequisite questions, or a prerequisite that is substantially more difficult than the target itself. Nevertheless, the overall PSR is expected to remain high. For LLMs, however, such violations probably suggest that correct answers may not be grounded in the requisite knowledge structure, but rather obtained through some surface-level pattern matching. We aggregate the PSR of all questions q ∈ Kl in either a macro- or micro-level manner: 1 X PSR(q), |Kl |
PSRmacro (Kl ) =
(3)
q∈Kl
P PSRmicro (Kl ) =
q∈Kl |Pre(q) ∩ Kl |
P
q∈Kl |Pre(q)|
,
(4)
where the macro-level PSR simply averages perquestion PSR scores, and the micro-level PSR is a weighted average that accounts for the number of prerequisites per question. NB2: For a question q with concept c, knowledge of c or c’s prerequisite concepts Pre(c) should improve a learner’s performance on q .
NB2 captures the functional role of relevant knowledge in scaffolding the acquisition of dependent or related concepts — a principle central to both KST and constructivist theories of learning (Narayan et al., 2013). If LLMs internalize knowledge units and their relationships similarly to humans, we would expect to observe a scaffolding effect in LLMs as well. We operationalize this for LLMs with In-Context Learning (ICL), where we provide questions and solutions of (1) prerequisite concepts or (2) the +pre same concept as in-context examples. Let Kl denote the knowledge state of the model when, for each question q ∈ Q, relevant examples are provided in context. We define Scaffolding Gain (SG) of a model l as: SG(Kl ) =
1 +pre (|Kl | − |Kl |), |Q|
(5)
which reflects the change in accuracy when relevant knowledge is provided as in-context scaffolding.
3.3.2
Do LLMs Share a Coherent Knowledge Structure? (RQ2)
Although human knowledge structure should in principle apply to LLMs, it is also possible that LLMs develop their own knowledge organization that is shared and consistent across models. If so, we would expect to observe the following behavior: NB3: The knowledge state of a more capable model should largely subsume that of a less capable model.
NB3 reflects the cumulative and hierarchical nature of mathematical knowledge. Unlike factual knowledge, which can often be acquired independently and in a fragmented manner, mathematical concepts are connected through dense prerequisite dependencies: advanced concepts build upon foundational ones and therefore require mastery of prior knowledge. Therefore, if LLMs possess a coherent and consistent knowledge structure of their own — even if it differs from that of humans — the knowledge state of a less capable model should still be largely subsumed by that of a more capable one, since greater competence presupposes mastery of the same underlying foundations. In practice, the extent of such subsumption is expected to increase with the density of prerequisite dependencies in the domain. In tightly interconnected knowledge structures, there are fewer independent pathways to mastery, making coherent and nested knowledge states more likely to emerge. We use the Knowledge Overlap Coefficient (KOC) to quantify the degree of subsumption, defined as: KOC(Kl1 , Kl2 ) =
|Kl1 ∩ Kl2 | , Min(|Kl1 |, |Kl2 |)
(6)
where KOC ∈ [0, 1]; a value of 0 indicates no overlap, while a value of 1 indicates that the weaker model’s knowledge is fully subsumed by that of the stronger model. Under NB3, we would ideally expect KOC ≈ 1. Note that this metric focuses solely on the alignment of question-level performance distributions between models, without making any assumptions about knowledge concepts or their dependencies. However, since all Kl are defined over a shared question set Q, KOC is inflated by chance overlap — the expected KOC between two randomly drawn knowledge states equals the accuracy of the
max(|K |,|K |)
l1 l2 stronger learner . We therefore com|Q| pute a normalized variant:
KOC(Kl1 , Kl2 ) − pmax , 1 − pmax (7) where pmax = max(|K1 |, |K2 |)/|Q| is the accuracy of the stronger learner. Under this normalization, a value of 0 indicates overlap consistent with chance, values greater than 0 indicate systematic subsumption beyond chance, and a value of 1 indicates perfect subsumption. KOCnorm (Kl1 , Kl2 ) =
4
Experimental Setup
Datasets We conduct experiments on the mathematical knowledge tracing dataset XES3G5M (Liu et al., 2023)2 , using its English translated version publicly available by Seo et al. (2026)3 . Unlike standard math benchmarks, this dataset is derived from real student problem-solving logs and includes ground-truth correctness labels, enabling direct comparison between model predictions and human performance in terms of knowledge structure4 . The dataset is used in compliance with the MIT License and its intended use. As our focus is on textual knowledge, we exclude samples containing images and remove duplicate instances, resulting in 3,103 fill-in-the-blank questions (FITB) and 1,015 multiple-choice questions (MCQ). The statistics of the data are summarized in Table 1. Interactions
# Students # Questions
18, 066 4, 118
Concepts
# Concepts # Concepts / question (avg.)
480 1.07
Dependency
# Prerequisites / concept (avg.) # Prerequisites / question (avg.)
1.58 16.4
Table 1: Statistics of the XES3G5M dataset and extracted concepts.
LLMs and inference setup We evaluate a broad set of large language models spanning multiple families and scales. This includes strong closedsource models C LAUDE S ONNET 4.6, GPT-4.1MINI (Achiam et al., 2023) and open-source models including M ISTRAL -7B-I NSTRUCT- V 0.3 (Jiang et al., 2023), L LAMA -3.1-8B-I NSTRUCT, 2
https://github.com/ai4ed/XES3G5M https://github.com/sjin4861/BAIM 4 To the best of our knowledge, this is the only publicly available mathematics dataset that contains both real student response records and full question content. 3
L LAMA -3.1-70B-I NSTRUCT (Grattafiori et al., 2024), Q WEN 2.5-7B-I NSTRUCT, Q WEN 2.532B-I NSTRUCT (Qwen et al., 2025), and Q WEN 3-N EXT-80B-A3B-I NSTRUCT (Team, 2025). For open-source models, we perform inference using vLLM (Kwon et al., 2023) on NVIDIA GH200 GPUs with temperature 0.6, top-p = 0.95, and random seed 42. Closed-source models (Claude Sonnet 4.6 and GPT-4.1-mini) are accessed through their official APIs with default chat templates. We use a context window of 16,384 tokens for the no-context baseline and 32,768 tokens for all scaffolding-based evaluations. All experiments are conducted using the FuseAI framework (Wan et al., 2024)5 , with its default chain-ofthought (CoT; Wei et al. (2022)) prompting templates (Prompt 1) applied consistently across all models. Concept annotation The prompt template for concept annotation is in Prompt 2. We use GPT4.1- MINI for concept extraction with a temperature of 0.2. Since LLM annotations may contain errors, we sampled 400 questions for manual verification in order to quantify the error rate and its potential impact on our results. Of these, 317 were confirmed to be correctly annotated, giving an accuracy of 79.3%. We denote the fully automatically annotated dataset as XESfull and the verified subset as XESverified . We additionally verified annotation accuracy on the exemplar questions from the NYS standards; details are given in Appendix B. Comparison against human learners Our evaluation measures the extent to which LLMs conform to or deviate from the three normative behaviors using the corresponding metrics proposed in Section §3.3. As discussed, these behaviors represent idealized conditions, but exceptions can happen in practice. To provide a meaningful reference point, we compare the conformance of both humans and LLMs with respect to each normative behavior, allowing us to quantify the gap, or potential advantage, between LLMs and human minds as a reliable reasoning system. In-context scaffolding setup. For the scaffolding experiments (NB2), we provide each model with k=3 ICL exemplars selected under five strategies: (1) No context (baseline); (2) Random: three random questions of the same item type 5
https://github.com/fanqiwan/FuseAI
Model
Acc.
M ISTRAL -7B- V 0.3 0.217 LLAMA-3.1-8B-I NSTRUCT 0.559 LLAMA-3.1-70B-I NSTRUCT 0.716 Q WEN 2.5-7B-I NSTRUCT 0.771 Q WEN 2.5-32B-I NSTRUCT 0.849 Q WEN 3-80B-I NSTRUCT 0.925
Avg. PSR Question-level PSR distribution Micro Macro [0, 0.2) [0.2, 0.4) [0.4, 0.6) [0.6, 0.8) [0.8, 1.0) = 1.0 0.299 0.638 0.790 0.828 0.862 0.939
0.353 0.618 0.748 0.820 0.817 0.925
30.25% 7.21% 5.23% 2.12% 3.69% 1.34%
30.25% 4.68% 2.10% 0.74% 0.95% 0.05%
22.22% 29.09% 9.20% 5.33% 5.86% 1.49%
8.23% 37.53% 32.17% 25.83% 19.74% 4.67%
0.82% 8.83% 30.49% 38.66% 44.33% 44.29%
8.23% 12.66% 20.81% 27.32% 25.44% 48.16%
C LAUDE -S ONNET-4-6 GPT-4.1- MINI
0.840 0.849 0.850 0.862
0.846 0.832
2.07% 3.23%
0.27% 0.37%
3.37% 4.55%
22.20% 20.36%
41.02% 43.52%
31.07% 27.97%
Human Learners
0.796 0.936
0.942
<0.01%
0.68%
0.44%
3.15%
23.7%
72.7%
Table 2: The accuracy and PSR results for all LLMs and human learners on all questions annotated by (XESfull ). For question-level PSR (Eq. 2), we report the proportion falling within different intervals. For aggregate PSR, we report both micro (Eq. 4) and macro (Eq. 3) results. For Acc, the proportion of questions with PSR = 1.0, and averaged PSR, we bold the best overall result and underline the best result among LLMs.
(FITB/MCQ); (3) Same-skill: three questions sharing at least one concept with the target; (4) Similarity: top-3 questions retrieved by BGE-M3 (Chen et al., 2024) embedding similarity; (5) Prerequisite: three randomly sampled questions from the prerequisite set Pre(q). We use all questions in XESfull for this experiment. For a fair comparison, all five conditions are evaluated on the common subset of questions where both prerequisite-based and concept-based selection yield ≥ 3 eligible exemplars (1,184 FITB and 233 MCQ questions). The prompt template is shown in Prompt 3.
5
Results
5.1
Prerequisite Satisfaction (NB1)
Results on XESfull . We present the results in Table 2. The left part compares the accuracy and average PSR of LLMs and human learners. Overall, average PSR tends to increase with accuracy, which is expected because PSR approximates the conditional probability of correctly answering prerequisite questions given that their advanced question is solved, and is therefore positively correlated with overall correctness probability. Nevertheless, human learners achieve a high PSR of 0.936/0.942 at a relatively modest accuracy of 0.796. In contrast, Q WEN 2.5-32B-I NSTRUCT, despite outperforming human learners in accuracy, obtains substantially lower PSR scores. Only Q WEN 3-80B-I NSTRUCT, with a notably higher accuracy of 0.925, slightly surpasses human learners in micro-PSR but remaining lower in macro-PSR. These results suggest that even when LLMs match or exceed human accuracy, their response patterns generally remain less consistent with the prerequisite structure.
Model
Acc.
M ISTRAL -7B- V 0.3 0.217 LLAMA-3.1-8B-I NST. 0.560 LLAMA-3.1-70B-I NST. 0.717 Q WEN 2.5-7B-I NST. 0.771 Q WEN 2.5-32B-I NST. 0.849 Q WEN 3-80B-I NST. 0.925
PSR Micro Macro =1.0 0.325 0.616 0.826 0.822 0.848 0.946
0.387 0.614 0.789 0.834 0.802 0.932
0% 5.63% 32.1% 21.3% 22.1% 55.7%
C LAUDE - SONNET-4-6 GPT-4.1- MINI
0.841 0.863 0.849 0.852
0.858 0.822
28.1% 21.5%
Human Learners
0.796 0.950
0.951 81.6%
Table 3: Accuracy and PSR results on the verified subset XESverified . We bold the best overall result and underline the best result among LLMs.
The right portion of Table 2 presents a more finegrained view through the distribution of questionlevel PSR across intervals. We separately highlight the proportion of questions with PSR = 1.0, which represents perfectly coherent knowledge for that question that strictly satisfies the closure property of KST. This distributional analysis reveals a deeper gap. For human learners, the vast majority of questions (72.7%) achieve a perfect PSR of 1.0, indicating that human correct answers are almost always grounded in mastery of the prerequisite knowledge. As model capability increases, the LLM distributions shift toward the higher PSR intervals; nevertheless, perfect-PSR rates remain substantially below that of human learners. Even the best-performing model, Q WEN 3-80B-I NSTRUCT, reaches 48.16%, exhibiting less consistent full prerequisite satisfaction. Results on XESverified To assess how LLM annotation errors affect our results, we repeat the evalua-
Finding on NB1: Despite achieving comparable or higher accuracy than humans, LLMs exhibit substantially lower prerequisite satisfaction, with far fewer correct answers being fully grounded in the requisite knowledge structure.
5.2
Scaffolding Effect (NB2)
To evaluate NB2, we compare five retrieval strategies for selecting in-context exemplars: (1) nocontext baseline, (2) random questions, (3) questions sharing the same skill, (4) prerequisite questions identified by our knowledge graph, and (5) semantically similar questions. All methods are evaluated on the common subset where both prerequisite and same-skill exemplars are available. We report task accuracy and Scaffolding Gain (SG), defined in Eq. 5 as the change in accuracy relative to the no-context baseline. In human learning, prerequisite or related knowledge facilitates the acquisition and application of more advanced concepts (Wood et al., 1976), and revisiting foundational concepts often improves performance on dependent tasks (Sweller, 1988; Falmagne et al., 2013). If LLMs organize math knowledge in similar prerequisite dependencies, providing prerequisite examples should therefore yield larger gains than others. Figure 3 shows that providing additional context generally improves LLM performance. However,
94 .1 94 .7 95 .9 96 .1 94 .6 88 .6 89 .3 89 .7 89 .8 89 .3
95
85 80
.8 82 .1 82 .2 82 .1 80 .2
90
80
Accuracy (%)
tion on the verified subset (Table 3). The main conclusions hold: micro-PSR and macro-PSR follow the same trends as on the full LLM-annotated set, and the proportion of questions with PSR = 1 shows the same overall pattern, with human learners at 81.6% against 55.7% for the best-performing LLM. Both groups achieve higher PSR = 1 rates than on the full set, as expected because the verified subset has fewer prerequisites per target question (7.69 on average, vs. 16.43), making the all-prerequisitescorrect condition easier to satisfy. Nevertheless, the human–LLM gap remains comparable (25.9 vs. 24.54 percentage points), indicating that this change in evaluation scale does not alter the underlying result. Taken together, the results on both XESfull and XESverified datasets demonstrate the same conclusion that human knowledge structures are more coherent than those of current LLMs. In addition, the consistent findings indicate that PSR is robust to a moderate level of concept annotation error.
Qwen2.5-7B
No context Random
Qwen2.5-32B
Same-skill Similarity
Qwen3-80B
Prerequisite No-context (ref. line)
Figure 3: Knowledge scaffolding across various context types on the common subset.
prerequisite-based context does not consistently outperform alternative retrieval strategies. While most in-context strategies improve performance relative to the no-context baseline, the strongest gains consistently come from same-skill and semantically similar exemplars. Even randomly selected examples frequently match the effectiveness of prerequisite retrieval. For Q WEN 2.5-7BI NSTRUCT, prerequisite contexts provide no measurable benefit and slightly reduces performance relative to the no-context baseline (SG=−0.56). For Q WEN 2.5-32B-I NSTRUCT and Q WEN 3-80BI NSTRUCT, prerequisite retrieval yields only modest gains (SG = +0.70 and +0.50, respectively), remaining below same-skill and semantically similar retrieval; further, in Q WEN 2.5-32B-I NSTRUCT, it merely matches random exemplars. These findings suggest that the benefits of incontext examples arise primarily from exposure to relevant solution patterns rather than from activating prerequisite knowledge required by the target problem. While prerequisite examples can sometimes improve performance, they do not provide a systematic advantage over same-skill or semantically similar contexts. This contrasts with the central prediction of KST, where prerequisite knowledge plays a privileged role in supporting downstream learning. Overall, the results suggest that current LLMs rely more on contextual pattern matching than on a structured prerequisite hierarchy during mathematical reasoning. Finding on NB2: Contrary to the premise of KST, contexts of relevant knowledge do not provide stronger scaffolding than semantically-related examples, suggesting that LLMs benefit primarily from contextual pattern matching rather than from activating relevant knowledge during reasoning.
Figure 4: Normalized knowledge overlap coefficients (Eq. 7) between the knowledge states of different LLMs and of different human learner groups. Models are ordered by performance from weaker to stronger within both the LLM and human groups.
5.3
Knowledge Subsumption (NB3)
In this experiment, we investigate whether the knowledge state of a stronger model subsumes that of a weaker model, as prescribed by NB3. Similar to previous experiments, we include human learners as a reference. To compare the knowledge states of different learners, we simulate three human groups of low, medium, and high ability. Specifically, we first compute the average accuracy of each student across all attempted questions, and partition students into three groups based on performance percentiles: the bottom 0–30% as the low group, 30–60% as the medium group, and 60– 90% as the high group. The accuracy scores for three groups are 0.64, 0.80, and 0.89, respectively. We exclude the top 10% of students, as their nearperfect knowledge states would trivially yield close to 100% overlap. The knowledge state of each human group is then defined as the set of questions whose average accuracy within that group exceeds a threshold of 0.5. We present the normalized KOCnorm results across models and human groups in Figure 4, and the unnormalized results can be found in Appendix Figure 5. From results across different regions (by orange lines), we can observe that human learner groups of different performance levels (lower-right) exhibit high overlap, consistent with our expectation that in a structured and interdependent knowl-
edge system, the knowledge of weaker learners should be largely subsumed by that of stronger ones. An interesting finding is that the overlap between LLMs and human learners (upper-right) is substantially lower, and stronger models appear to show even less alignment with human learners. Among LLMs (upper-left region), the normalized KOC reveals more nuanced patterns. Opensource models exhibit relatively strong mutual overlap, though still lower than that among human learner groups. In contrast, the two closed-source models, GPT-4.1- MINI and C LAUDE, show considerably lower overlap with open-source models, and only 0.38 overlap with each other. A plausible reason is that open-source models share substantial portions of their training corpora, while closed-source models likely draw from more diverse and proprietary data sources, leading to divergent knowledge distributions. Taken together, these results suggest that LLMs do not necessarily follow a consistent shared knowledge structure. The locally high overlap observed among some model pairs is more likely a reflection of shared training data than evidence of a coherent common knowledge organization. Finding on NB3: Human learners exhibit nearperfect knowledge subsumption across ability levels, while LLMs show substantially lower and more fragmented knowledge growth — with stronger LLMs paradoxically diverging further from human knowledge progression patterns.
6
Conclusion
We introduced a KST–grounded framework that evaluates the structural coherence of LLM knowledge via various normative behaviors. Across eight LLMs and more than 18,000 human learners, we find that high accuracy masks pervasive structural inconsistency: even the strongest model fully satisfies prerequisites for only 48.16% of its correct answers, compared to 72.7% for human learners. Moreover, providing prerequisite-grounded context yields no clear advantage over surface-similar baselines, indicating that LLMs do not reliably use prerequisite knowledge as human-like scaffolding. These findings suggest that current LLM knowledge is fragmented rather than hierarchical, and motivate structure-aware assessment as a complementary lens for rigorous evaluation.
7
Limitations
We state the limitations of this work from the following aspects. First, our framework assumes the availability of an expert-defined concept dependency graph. While such resources exist for mathematics and several educational domains, constructing reliable prerequisite structures may be challenging in domains where knowledge dependencies are less explicit or less well documented. Second, we focus exclusively on mathematics, a domain with relatively well-established prerequisite relations. Whether the same observations extend to other domains, such as science, programming, or general factual knowledge, remains an open question. Finally, we operationalize knowledge states at the question level rather than directly modeling latent concept mastery. Although this enables large-scale evaluation using observable responses, questionlevel correctness is only an imperfect proxy for underlying knowledge states. Future work could incorporate concept-level mastery estimation to more closely align the evaluation with the original formulation of Knowledge Space Theory.
8
Ethical Statement
This work investigates the knowledge structure of large language models using publicly available mathematics datasets and benchmark questions. All data are used only for research purposes and do not contain personal or sensitive information. AI assistance was employed for language editing and proofreading.
Acknowledgments This research was supported by the Swiss National Science Foundation (SNSF) under grant number 10009282 and by a Swiss AI large grant. Heejin Do was also supported by the ETH AI Center through an ETH AI Center postdoctoral fellowship to H.
References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
multi-granularity text embeddings through selfknowledge distillation. In Findings of the association for computational linguistics: ACL 2024, pages 2318–2335. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Eric Cosyn, Hasan Uzun, Christopher Doble, and Jeffrey Matayoshi. 2021. A practical perspective on knowledge space theory: Aleks and its data. Journal of Mathematical Psychology, 101:102512. Peng Cui and Mrinmaya Sachan. 2023. Adaptive and personalized exercise generation for online language learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10184–10198, Toronto, Canada. Association for Computational Linguistics. Debora De Chiusole, Umberto Granziol, Andrea Spoto, and Luca Stefanutti. 2024. Reliability of a probabilistic knowledge structure. Behavior Research Methods, 56(7):8022–8037. Aniket Rajiv Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy P Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael Curtis Mozer, and Sanjeev Arora. 2024. Metacognitive capabilities of LLMs: An exploration in mathematical problem solving. In The Thirtyeighth Annual Conference on Neural Information Processing Systems. Heejin Do, Jaehui Hwang, Dongyoon Han, Seong Joon Oh, and Sangdoo Yun. 2025. What defines good reasoning in llms? dissecting reasoning steps with multiaspect evaluation. arXiv preprint arXiv:2510.20603. Jean-Paul Doignon and Jean-Claude Falmagne. 1985. Spaces for the assessment of knowledge. International journal of man-machine studies, 23(2):175– 196. Jean-Paul Doignon and Jean-Claude Falmagne. 2012. Knowledge spaces. Springer Science & Business Media. J-Cl Falmagne and J-P Doignon. 1988. A class of stochastic procedures for the assessment of knowledge. British Journal of Mathematical and Statistical Psychology, 41(1):1–23.
National Governors Association and 1 others. 2010. Common core state standards. Washington, DC.
Jean-Claude Falmagne, Dietrich Albert, Christopher Doble, David Eppstein, and Xiangen Hu. 2013. Knowledge spaces: Applications in education. Springer Science & Business Media.
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3embedding: Multi-linguality, multi-functionality,
Jean-Claude Falmagne and Jean-Paul Doignon. 2010. Learning spaces: Interdisciplinary applied mathematics. Springer Science & Business Media.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. Preprint, arXiv:2310.06825. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY, USA. Association for Computing Machinery. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, and 1 others. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702. Hang Li, Tianlong Xu, Jiliang Tang, and Qingsong Wen. 2024. Automate knowledge concept tagging on math questions with llms. arXiv preprint arXiv:2403.17281. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step, 2023. URL https://arxiv. org/abs/2305.20050, 17. Zitao Liu, Qiongqiong Liu, Teng Guo, Jiahao Chen, Shuyan Huang, Xiangyu Zhao, Jiliang Tang, Weiqi Luo, and Jian Weng. 2023. Xes3g5m: A knowledge tracing benchmark dataset with auxiliary information. Advances in Neural Information Processing Systems, 36:32958–32970. Ratna Narayan, Cynthia Rodriguez, Juan Araujo, Ali Shaqlaih, and Glenda Moss. 2013. Constructivism—constructivist learning theory. Roger Nkambou, Riichiro Mizoguchi, and Jacqueline Bourdeau. 2010. Advances in intelligent tutoring systems, volume 308. Springer Science & Business Media. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, and 1 others. 2025. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. Jun Seo, Sangwon Ryu, Heejin Do, Hyounghun Kim, and Gary Geunbae Lee. 2026. Behavior-aware item
modeling via dynamic procedural solution representations for knowledge tracing. arXiv preprint arXiv:2604.08260. Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, and 1 others. 2024. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:2407.21009. John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science, 12(2):257–285. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-ofthought prompting. Advances in Neural Information Processing Systems, 36:74952–74965. Jonathan Uesato, Nate Kushman, Ramana Kumar, H Francis Song, Noah Yamamoto Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with processbased and outcome-based feedback. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837. David Wood, Jerome S Bruner, and Gail Ross. 1976. The role of tutoring in problem solving. Journal of child psychology and psychiatry, 17(2):89–100. Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2025. Evaluating mathematical reasoning beyond accuracy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27723–27730. Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen, Jianqiao Lu, Haochen Tan, Yukang Chen, Hao Zhang, Zhan Shi, Bailin Wang, Zhijiang Guo, and Jiaya Jia. 2024. Mr-ben: A meta-reasoning benchmark for evaluating system-2 thinking in llms. In Advances in Neural Information Processing Systems, volume 37, pages 119466–119546. Curran Associates, Inc.
A
Prompts
Prompt 3: Scaffolding In-Context Prompt System: Please reason step by step, and put your final answer within \boxed{}. User: The following problems and their solutions are provided as background knowledge:
Prompt 1: LLM Evaluation Prompt System: Please reason step by step, and put your final answer within \boxed{}. User: {Question}
Problem 1: {Question 1} Options: {Options 1} % MCQ-type Only Solution: {Solution 1} Answer: {Answer 1} Problem 2: {Question 2} Options: {Options 2} % MCQ-type Only Solution: {Solution 2} Answer: {Answer 2} Problem 3: {Question 3} Options: {Options 3} % MCQ-type Only Solution: {Solution 3} Answer: {Answer 3}
Prompt 2: Concept Extraction Prompt
—
You are an expert in mathematics education. You will be given a list of reference skills and a mathematical question along with its solution. Your task is to identify all the skill(s) from the given skill list that are required to solve the problem.
Now solve the main problem: {Target Question}
Do NOT invent new skills. Do NOT modify skill names. Only choose from the provided skill list. Output a JSON array containing only the ids of selected skills. If no skill from the list applies, output an empty list: [] Output ONLY valid JSON without any additional text. Skill list: {Reference skills} Question: {Question} Solution: {Solution}
% MCQ-type Only Answer Choices: (A) {Target Option A} (B) {Target Option B} (C) {Target Option C} (D) {Target Option D}
B
Evaluation of LLM-based Concept Annotation on NYS Example Questions
Quantitative Evaluation on NYS Standards. Among the 480 NYS mathematics concept descriptions, 325 are associated with an example problem. We treat these example <Problem, Concept> pairs as ground-truth annotations and evaluate the accuracy of LLM-based concept annotation on them. Out of the 325 instances, 192 are correctly annotated, 74 are incorrectly annotated, and the remaining 59 are left unannotated, i.e., no matching concept was identified. This suggests that the LLMbased annotation has some limitations in recall.
ID Question
LLM-annotated concepts
1
Distance between A and B is 350 km. A car leaves from A at 8:00 with speed 40 km/h toward B. After 2 hours, another car leaves from B toward A at 50 km/h. At what time do the two cars meet on the road? ( )
2
A rectangle with perimeter $$20$$ meters and length $$7$$ meters; what is its width ( ) meters?
3
From the numbers $$5$$, $$6$$, and $$7$$, choose two numbers to form the largest two-digit number ( ).
Solve two-step word problems using the four operations. Represent these problems using equations with a letter standing for the unknown quantity. Assess the reasonableness of answers using mental computation and estimation strategies including rounding. Solve real world and mathematical problems involving perimeters of polygons, including finding the perimeter given the side lengths, finding an unknown side length, and exhibiting rectangles with the same perimeter and different areas or with the same area and different perimeters. Understand that the two digits of a two-digit number represent amounts of tens and ones.
4
There are $$8$$ boxes of apples, with $$40$$ apples per box. If evenly distributed among $$4$$ classes, how many per class?
5
A fraction with denominator 6 and numerator 5 is ____.
6
On one side of a road that is $$24$$ meters long, plant a tree every $$4$$ meters, with neither end planted. How many trees can be planted along this road?
Use multiplication and division within 100 to solve word problems involving equal groups, arrays, and measurement quantities, for example by using drawings and equations with a symbol representing the unknown number. Understand a fraction 1/b as the quantity formed by one part when a whole is divided into b equal parts, and understand a fraction a/b as the quantity formed by a parts of size 1/b. Express the length of an object as a whole number of length units by laying multiple copies of a shorter object (the length unit) end to end. Understand that the length measurement of an object is the number of same-size length units that span it with no gaps or overlaps. Limit to cases where the object is measured by a whole number of length units with no gaps or overlaps.
Table 4: Examples of questions and their LLM-annotated concepts on the XES dataset.
true proportion of PSR = 1 cases in Table 2 is likely overestimated for both humans and LLMs. Excluding cases without identified concepts, the annotation accuracy of the LLM reaches 72%. There remains substantial room for improvement, which we expect could be achieved with stronger annotation LLMs.
Figure 5: Raw knowledge overlap coefficients (Eq. 6) between the knowledge states of different LLMs and human learner groups.
However, in our framework, questions without an identified concept are discarded and excluded from subsequent computations. As a result, this may reduce the number of prerequisite relations we are able to discover, but we prioritize the precision of discovered prerequisite relations over introducing noisy ones. Lower recall also implies that the
Case Study on XES. We list several examples of both high-quality and imperfect concept annotations on our XES dataset in Table 4. Examples 1–3 illustrate accurate annotations with strong item–concept alignment. The remaining examples correspond to cases that are still related to the items, but exhibit imperfect alignment in different ways. In Example 4, the annotated concept is conceptually relevant, but involves numerical values beyond 100, exceeding the scope of the item itself. Example 5 is associated with a concept that is substantially broader than the competency required by the question. In Example 6, the concept captures the underlying idea of equal-length partitioning, but the item itself primarily requires discrete interval counting with endpoint constraints rather than length measurement.
C
Reasoning Scores and PSR Provide Complementary Views of Capability
A natural question is whether existing reasoningquality metrics capture the same information as prerequisite satisfaction. To investigate, we score each model’s reasoning traces using GPT-4.1- MINI as an LLM-as-judge along three dimensions: Relevance, Coherence, and Accuracy, as defined by Do et al. (2025). Figure 6 compares reasoning scores and PSR across the five open-source models. Although both metric families generally improve with model capability, they exhibit different patterns in the highperformance regime. Among the three strongest models, reasoning scores differ only modestly, whereas the PSR= 1.0 rate remains more variable and non-monotonic. Specifically, Relevance increases from 4.76 for L LAMA -3.1-70B-I NSTRUCT to 4.86 for Q WEN 2.5-7B-I NSTRUCT and 4.93 for Q WEN 2.5-32B-I NSTRUCT, while the corresponding PSR= 1.0 rates are 20.81%, 27.32%, and 25.44%, respectively. (a) Reasoning Scores Reasoning Score (1-5)
5.0 4.5 4.0 3.5 3.0 Relevance Coherence Accuracy (score)
2.5 2.0
Mistral 7B
Llama 8B
Llama 70B
Qwen2.5 7B
(b) PSR Metrics
0.9
PSRmicro PSR = 1.0 rate
0.8
0.20
PSR = 1.0 rate
PSRmicro
0.30 0.25
0.7 0.6
0.15
0.5
0.10
0.4
0.05
0.3 0.2
Qwen2.5 32B
Mistral 7B
Llama 8B
Llama 70B
Qwen2.5 7B
0.00
Qwen2.5 32B
Figure 6: Comparison of reasoning scores (a) and prerequisite satisfaction (b) across five open-source models.
This divergence reflects the different behaviors captured by the two metrics. LLM-as-judge evaluation measures the local relevance, coherence, and correctness of individual reasoning traces, whereas PSR measures cross-question consistency with pre-
requisite relations. Consequently, models that appear similarly strong under conventional reasoning evaluation may still differ in strict prerequisite satisfaction. Therefore, we view PSR as a complementary diagnostic: reasoning scores capture the quality of individual reasoning traces, while PSR captures whether a model’s response patterns consistently respect the prerequisite structure.