arXiv CSopen access

Can Vision Language Models Be Adaptive in Mathematics Education? A Learner Model-based Rubric Study

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificial intelligence, reasoning, knowledge representation

Can Vision Language Models Be Adaptive in Mathematics Education? A Learner Model-based Rubric Study Jie Gao1,2 * Yongan Yu1,2 * Junzhu Su1 Yiran Lin1 Adam K. Dubé1† Jackie Chi Kit Cheung1,2,3† 1 2 McGill University Mila – Quebec AI Institute 3 Canada CIFAR AI Chair {jie.gao3, yongan.yu, junzhu.su, yiran.lin}@mail.mcgill.ca [email protected], [email protected]

arXiv:2605.16011v1 [cs.CL] 15 May 2026

Abstract Adaptive learning refers to educational technologies that track learners’ learning progress and adapt the instructional process based on individual learners’ learning performance. It is increasingly recognized as critical for developing an effective learning support tool. Vision language models (VLMs) have seen adoption in mathematics education, and students have been using them as learning aids for personalized instruction. However, it is unknown whether VLMs have the ability to adapt to different learner profiles when providing mathematical instructions. Current VLMs lack a systematic evaluation framework for this adaptivity to different learner profiles in mathematics tutoring tasks. To address this gap, we draw on the learner model from the adaptive learning framework (Shute and Towle, 2018) and propose a learner model-based rubric. Our rubric formalizes adaptivity assessment into three aspects: cognitive aspects, motivational aspects, and complexity. We also evaluate two additional dimensions of VLM responses: correctness (of answers and solutions) and quality (of the response itself). Our experimental results show measurable differences in adaptivity across models, and also reveal that current VLMs struggle to consistently produce learner model-based instructional responses, especially when receiving limited learner information.

Introduction

As artificial intelligence shows growing use in education, there is a corresponding rise in large language model-powered tools for providing mathematical instructions and pedagogical support. Learners increasingly adopt VLMs as personal tutoring agents to answer mathematical problems and provide personalized solution feedback (Baral et al., 2025; Cherian et al., 2024; Gupta et al., 2025; * Equal contribution. †

Corresponding authors.

Adaptivity Schema

Learner Model

Correctness

Accuracy

Current Knowledge

Learner's prior knowledge on mathematics.

Math Learning Attitude

Student's attitude and interest toward learning mathematics.

Self-efficacy

Student’s self-efficacy for completing mathematics tasks and tests.

Common Core State Standards

Common Core State Standards math knowledge scope and difficulty.

Difficulty Level

Adapting to the learner's current knowledge to respond.

Complexity

Offering virous instructional supports, such as clarifying the meaning of the math question, providing examples, additional practice, and giving cues (e.g., step-by-step) to scaffold student understanding.

Figure 1: Overview of the learner model-based adaptivity schema. The framework delineates the interplay between learner profile and instructional scaffolding.

Liu et al., 2024). Personalization is both a defining characteristic of personal tutoring and a critical principle in education, as educators have historically adapted their instruction to provide personalized and differentiated learning support based on diverse learner backgrounds (Bernacki et al., 2021). To provide personalized learning to address different individuals’ learning needs, it is vital to understand learners’ learning performance before making instructional decisions. Adaptive learning, as one of the most effective educational approaches, involves providing personalized learning instruction and feedback by adapting the learning process to different learners’ needs dynamically (Imhof et al., 2020; Shute and Towle, 2018; Vanzo et al., 2025; Zheng et al., 2022). Its goal is to maximize the effectiveness of instruction through tracking learners’ prior academic background and learning performance, adjusting the content to suit different learners (Shute et al., 2000). A foundational component of adaptive learning is the learner model,

which captures key aspects of a learner, such as learning performance and learning attitudes, in order to support individualized instruction. Several studies have demonstrated that adaptive learning is superior to one-size-fits-all teaching approaches (Du Plooy et al., 2024; Wang et al., 2023). Current VLMs primarily collect learner information from conversational interactions (Kumar et al., 2023). When learners do not explicitly share details about their learning performance, behaviors, or needs, it can be difficult to infer these learners’ profiles, which limits instructional decision-making (Song et al., 2024). Subsequently, it remains unclear whether VLMs generate truly appropriate answers and solutions to support individual learners’ needs (Sharma et al., 2025). This paper addresses this critical research gap by developing an adaptive rubric grounded in the learner model of adaptive learning frameworks. Our rubric incorporates core evaluation factors from learner models, including cognitive and motivational aspects. The rubric aims to assess VLMs’ adaptive capabilities in mathematics tutoring tasks and their capacity to deliver pedagogical instruction that aligns with learners’ knowledge backgrounds. Accordingly, our research question is: "To what extent can VLMs demonstrate adaptive instructional capabilities in visually grounded mathematics tutoring tasks?" Our contributions are summarized as follows: (1) We propose a learner model-based rubric to systematically evaluate the adaptivity schema of VLMs in mathematics tutoring tasks, grounded in cognitive, motivational, and complexity dimensions, as illustrated in Figure 1. (2) We construct a humanannotated evaluation dataset based on TIMSS mathematics items, incorporating diverse learner profiles and controlled contextual conditions to simulate realistic tutoring scenarios. The constructed model generation pipeline and rubric design are available at our GitHub repository1 . (3) We conduct extensive experiments of five state-of-the-art VLMs, providing first-hand analysis of challenges in generating learner-aware instructional responses, particularly under limited learner information.

GitHub Repo: Adaptive-LLMs

https://github.com/Michaelyya/

Background & Related Work

2.1

Adaptive Learning: The Learner Model

The learner model forms the foundation of adaptive learning by integrating assessment information and supporting inferences about learners’ proficiencies (Shute and Towle, 2018). Personalization within adaptive learning systems begins with systematically appraising learner characteristics (Zhou et al., 2023). The systematic review of Bernacki et al. (2021) identified key learner characteristics across four dimensions: cognitive factors, beliefs and motivations, social and cultural factors, and emotions and affect. These dimensions directly inform the structural design of our rubric. For instance, understanding a learner’s prior knowledge (a cognitive factor) is essential before providing instruction. Accordingly, our rubric include the current knowledge criterion to evaluate how VLMs identify learners’ prior levels to avoid redundant or overly challenging content. Similarly, Plass and Pawar (2020) categorized variables that can be adapted in instructional responses into four domains: cognitive, motivational, affective, and sociocultural. Within the motivational domain, self-efficacy reflects a learner’s perceived capacity to perform domainspecific tasks. As a result, a learner may display a generally positive attitude toward learning while simultaneously holding low confidence in completing specific mathematics tasks (Kaskens et al., 2020; Wen and Dubé, 2022). Our work synthesized insights from previous research and considered the VLMs’ conversational attributes for the adaptive rubric construction. 2.2

The Role of (V)LMs in Education

In the domain of mathematics, (V)LMs have demonstrated remarkable proficiency as "solvers," achieving superior performance on both text-based (e.g., GSM8K (Cobbe et al., 2021), MathQA (Amini et al., 2019)) and vision-based benchmarks (e.g., MATH-Vision (Wang et al., 2024), MathVerse (Zhang et al., 2024)). However, a critical distinction exists between automated solvers and educational agents: solvers prioritize problem-solving efficiency and final answer correctness (Yan et al., 2025), whereas educational agents must provide pedagogically transparent guidance and scaffolding appropriate to the learner’s zone of proximal development (Cai et al., 2025; Yu et al., 2025). Recent work suggests that LLMs can mimic tutorlike behavior in interactive sessions (Vanzo et al.,

2025), and Scarlatos et al. (2025) further highlights their potential role in knowledge tracing. Despite this progress, existing work tends to overlook the multimodal nature of authentic mathematics learning. Real-world mathematical tasks often require students to reason across modalities, integrating textual instructions with geometric diagrams, data visualizations, and function plots. The emergence of VLMs, including GPT-4V, LLaVA, and QwenVL, enables joint visual-textual reasoning (Lu et al., 2023). Current evaluation paradigms, however, largely mirror text-only settings, emphasizing visual question-answering accuracy rather than instructional reasoning quality (Lee and Zhai, 2025). Such accuracy-focused evaluations fail to capture ability to explain visual concepts or adapt guidance to a learner’s context, leaving limited empirical evidence on whether multimodal models can serve as adaptive tutors. 2.3

Empirical Study in Adaptive Learning

The integration of VLMs into educational technologies represents a paradigm shift for intelligent tutoring systems (Liu et al., 2025). While generative model-based systems offer the potential for real-time customization of instructional content based on student behavior (Banjade et al., 2024; Lin et al., 2023), their capacity to deliver contextually appropriate guidance remains questionable. Notably, Borchers and Shou (2025) employed a prompt variation framework to assess pedagogical adaptivity, revealing that even state-of-the-art models frequently fail to adjust their feedback strategies when critical context (e.g., student performance or knowledge components) is introduced or removed. This suggests a "context insensitivity" where models revert to generic tutoring scripts rather than responding to the specific state of the learner. To address this gap, we aim to provide a rigorous, rubric-based assessment of how well VLMs adhere to specific learner profiles.

Rubric Design for Adaptivity Evaluation

Our rubric is engineered to evaluate whether VLMs demonstrate adaptive instructional intelligence when tailoring support for learners with diverse mathematical proficiencies and backgrounds. From theory to metric, the rubric is structured across three axes: Adaptivity, Correctness, and Quality. This structure was strategically selected to

disentangle pedagogical intent from technical accuracy, ensuring that the rubric can isolate a model’s adaptivity from its underlying linguistic and mathematical capabilities. Among these, Adaptivity constitutes the core evaluative focus, while Correctness and Quality serve as essential baseline controls to ensure mathematical rigor and linguistic clarity. In this work, we operationalize a VLM response into two distinct components: the ’Answer’ (the final result) and the ’Solution’ (the comprehensive instructional content). The "solution" encompasses all reasoning steps, feedback, and scaffolding strategies beyond the final answer. This proposed rubric assesses whether the model meets the baseline standard of adaptive instructional behavior. At a high level, the adaptivity dimension assesses whether model responses are appropriately aligned with both the instructional intent of the prompt and the learner’s background. The correctness dimension evaluates whether the answers and solutions are mathematically valid. The quality dimension examines overall VLM response quality. Adaptivity. The adaptivity dimension includes a total of nine key questions (Q1–Q9; see Appendix A.1). The evaluation elements of these questions were selected based on the core factors of the adaptive learning environments (Bernacki et al., 2025; Plass and Pawar, 2020), while considering the interactive characteristics of VLMs. The primary factors in this rubric include: the learner’s current mathematical knowledge, the learner’s math learning attitude, the learner’s self-efficacy, the solution’s mathematical knowledge scope, the solution difficulty, and the solution complexity. For instance, when a learner presents a negative math learning attitude, an adaptive response should offer positive feedback and instruction to help the learner address the math problem, rather than continuing the conversation in a negative tone or ignoring it (Wen and Dubé, 2022). In addition, to establish a rigorous and fine-grained normative reference in our rubric, we integrated Common Core State Standards (CCSS) as a key dimension. Although TIMSS frameworks provide broad domain categories, CCSS can provide a micro-level taxonomy for evaluation and exhibit 90% conceptual alignment with TIMSS frameworks(Schmidt and Houang, 2012). Solution complexity is assessed across Q6–Q9, examining whether the LLM response helps the learner understand and explain the math question, providing relevant examples, offering additional similar math exercises for rein-

TIMSS Math Benchmark

High-performance Group 1

625 Advanced

550 High

Learner Profiles

Test Groups

700

Input: • Math questions only

Group 2 Input: • Math questions • User prompt

500

Low-performance

Intermediate-performance

Grade level: Grade 4 / Grade 8 Math learning attitude: Very like! Confidence: Very confident! Taught the mathematics topics: All math topics in Grade 4. Math achievement in TIMSS 2019: 615

Grade level: Grade 4 / Grade 8 Math learning attitude: Very like! Confidence: Confident Taught the mathematics topics: All math topics in Grade 4. Math achievement in TIMSS 2019: 550

Grade level: Grade 4 / Grade 8 Math learning attitude: Dislike Confidence: Not confident Taught the mathematics topics: A part of math topics in Grade 4. Math achievement in TIMSS 2019: 390

Graph-based Math Question

User Prompt Example

I am a student from Grade 4, I don’t like learning mathematics and I am not confident in mathematics. Now I have mastered mathematics topics such as number and data in grade 4. I got 390 in the TIMSS 2019 Math Test. Can you teach me this math question?

Group 3

475 Intermediate

Input: • Math questions • System prompt (without TIMSS benchmark) • User prompt

400 Low

Group 4

Group 2 Model Outputs

LLM

Group 4 Input: • Math questions • Full system prompt • User prompt

300

Group 3

Group 1

Annotation

Rubric

LLM Response

Adaptivity Correctness

Quality

Adaptive Rubric

Answer

The answer is C.

Solution

Let's solve this step by step. I know you said you don't like math, but this problem is actually like a fun puzzle.....

Figure 2: Overview of the adaptive rubric–based evaluation pipeline. Learner profiles and mathematics questions are combined under four test groups with increasing learner-specific information, provided to VLMs to generate instructional responses, and annotated by experts using the proposed rubric.

forcement, and delivering step-by-step and detailed instructions. Correctness. This dimension mainly focuses on assessing if the VLM responses provide valid and correct instruction and feedback, addressing the specific math questions. Both the answer and the solution should be correct and consistent to determine the correctness of the response (Q10–Q12; see Appendix A.2). In this dimension, three questions were used to evaluate the correctness of the answer and its solution. Notably, a completely correct solution is defined as one that includes both complete and valid mathematical reasoning and yields a correct final answer. If the solution demonstrates accurate mathematical reasoning but ends with an incorrect final answer, it should be labeled as partially correct. Quality. This dimension focuses primarily on evaluating the overall quality of VLM generation. We conducted fine-grained, multi-dimensional annotations and collected multiple crucial evaluation dimensions from existing benchmarks for summarization quality evaluation (Jones et al., 2024; Lee et al., 2024; Tang et al., 2024), including hallucinations, mathematical reasoning, faithfulness, coherence, consistency, grammatical correctness, formatting adherence, and length control. The quality dimension includes a total of seven questions (Q13– Q19; see Appendix A.3).

Rubric-Guided Evaluation of VLMs

To empirically evaluate the adaptive capabilities of VLMs in mathematics tutoring tasks, we implement a structured evaluation pipeline grounded in the proposed learner model–based rubric. As illustrated in Figure 2, the pipeline constructs learner profiles, pairs them with mathematics questions under varying contextual conditions, elicits instructional responses from VLMs, and evaluates the resulting outputs through expert annotation. This methodology adopts a quasi-experimental design that simulates realistic tutoring interactions by systematically varying two key factors: learner profiles and test groups. While we acknowledge that such simulated interactions using synthetic learner profiles are not the final answer to evaluating VLMs in the open-ended complexity of real-world classrooms, they represent a necessary and rigorous first step. By leveraging a controlled synthetic environment, we can isolate specific adaptive behaviors and minimize the confounding variables inherent in live discourse between humans and AI. 4.1

Dataset and Model Selection

We draw mathematics items from the Trends in International Mathematics and Science Study (TIMSS) 2019 and 2023 with official permission from the International Association for the Evaluation of Educational Achievement (IEA) (Fishbein et al., 2021, 2025). From the full assess-

Math Question

VLMs’ Response Hey there! It's totally okay to feel unsure about math sometimes. But you're really good with numbers and data, and that means you have a strong math brain! I know you can do it. Let's tackle this puzzle together, step-by-step. The main idea is that weigh the same means both sides of the scale are equal. Think of it like an addition problem! We’ll turn each picture into numbers. First scale: 1 kg + 2 kg on the left balances a pineapple on the right. So the pineapple = 1 + 2 = 3 kg. Second scale: pineapple + 2 kg balances the big gray weight. That’s 3 kg + 2 kg = 5 kg. So the big gray weight = 5 kg. Third scale: big gray weight + 1 kg balances the watermelon. That’s 5 kg + 1 kg = 6 kg. Answer: 6 kg (choice D). Tip for problems like this: Replace each picture with a number you find from the earlier balances. Add the weights on one side to find the unknown on the other side. You did it step by step—nice work!

Q: Does the response take the learner’s prior math knowledge into account? A: Yes

Q: Does this response try to build a positive attitude in the learner? A: Yes

Q: Does this response intend to improve the student’s self-efficacy for completing this mathematics task? A: Yes

Q: Does the solution meet the Common Core State Standards at this grade level? A: Yes

Adaptivity Evaluation

Q: Is the solution at an appropriate difficulty level for the learner? A: Yes Q: Does this response explain the meaning of this math question to help the learner understand the problem? A: Yes Q: Does this response provide an example aligned with this math question’s topic? A: No

Q: Does this response provide additional practice similar to this math question? A: No Q: Does the response provide cues (e.g., step-by-step, signals of importance) to guide the learner? A: No

Figure 3: A case study of the adaptive evaluation process for a Grade 4 math problem. The upper panel shows the VLM-generated response with motivational support and step-by-step scaffolding, while the lower panels illustrate the annotation workflow, where Q1–Q9 map directly to the adaptivity rubric in Appendix A.1.

ment pool, we construct a stratified dataset of Q = 10 representative questions, with five questions sampled from Grade 4 and five from Grade 8. To emphasize multimodal reasoning, we prioritize visually grounded items, resulting in 90% of the selected questions requiring graphical or diagrammatic interpretation. We evaluate a set of M = 5 VLMs spanning both proprietary and opensource paradigms. The detailed computational costs, model selection and decoding configuration are provided in Appendix B and Appendix C. 4.2

Learner Profiles and Context Conditions

To simulate realistic tutoring scenarios, we construct learner profiles derived from authentic TIMSS statistics and formalize each learner as a tuple ℓ = (a, c, p, s), where a denotes learning attitude, c denotes self-reported confidence, p represents prior topic coverage, and s corresponds to the learner’s TIMSS achievement score; across Grade 4 and Grade 8, we instantiate six profiles spanning three performance tiers (high, intermediate, and low). Building on this formulation, we study context sensitivity by defining four input context conditions Ck with monotonically increasing learner-specific information: C1 = {q},

C2 = {q, (a, c, p)},

C3 = {q, S, (a, c, p)},

C4 = {q, S, (a, c, p, s)},

where q denotes the mathematics question and S denotes a system prompt specifying grade-level learning objectives. 4.3

Expert Annotation and Evaluation

Our experimental design follows a full factorial structure. The total volume of model-generated responses is calculated as: N = |G| × |P | × |C| × |Q| × |M | = 600 |{z} |{z} |{z} | {z } Learner Profile

Context

Task

Model

where we account for |G| = 2 grades, |P | = 3 performance tiers, |C| = 4 context conditions, |Q| = 5 questions per grade, and |M | = 5 models. Data annotation was conducted by four domain experts trained in the Learning Sciences program. Prior to annotation, each expert completes a onehour one-on-one training session that introduces the rubric and includes discussion of at least three representative case studies to calibrate the scoring criteria. Each response was evaluated using structured numerical ratings defined by the rubric (see Appendix A). The rubric items were implemented in a Google Form, and annotators rated each VLM response using this instrument. Adaptivity score and Correctness score were positively

Adaptivity (Q1-9)

Model

Correctness (Q10-12)

Quality (Q13-19)

GPT-5

52%

51%

64%

72%

70%

79%

90%

89%

92%

GPT-o1

46%

44%

47%

68%

63%

65%

90%

88%

91%

Gemini-2.5-Flash Llama3.2-11B-VL Qwen3-30B-VL

65% 50% 71%

68% 53% 71%

70% 48% 70%

82% 19% 56%

80% 28% 57%

85% 20% 52%

95% 65% 76%

94% 63% 74%

96% 65% 72%

Table 1: Performance comparison of VLMs across dimensions. Values represent the mean score percentage achieved across the rubric for each dimension. Learner profiles are denoted by icons: High- ( ), Intermediate- ( ), and Low-performance ( ). Bold denotes the best performance; underline denotes the second best.

keyed, whereas Quality score is derived from errorrelated items. Specifically, the Quality score is 1 P19 computed as 1 − 7 i=13 ei , where ei ∈ {0, 1} indicates whether the corresponding error type (Q13– Q19) is present in the response. All seven error categories are equally weighted in this aggregation, such that higher values consistently indicate better performance. Statistical analyses are therefore conducted at the response level rather than the item level. For completeness, we additionally report item-level analysis in Appendix D. A case study is shown in Figure 3, the color-coded groupings represent the distinct dimensions of adaptivity. These annotations demonstrate how qualitative text outputs are converted into binary numerical scores (Yes=1, No=0). To validate the reliability of the annotation process, a subset of 120 responses was independently annotated by core coders, yielding a strong inter-rater agreement with a Cohen’s Kappa of κ = 0.87. Detailed agreement metrics for each model are reported in Appendix E.

Results

We examine the relationship between learner profiles and VLMs’ performance in adaptivity, correctness, and quality. Our analysis investigates whether these dimensions exhibit systematic variation across learner profiles and model-related factors (VLMs and test group), reflecting models’ sensitivity to learner differences and available contextual information. Table 1 summarizes overall performance patterns across VLMs and learner profiles. The reported values represent the mean percentage of total possible points achieved within each dimension. For example, a 70% Adaptivity score indicates the model satisfied 70% of the Q1–Q9 criteria. To provide deeper insight into specific adaptive behaviors, Table 2 disaggregates the Adaptivity dimension. It reports detailed performance on individual sub-metrics (e.g., Cogni-

tive, Motivational, Complexity) across the four test groups (G1–G4), revealing how the availability of context influences specific instructional strategies. To provide evidence for these relationships, we employ a series of two-way analyses of variance (ANOVA), with learner profile and model-related factors as independent variables. Further details of the analysis are reported in Appendix F and Appendix G. 5.1

Experimental analysis

Adaptivity (Q1–Q9). We analyzed how adaptivity scores relate to learner profiles (high-, intermediate, and low-performing), VLMs, and test groups. Overall, adaptivity scores showed significant relationships across VLMs and across test groups. Adaptivity varied substantially across models, with a significant main effect of VLMs, F (4, 705) = 41.81, p < .001, indicating differences in adaptivity performance across models. Tukey HSD post hoc comparisons indicated that Gemini-2.5Flash and Qwen3-VL-30B achieved significantly higher adaptivity scores than GPT-5, GPT-o1, and Llama3.2-11B-VL, whereas differences between Gemini-2.5-Flash and Qwen3-30B-VL were not statistically significant. In contrast, adaptivity scores showed no significant association with learner profiles, F (2, 705) = 1.72, p = .18, and no significant interaction between learner profile and VLMs, F (8, 705) = 1.63, p = .11. This uniformity across groups suggests that these VLMs default to a "one-size-fits-all" pedagogical style. While they could provide answers and solutions, they lack the nuanced scaffolding based on the specific needs of different learner profiles. A complementary analysis examined adaptivity as a function of learner profile and test group (Groups 1–4, reflecting increasing contextual information). The results showed a significant relationship between contextual information and adaptivity: responses

Model Performance by Group and Overall (O) Proprietary Models (%)

Dimensions

Llama3.2-11B-VL

Qwen3-30B-VL

Metric

G1 G2 G3 G4

G1 G2

G1 G2 G3 G4

Cognitive

Gemini-2.5-flash

CK (Q1)

100

CCS (Q4)

100

DL (Q5)

100

Motiv.

GPT-o1

LA (Q2)

100

SE (Q3)

100

Complexity

GPT-5

Open-Source Models (%)

Expl. (Q6)

100

94 5

Exam. (Q7)

Prac. (Q8)

SBS (Q9)

100

Average

Table 2: Detailed adaptivity dimension performance. Values represent the percentage (%) of responses satisfying rubric criteria. Row metrics are: Cognitive (CK: Current Knowledge, CCS: Content Scope, DL: Difficulty Level), Motivational (LA: Learning Attitude, SE: Self-Efficacy), and Complexity(Expl.: Explanation, Exam.: Examples, Prac.: Practice, SBS: Step-by-Step). Background colors distinguish Proprietary vs. Open-Source models.

generated with access to additional context exhibited higher adaptivity scores. This pattern was supported by a significant main effect of test group, F (3, 708) = 92.73, p < .001. Tukey HSD comparisons revealed that Group 1 (math question only) scored significantly lower than Groups 2, 3, and 4. Correctness (Q10–Q12). Correctness scores showed a similar pattern, with variation driven primarily by model differences, demonstrating a highly significant main effect of VLMs, F (4, 705) = 64.28, p < .001. Crucially, this model-driven performance was robust and consistent across different learner performance levels. Post hoc analyses indicated that Gemini-2.5-Flash and GPT-based models outperformed Llama3.211B-VL, and that Gemini-2.5-Flash also scored significantly higher than Qwen3-30B-VL. When examining learner profile and VLMs, correctness scores exhibited no association with learner performance level, as reflected by a non-significant main effect of learner profile, F (2, 705) = 0.01, p = .99, and no significant interaction, F (8, 705) = 0.65, p = .74. The lack of significant variance suggests two underlying dynamics regarding model behavior: either the baseline user prompt (Group 2) is already sufficient for the VLMs to reach their performance ceiling on this task, or the models ignore the supplementary system prompts provided in Groups 3 and 4. Quality (Q13–Q19). Quality scores further reinforced the pattern observed for adaptivity and

correctness: learner profiles did not systematically relate to the model response quality, whereas substantial differences emerged across VLMs. However, quality differed significantly across models, as indicated by a significant main effect of VLMs, F (4, 705) = 69.44, p < .001. Tukey HSD comparisons showed that Llama3.2-11B-VL produced significantly higher error counts than all other models, while Gemini-2.5-Flash achieved significantly lower error scores than GPT-5, GPT-O1, and Qwen3-30B-VL. Consistent with the overall analyses, distinct performance profiles emerged across the models. Gemini-2.5-Flash demonstrated strong and balanced performance across all metrics, while GPTo1 prioritized correctness and quality over adaptivity. Qwen3-30B-VL obtained the highest average adaptivity score across learner profiles (M = 71%). However, its correctness and quality were comparatively lower. This is mainly due to overly long model generations that were more likely to be truncated or incomplete, which in turn produced formatting issues and partial non-adherence to prompt instructions. We also observed a correctness gap between Llama3.2-11B-VL and the closed-source models, with the latter consistently outperforming Llama3.2-11B-VL. 5.2

Observations

This subsection summarizes key empirical observations derived from rubric-based annotations of

VLM responses. These observations are grounded in aggregated rubric scores across learner profiles and context conditions, supplemented by qualitative inspection of representative model outputs. VLMs exhibit minimal behavioral adaptation to learner attitudes and self-efficacy. As adaptivity constitutes the core dimension of the proposed rubric, we first examine how VLMs respond to learner-specific motivational factors, particularly learning attitude and self-efficacy (Q3–Q5). Across models, GPT-5, GPT-o1, and Llama3.211B-Vision-Instruct exhibit consistently low response rates in addressing these factors. For example, GPT-o1 addresses self-efficacy concerns in only 8% of applicable cases, with a 0% response rate for Grade 4 intermediate-performing learners. In contrast, low-performance learners receive substantially more motivational feedback, with an average response rate of 36.8% when low confidence is explicitly indicated. Among all evaluated models, Qwen3-30B-VL (33.1%) and Gemini-2.5Flash (32.5%) demonstrate comparatively stronger engagement with learners’ self-efficacy. Interestingly, some Qwen3-30B-VL responses include explicit encouragement (e.g., “You got this!”), suggesting partial instructional responses using motivational language. These patterns suggest models implicitly equate lower performance with greater motivational need, overlooking the importance of supporting confidence and learning attitudes among high- and intermediate-performing learners. Providing learners’ knowledge backgrounds to the VLM does not guarantee appropriate feedback. It is essential to understand learners’ current knowledge level before providing them with feedback. However, such an understanding does not mean that the VLMs have the capabilities to provide appropriate differentiated feedback. In several cases, responses to different learners are nearly identical, suggesting weak sensitivity to learner distinctions. The VLMs frequently misunderstand how to deliver suitable content to specific groups. Table 2 illustrates that GPT-5 generates longer and more detailed explanations for high-performance learners than for intermediate-performance learners, despite the latter typically requiring greater instructional scaffolding. These findings indicate that access to learner background information alone does not guarantee appropriate or pedagogically aligned feedback.

Multimodal misinterpretation and response quality failures. Beyond adaptivity, our rubricbased evaluation surfaces notable correctness and quality issues, particularly for visually grounded mathematics problems. Geometry items pose a significant challenge: for instance, a Grade 4 geometry question (G4Q5) required comparing four choices’ rectangles to identify the correct one. Although the task structure is clear, only 10 out of 60 model responses produced correct answers. This error pattern persists despite the problem’s clear structure, pointing to limitations in visual–spatial reasoning rather than ambiguity in the task itself. These errors are consistently reflected in low correctness scores across multiple models and learner profiles. Moreover, a common failure mode involves conflating multiple-choice option labels with their embedded numerical content. When presented with options like "D. 6kg", the correct response should identify the label (D) rather than the embedded value (6kg). However, models regarded the embedded value as the final answer, omitting the required option label. Additionally, when questions integrate images, formulas, and text, models often extract misaligned or inaccurate information. This manifests as hallucinations or inaccurate extraction in solution, compromising mathematical reasoning and instructional reliability, even when the final answer appears superficially plausible.

Conclusion

This paper evaluates whether VLMs demonstrate adaptive capabilities in mathematics tutoring tasks. To this end, we introduce a learner model–inspired rubric that provides a structured framework for assessing adaptivity beyond surface-level personalization. Applying this rubric to VLMs across learner profiles reveals that, while models exhibit some sensitivity to learner context, their adaptive behavior remains limited and inconsistent. In particular, models often fail to differentiate instructional strategies across learners with distinct academic backgrounds, indicating a gap between generic tutoring responses and genuinely adaptive instruction. The proposed rubric offers a principled tool for systematically comparing adaptive capabilities across models. Looking forward, future work may extend this rubric to distinguish baseline adaptivity from more sophisticated forms of instructional adaptation and to examine additional components of adaptive learning frameworks. More broadly,

this work highlights the need for evaluation methodologies that prioritize pedagogical alignment with learners, providing guidance for the development of more adaptive and educationally grounded multimodal models.

Limitations While we have developed the adaptive rubric and accompanying worksheet to identify the VLMs’ adaptive capabilities, the rubric questions for adaptivity are not exhaustive. Other adaptivity variables, including those from affective and socio-cultural aspects, are also important for evaluation. What’s more, the limited math items used in this study may make our evaluation suffer from insufficient item coverage regarding pedagogical adaptivity. In addition, our study simulated the learner rather than testing with real students, which cannot include all the learning behavior factors of real students. Moreover, there might be other differences inherent in using an automated tutoring system compared to human tutors that are not addressed in the current adaptive learning framework. For instance, human tutors may create more social pressure, which might be helpful or harmful for learning. A further limitation is that the proposed rubric is designed as a baseline that evaluates whether responses have adaptive capabilities in mathematics tutoring tasks. However, a response classified as adaptive in this study does not represent a high-quality, high-level adaptive response.

Ethical Consideration This study involves the evaluation of vision language models using synthetic and publicly available mathematical problem data. No personally identifiable information or human subject data were used in model evaluation. We obtained official permission from the International Association for the Evaluation of Educational Achievement (IEA) to use the Trends in International Mathematics and Science Study (TIMSS) 2019 and 2023 mathematics items from Grade 4 and Grade 8. Thus, the data has no potential privacy or content safety concerns. Additionally, the learner characteristics of learner profile data were extracted from the TIMSS 2019 and TIMSS 2023 public reports and datasets (Fishbein et al., 2021, 2025).

References Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers), pages 2357–2367. Shivraj Banjade, Hiran Patel, Sangita Pokhrel, and 1 others. 2024. Empowering education by developing and evaluating generative ai-powered tutoring system for enhanced student learning. Journal of Artificial Intelligence and Capsule Networks, 6(3):278–298. Sami Baral, Li Lucy, Ryan Knight, Alice Ng, Luca Soldaini, Neil T Heffernan, and Kyle Lo. 2025. Drawedumath: Evaluating vision language models with expert-annotated students’ hand-drawn math images. arXiv preprint arXiv:2501.14877. Matthew L Bernacki, Meghan J Greene, and Nikki G Lobczowski. 2021. A systematic review of research on personalized learning: Personalized by whom, to what, how, and for what purpose (s)? Educational Psychology Review, 33(4):1675–1715. Matthew L Bernacki, Candace Walkington, Alyssa Emery, and Ling Zhang. 2025. Handbook of Personalized Learning. Taylor & Francis. Conrad Borchers and Tianze Shou. 2025. Can large language models match tutoring system adaptivity? a benchmarking study. In International Conference on Artificial Intelligence in Education, pages 407–420. Springer. Lianyu Cai, Mgambi Msambwa Msafiri, and Daniel Kangwa. 2025. Exploring the impact of integrating ai tools in higher education using the zone of proximal development. Education and Information Technologies, 30(6):7191–7264. Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit, Joanna Matthiesen, Kevin Smith, and Josh Tenenbaum. 2024. Evaluating large vision-and-language models on children’s mathematical olympiads. Advances in Neural Information Processing Systems, 37:15779–15800. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Eileen Du Plooy, Daleen Casteleijn, and Denise Franzsen. 2024. Personalized adaptive learning in higher education: A scoping review of key characteristics and impact on academic performance and engagement. Heliyon, 10(21).

B. Fishbein, P. Foy, and L. Yin. 2021. TIMSS 2019 User Guide for the International Database, 2nd edition. Boston College, TIMSS & PIRLS International Study Center.

Vincent Liu, Ehsan Latif, and Xiaoming Zhai. 2025. Advancing education through tutoring systems: A systematic literature review. arXiv preprint arXiv:2503.09748.

B. Fishbein, M. Taneva, and K. Kowolik. 2025. TIMSS 2023 User Guide for International Database. Retrieved from TIMSS 2023 International Database.

Zhengyuan Liu, Stella Xin Yin, Geyu Lin, and Nancy Chen. 2024. Personality-aware student simulation for conversational intelligent tutoring systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 626– 642.

Adit Gupta, Jennifer Reddig, Tommaso Calo, Daniel Weitekamp, and Christopher J MacLellan. 2025. Beyond final answers: Evaluating large language models for math tutoring. In International Conference on Artificial Intelligence in Education, pages 323–337. Springer. Gail Hackett and Nancy E Betz. 1989. An exploration of the mathematics self-efficacy/mathematics performance correspondence. Journal for research in Mathematics Education, 20(3):261–273. Christof Imhof, Per Bergamin, and Stéphanie McGarrity. 2020. Implementation of adaptive learning systems: Current state and potential. Online teaching and learning in higher education, pages 93–115. Jaylen Jones, Lingbo Mo, Eric Fosler-Lussier, and Huan Sun. 2024. A multi-aspect framework for counter narrative evaluation using large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 147–168. Jarise Kaskens, Eliane Segers, Sui Lin Goei, Johannes Eh Van Luit, and Ludo Verhoeven. 2020. Impact of children’s math self-concept, math selfefficacy, math anxiety, and teacher competencies on math development. Teaching and teacher education, 94:103096.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, KaiWei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. Matthew D Marraffino, Bradford L Schroeder, Nicholas W Fraulini, Wendi L Van Buskirk, and Cheryl I Johnson. 2021. Adapting training in real time: an empirical test of adaptive difficulty schedules. Military Psychology, 33(3):136–151. Jan L Plass and Shashank Pawar. 2020. Toward a taxonomy of adaptivity for learning. Journal of Research on Technology in Education, 52(3):275–300. Alexander Scarlatos, Ryan S Baker, and Andrew Lan. 2025. Exploring knowledge tracing in tutor-student dialogues using llms. In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pages 249–259. William H Schmidt and Richard T Houang. 2012. Curricular coherence and the common core state standards for mathematics. Educational Researcher, 41(8):294–308.

Nermin Kibrislioglu. 2015. An investigation about 6th grade students’ attitudes towards mathematics. Procedia-Social and Behavioral Sciences, 186:64– 69.

Sahil Sharma, Puneet Mittal, Mukesh Kumar, and Vivek Bhardwaj. 2025. The role of large language models in personalized learning: a systematic review of educational impact. Discover Sustainability, 6(1):1–24.

Harsh Kumar, Ilya Musabirov, Mohi Reza, Jiakai Shi, Xinyuan Wang, Joseph Jay Williams, Anastasia Kuzminykh, and Michael Liut. 2023. Impact of guidance and interaction strategies for llm use on learner performance and perception. arXiv preprint arXiv:2310.13712.

Valerie Shute and Brendon Towle. 2018. Adaptive elearning. In Aptitude, pages 105–114. Routledge.

Gyeonggeon Lee and Xiaoming Zhai. 2025. Realizing visual question answering for education: Gpt-4v as a multimodal ai. TechTrends, 69(2):271–287. Yuho Lee, Taewon Yun, Jason Cai, Hang Su, and Hwanjun Song. 2024. Unisumeval: Towards unified, finegrained, multi-dimensional summarization evaluation for llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3941–3960. Chien-Chang Lin, Anna YQ Huang, and Owen HT Lu. 2023. Artificial intelligence in intelligent tutoring systems toward sustainable education: a systematic review. Smart learning environments, 10(1):41.

VJ Shute, SP Lajoie, and KA Gluck. 2000. Individualized and group approaches to training. Training and retraining: A handbook for business, industry, government, and the military, pages 171–207. Chuanxiang Song, Seong-Yoon Shin, and Kwang-Seong Shin. 2024. Implementing the dynamic feedbackdriven learning optimization framework: a machine learning approach to personalize educational pathways. Applied Sciences, 14(2):916. Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, and 1 others. 2024. Tofueval: Evaluating hallucinations of llms on topicfocused dialogue summarization. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:

Human Language Technologies (Volume 1: Long Papers), pages 4455–4480. Alessandro Vanzo, Sankalan Pal Chowdhury, and Mrinmaya Sachan. 2025. Gpt-4 as a homework tutor can improve student engagement and learning outcomes. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 31119–31136. Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169. Shuai Wang, Claire Christensen, Wei Cui, Richard Tong, Louise Yarnall, Linda Shear, and Mingyu Feng. 2023. When adaptive learning is effective learning: comparison of an adaptive learning system to teacherled instruction. Interactive learning environments, 31(2):793–803. Run Wen and Adam K Dubé. 2022. A systematic review of secondary students’ attitudes towards mathematics and its relations with mathematics achievement. Journal of Numerical Cognition, 8(2):295–325. Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, and Xuming Hu. 2025. A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11798–11827, Vienna, Austria. Association for Computational Linguistics. Yongan Yu, Alexandre Krantz, and Nikki G Lobczowski. 2025. From recall to reasoning: Automated question generation for deeper math learning through large language models. In International Conference on Artificial Intelligence in Education, pages 414–422. Springer. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, and 1 others. 2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Springer. Lanqin Zheng, Miaolang Long, Lu Zhong, and Juliana Fosua Gyasi. 2022. The effectiveness of technology-facilitated personalized learning on learning achievements and learning perceptions: A metaanalysis. Education and Information Technologies, 27(8):11807–11830. Wangchunshu Zhou, Qifei Li, and Chenle Li. 2023. Learning to predict persona information for dialogue personalization without explicit persona description. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2979–2991.

Detailed Rubric

This section comprehensively documents the rubric criteria, scoring hierarchies, and reference examples established in the evaluation framework. To ensure reproducibility and transparency, each entry below specifies the exact text used to prompt evaluators, the theoretical dimensions underpinning the metric (e.g., Cognitive Aspects vs. Motivational Aspects), and the precise logic required to assign values (e.g., −1 vs. 0 vs. 1). Additionally, we provide the canonical examples that define the boundary conditions for each score, presenting the material verbatim from the study’s design documents. A.1

Adaptivity

Q1. Does the response take the learner’s prior math knowledge into account? Cognitive aspects - Current knowledge (Plass and Pawar, 2020) Criteria: The response should match the learner’s stated math level. A response fails when it: 1) The content area is more than the learner’s. 2) The content area is out of this grade’s scope. Scoring & Examples: • 0 = No: Prompt: I just learned algebra in grade 8. Could you teach me this question "y = ab. Which is the value of y when a = 1 and b = 2" Response: Sure! Think about a x b in a higher-dimensional space... • 1 = Yes Q2. Does this response try to build a positive attitude in the learner? Motivational aspects - Learning attitude (Bernacki et al., 2021) Criteria: Definition of Learning Attitude in Math: Kibrislioglu (2015) defines attitude towards mathematics as liking or disliking of the subject; a tendency to engage in or avoid mathematical activities; a belief that one is good or bad at mathematics; and a belief that mathematics is useful or useless. Scoring & Examples: • -1 = No, it responds negatively • 0 = No

• 1 = Yes: Prompt: I don’t like math. Can you help me understand 2 × 3? Response: "I can help you with that...." Q3. Does this response intend to improve the student’s self-efficacy for completing this mathematics task? Motivational aspects - Self-efficacy (Plass and Pawar, 2020) Criteria: Math self-efficacy is a situational or problem-specific assessment of an individual’s confidence in her or his ability to successfully perform or accomplish a particular mathematical task or problem (Hackett and Betz, 1989). The response should help the student feel capable of solving the math question. Scoring & Examples: • -1 = No, it responds negatively • 0 = No • 1 = Yes: Prompt: I am not confident in math learning. Can you help me understand 2 × 3? Response: "Math can be tricky, but we can solve this together." Q4. Does the solution meet the Common Core State Standards at this grade level? Cognitive aspects - Difficulty (Marraffino et al., 2021) Criteria: The response aligns with the Common Core standard math knowledge scope and difficulty. Scoring & Examples: • 0 = No: Not followed the Common Core Standard • 1 = Yes: Prompt: I’m not sure I can solve 12 × 13. Can you teach me? Response: Of course! This is a Number and Operation question in Grade 4......? Q5. Is the solution at an appropriate difficulty level for the learner? Cognitive aspects - Difficulty (Marraffino et al., 2021) Criteria: The response should match the learner’s skill level. A response fails when the

explanation is too advanced, too complex, or uses concepts the learner is not ready for. Scoring & Examples: • 0 = No: Not provided a solution at an appropriate difficulty level • 1 = Yes: Prompt: What is 2 + 2? Response: It is 4 because adding two items to another two items makes four items. Q6. Does this response explain the meaning of this math question to help the learner understand the problem? Complexity Criteria: The response should explain the meaning of the math question to help the learner know its logic and better know how to solve it. A response fails when the response does not provide an explanation of the meaning of the math question. Scoring & Examples: • 0 = No: Not explained • 1 = Yes: Response: ...This question means...? Q7. Does this response provide an example aligned with this math question’s topic? Complexity Criteria: The response should give an example that matches the math topic in the question. A response fails when the example is unrelated, off-topic, or does not help illustrate the math idea being asked about. Scoring & Examples: • 0 = No: No example provided • 1 = Yes: Response: ...Here is an example... Q8. Does this response provide additional practice similar to this math question? Complexity Criteria: The response should give extra practice problems that match the math concept in the question. A response fails when the response does not provide extra practice. Scoring & Examples: • 0 = No: No additional practice provided

• 1 = Yes: Response: ...Let’s practice...

Scoring & Examples: • 0 = No : More than 50% incorrect

Q9. Does the response provide cues (e.g., step-by-step, signals of importance) to guide the learner? Complexity Criteria: The response should include helpful cues such as steps, hints, or pointing out important parts of the problem. A response fails when it gives the answer with no guidance or offers no structure to help the learner follow the process. Scoring & Examples: • 0 = No: No cues provided • 1 = Yes: Response: ...Let’s go step by step. First... A.2

Correctness

Q10. Does the response provide a correct answer to this math question? The response provides a correct answer. Criteria: The final answer provided by the model must be mathematically correct. Scoring & Examples: • 0 = No: Prompt: 2 + 2 =? Response: 5

• 1 = Mostly correct: Minor error that doesn’t affect logic • 2 = Completely correct A.3

Quality

Q13. Does the response have hallucinations or inaccurately extracted information? Criteria: The model responds with content that contains hallucinations and/or extracts inaccurate information from the provided prompt. Scoring & Examples: • 1 = No • 0 = Yes Q14. Does the response have failures in mathematical or logical reasoning? Criteria: The model in its responses makes errors in arithmetic calculations, statistical analysis, or other quantitative conceptual or numerical operations. Scoring & Examples: • 1 = No

• 1 = Yes: Prompt: 2 + 2 =? Response: 4

• 0 = Yes

Q11. Does the solution support and align with the answer?

Q15. Does the response fail to accurately and completely follow the prompt instructions?

Criteria: The key explanation and logic should clearly match the final answer. Scoring & Examples:

Criteria: The model doesn’t follow explicit or clearly implied instructions in the model input. This includes responding in the wrong language, ignoring the math exercise’s key information, or not adhering to the user’s requirement. Scoring & Examples:

• 1 = No • 0 = Yes: Prompt: Can you teach me how to solve 12 × 13? Response: The answer is 156. This is a number & operation math question... Q12. Does the response provide a correct solution to this math question? Criteria: The response should give the correct and accurate solution based on the math problem.

• 1 = No • 0 = Yes Q16. Is the response incoherent? Criteria: The model’s reasoning contains logical fallacies, contradicts itself within the same response, or reaches conclusions that don’t follow from the premises. It includes cases where

is available at https://platform.openai. com/docs/models.

the model presents information and then contradicts it later in the same response. Scoring & Examples:

2. Gemini-2.5-flash is provided by Google Gemini, with API documentation available at https://ai.google.dev/gemini-api/ docs.

• 1 = No • 0 = Yes

3. Qwen3-VL-30B-A3B-Instruct2 and Llama3.2-11B-Vision-Instruct3 are open-source base model weights obtained from Hugging Face (https://huggingface.co/).

Q17. Does the response provide irrelevant information? Criteria: The model in its responses includes information that doesn’t address the user’s query or goes beyond the scope of what was asked. This includes discussing tangential topics, adding unnecessary background, or focusing on aspects not central to the query. Scoring & Examples: • 1 = No • 0 = Yes Q18. Does the response have formatting, grammatical, or vocabulary errors? Criteria: This failure mode occurs when the model’s response contains errors in formatting, grammar, or word usage. Scoring & Examples: • 1 = No • 0 = Yes Q19. Is the response overly brief or lacking necessary detail? Criteria: The model provides a response that lacks sufficient detail or explanation to adequately address the query, given the complexity of the topic and the apparent depth of information requested. Scoring & Examples: • 1 = No

For large proprietary models (e.g., GPT-5), a one-time evaluation on 600 samples incurs a cost of approximately $6 for generating explanations. All open-source model evaluations are conducted on a system equipped with two NVIDIA RTX 4090 GPUs (32GB memory each). Overall, the modest computational requirements demonstrate that our evaluation protocol is accessible to researchers with limited computational resources, while still enabling comprehensive assessment of state-of-theart models.

Decoding configuration

To ensure comparability across models, we standardize the decoding parameters for all evaluated VLMs. Specifically, we set the maximum generation length to 512 tokens and use a temperature of 0.7 for all models. Generation terminates when the model reaches the token limit or produces an end-of-sequence token. These settings are chosen to allow sufficient space for step-by-step explanations while preventing excessively long outputs. The token limit is determined based on pilot annotations by human experts, which estimate the typical length required to provide a complete pedagogical explanation for the evaluated mathematics problems. Importantly, the same configuration is applied across all models to ensure that differences in response quality reflect model behavior rather than decoding differences.

• 0 = Yes

D B

Model Sources and Computational Cost

The models evaluated in this paper are obtained from the following sources: 1. GPT-5 and GPT-o1 are provided by OpenAI. The corresponding API documentation

Item-Level Robustness Analysis

To examine whether specific evaluation dimensions drive the reported findings, we conduct an itemlevel robustness analysis across the 19 evaluation 2

https://huggingface.co/Qwen/ Qwen3-VL-30B-A3B-Instruct 3 https://huggingface.co/meta-llama/Llama-3. 2-11B-Vision-Instruct

Item

Gemini

GPT-5

GPT-O1

Llama

Qwen

Instructional Adaptivity and Pedagogical Quality Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11

88.0 75.7 73.3 94.2 83.3 90.5 8.5 0.0 95.0 79.2 83.8

76.5 63.5 36.0 83.7 78.7 53.8 7.2 26.0 66.2 71.5 78.0

63.5 32.2 7.7 93.7 80.3 48.5 5.7 0.0 75.7 54.3 76.5

64.2 53.0 32.8 78.7 67.7 77.2 6.3 4.3 67.3 13.5 48.3

92.2 82.3 74.3 95.3 90.5 94.5 5.7 0.0 100.0 45.0 76.7

Failure Modes (Lower is Better) Q13 Q14 Q15 Q16 Q17 Q18 Q19

5.3 11.5 7.0 0.7 4.8 2.3 6.3

10.5 19.0 17.0 0.8 5.8 0.5 17.0

16.8 23.3 22.8 1.7 2.5 0.0 6.3

48.8 64.0 59.2 20.0 12.3 13.7 20.3

22.3 30.8 43.5 4.3 17.0 43.2 22.5

Table 3: Item-level mean percentages across learner profiles (N=6).

questions used in our human annotation framework. Each item captures a distinct aspect of instructional quality, including learner adaptation, motivational support, instructional scaffolding, correctness, and potential failure modes. For each item, we compute the mean percentage across the six learner profiles for each model. Table 3 reports the resulting item-level averages. The results reveal consistent patterns across items. Gemini and Qwen achieve the highest scores on most positive instructional dimensions, including prior-knowledge adaptation (Q1), motivational support (Q2–Q3), explanation quality (Q6), and instructional guidance (Q9). In contrast, GPT-O1 and Llama exhibit substantially higher failure rates in reasoning errors (Q14), instruction-following failures (Q15), and incoherence (Q16). Importantly, these patterns remain stable across evaluation items, suggesting that the reported model differences are not driven by a small subset of evaluation criteria. Instead, the relative strengths and weaknesses of models appear consistently across multiple pedagogical dimensions.

Human Annotation Details

To verify the reliability of our evaluation rubric, we conducted a human annotation study on a stratified subset of 120 model responses. Two expert annotators independently evaluated these instances, achieving an overall inter-rater agreement

of 87.11%. As detailed in Table 4, these results demonstrate strong consistency across different models and learner profiles, confirming the reproducibility of our scoring framework. Model

Agreement (%)

GPT-o1 GPT-5 Qwen3-VL-30B Gemini-2.5-Flash Llama3.2-11B-VL

100.00 89.47 84.21 73.68 73.68

Average

87.11

Table 4: Detailed inter-rater agreement metrics for each evaluated model on the annotated subset (N = 120). The overall average agreement across all models is 87.11%.

Statistical Analysis of Model Performance p(aj , bj | Dt+1 ) ∝

p(Rnew | aj , bj , θnew ) {z } |

Likelihood: how well new responses fit the current item parameters

p(aj , bj | Dt ) {z } |

Prior: learned from previous calibration data

We conducted a Repeated Measures ANOVA to statistically evaluate the performance differences between VLMs and test items. As detailed in Table 5, the analysis revealed a statistically significant main effect for the LLM (F = 41.39, p < .001), confirming that the choice of model significantly impacts performance quality. Additionally, we observed a significant main effect for Item (F = 721.61, p < .001) and a significant Item × LLM Interaction (F = 21.75, p < .001). This interaction indicates that the performance gap between models is not uniform; rather, it varies depending on the specific difficulty or type of math question presented.

Cases

Sum of Squares

Mean Square

Within Subjects Effects Item 593.60a 8a 74.201a 721.61a a a a Item ∗ LLM 71.56 32 2.236 21.75a Residuals 588.17 5720 0.103 a Mauchly’s test indicates sphericity assumption is violated (p < .05). Between Subjects Effects LLM 61.25 Residuals 264.49

4 715

15.312 0.370

41.39

p < .001 < .001

< .001

Table 5: Repeated Measures ANOVA Results. Analysis of Within-Subjects Effects (Item, Interaction) and BetweenSubjects Effects (LLM).

Figure 4 visualizes these descriptive trends. While some items (e.g., Item 7) proved uniformly difficult for all models, causing performance to converge near zero, other items (e.g., Item 1 and 9) exhibit distinct stratification, where higher-performing models separate clearly from lower-performing ones.

Figure 4: Descriptive Plots of Model Performance by Item. The x-axis represents the math items, and the y-axis represents the normalized performance score. The divergence of lines illustrates the Item × LLM interaction effect.

Statistical Analysis by Test Group

We extended our analysis to evaluate performance differences between distinct Test Groups using a Repeated Measures ANOVA. As presented in Table 6, we found a statistically significant main effect for

Cases

ηp2

Within Subjects Effects Item 593.64a 8a 74.205a 695.91a < .001 0.493 a a Item ∗ TestGroup 48.95 24 2.040a 19.13a < .001 0.074 Residuals 610.78 5728 0.107 a Mauchly’s test indicates sphericity assumption is violated (p < .05). Between Subjects Effects TestGroup 91.14 Residuals 234.59

3 716

30.381 0.328

92.73

< .001

0.280

ω2 0.412 0.022

0.088

Table 6: Repeated Measures ANOVA (Test Group). Comparison of Within-Subjects and Between-Subjects effects, including effect sizes (ηp2 and ω 2 ).

TestGroup (F (3, 716) = 92.73, p < .001), with a substantial effect size (ηp2 = 0.280). This indicates that the grouping factor accounts for 28% of the variance in between-subjects performance. Furthermore, the Item × TestGroup interaction was significant (F (24, 5728) = 19.13, p < .001), suggesting that performance gaps between groups are dependent on the specific item being solved. As illustrated in Figure 5, while all groups struggled similarly on the most difficult items (e.g., Item 7), the lower-performing group (bottom line) showed significantly higher divergence on Items 1, 3, and 5 compared to the clustered higher-performing groups.

Figure 5: Descriptive Plots by Test Group. Performance stratification across 9 items. The x-axis represents the items, and the y-axis represents the normalized score. One group (open circles) consistently underperforms compared to the others.

Case Profiles

To qualitatively illustrate the adaptive capabilities of the evaluated models, we present a case study of a Grade 4 measurement problem. Figure 6 displays the responses generated by the model for the same math question across three distinct learner profiles: High-performance, Intermediate, and Low-performance. This visual comparison demonstrates how the model adjusts its pedagogical strategy, ranging from direct calculation to scaffolded instruction, based on the learner’s stated confidence and prior knowledge.

G4Q5 Responses Across Learner Profiles

High Performance

Intermediate Performance

Here’s one way to figure it out step by step: From the first balance: 1 kg weight + 2 kg weight = pineapple So, the pineapple must weigh 3 kg. From the second balance: pineapple + 2 kg weight = large single weight Since the pineapple is 3 kg, 3 kg + 2 kg = 5 kg. So, that large single weight is 5 kg.

From the third balance: large single weight + 1 kg weight = watermelon That’s 5 kg + 1 kg = 6 kg.

Let’s read each balanced scale one by one and find the weight of each item: 1) On the first scale, the 1 kg weight plus the 2 kg weight together balance with the pineapple. This tells us the pineapple weighs 1 kg + 2 kg = **3 kg. 2) On the second scale, the pineapple (3 kg) plus the 2 kg weight balance with the large black weight on the right. So the large weight must be 3 kg + 2 kg = 5 kg. 3) On the third scale, the large weight (5 kg) plus the 1 kg weight balance with the watermelon on the right. That means the watermelon weighs 5 kg + 1 kg = 6 kg. Therefore, the answer is 6 kg.

Therefore, the watermelon weighs 6 kilograms.

Low Performance Let’s solve it step by step. Look at each balance (scale) one by one: 1) Top scale: Left side: 1 kg + 2 kg Right side: pineapple Because they balance, the pineapple must weigh 1 kg + 2 kg = 3 kg. 2) Middle scale: Left side: pineapple (3 kg) + 2 kg Right side: the large anvil These are equal, so the large anvil weighs 3 kg + 2 kg = 5 kg. 3) Bottom scale: Left side: large anvil (5 kg) + 1 kg Right side: watermelon They balance, which means the watermelon weighs 5 kg + 1 kg = 6 kg. Answer: 6 kg. -------------------------------------------------A GOOD WAY TO THINK ABOUT THIS Try giving each unknown object (pineapple, anvil, watermelon) a “mystery number.” Then use each balance to write an addition statement. Solving them step by step shows the weight of each mystery object.

Figure 6: Case profiles for three learner profiles in G4Q5.

Example Responses Across Context Conditions

Table 7 illustrates how an O1 model adapts its explanations across four context conditions (G1-G4) for a Grade 4 learner solving a balance scale problem. C

Prompt / Learner Profile

Model Response

Grade 4, low prior knowledge

A concise way to see this is: • From the first balance, the pineapple must weigh 1 kg + 2 kg = 3 kg. • The second balance shows that the 3 kg pineapple plus 2 kg equals the unlabeled weight, so that weight is 5 kg. • Finally, the third balance has that 5 kg weight plus 1 kg equals the watermelon’s mass. Hence, the watermelon weighs 6 kg.

Grade 4, low prior knowledge + partial context

Here’s one way to think this through step-by-step:

Grade 4, low prior knowledge + full context

Let’s solve it step by step:

Grade 4, low prior knowledge + full context + learner motivation

Let’s work through each scale, step by step:

1) Top scale: 1 kg + 2 kg = pineapple → 3 kg 2) Middle scale: pineapple (3 kg) + 2 kg = single heavy weight → 5 kg 3) Bottom scale: 5 kg + 1 kg = watermelon → 6 kg Tip: Writing equations for each scale helps solve the puzzle step by step.

1) Top scale: 1 kg + 2 kg = pineapple → 3 kg 2) Middle scale: pineapple (3 kg) + 2 kg = anvil → 5 kg 3) Bottom scale: anvil (5 kg) + 1 kg = watermelon → 6 kg GOOD WAY: Assign a “mystery number” to each object and solve each scale sequentially.

1) Top: 1 kg + 2 kg = pineapple → 3 kg 2) Middle: pineapple (3 kg) + 2 kg = unlabeled weight → 5 kg 3) Bottom: 5 kg + 1 kg = watermelon → 6 kg Therefore, the watermelon weighs 6 kilograms.

Table 7: O1 model responses across context conditions (G1–G4) for a Grade 4 learner on a balance scale problem. Increasing context leads to more step-by-step guidance and scaffolding.

Prompt Design Strategies

This section provides a comprehensive documentation of the evaluation rubric, presenting the specific criteria, scoring hierarchies, and illustrative examples utilized in the study. To ensure full transparency and reproducibility, the content below reproduces the definitions exactly as they appear in the evaluation framework. Each entry details the theoretical sub-dimension (e.g., Cognitive vs. Motivational aspects), the precise conditions required for each score (e.g., the distinction between a score of 0 and 1), and the reference examples that serve as ground truth for annotators. Learner Profile Type

Grade

Prompt

High-performance

I am a student from Grade 4, I like learning mathematics very much and I am very confident in mathematics. Now I have mastered all mathematics topics in grade 4, including number, measurement and geometry, and data. I got 615 in the TIMSS 2019 Math Test. Can you teach me this math question?

High-performance

I am a student from Grade 8, I like learning mathematics very much and I am very confident in mathematics. Now I have mastered all mathematics topics in grade 8, including number, algebra, geometry and measurement, and data and probability. I got 625 in the TIMSS 2019 Math test. Can you teach me this math question?

Intermediateperformance

I am a student from Grade 4, I like learning mathematics very much and I am confident in mathematics. Now I have mastered all mathematics topics in grade 4, including number, measurement and geometry, and data. I got 550 in the TIMSS 2019 Math Test. Can you teach me this math question?

Intermediateperformance

I am a student from Grade 8, I like learning mathematics very much and I am very confident in mathematics. Now I have mastered all mathematics topics in grade 8, including number, algebra, geometry and measurement, and data and probability. I got 550 in the TIMSS 2019 Math test. Can you teach me this math question?

Low-performance

I am a student from Grade 8, I don’t like learning mathematics and I am not confident in mathematics. Now I have mastered mathematics topics such as number and geometry in grade 8. I got 390 in the TIMSS 2019 Math Test. Can you teach me this math question?

Table 8: User prompts. User prompts across learner profiles (high-performance, intermediate-performance, and low-performance) and grade levels (grade 4 and 8).

Learning Goals & Content Areas

Content Area

Topic

Learning Goal

Whole Numbers

1. Demonstrate knowledge of place value (2-digit to 6-digit numbers); represent whole numbers with words, diagrams, number lines, or symbols; order numbers. 2. Add and subtract (up to 4-digit numbers), including computation in simple contextual problems. 3. Multiply (up to 3-digit by 1-digit and 2-digit by 2-digit numbers) and divide (up to 3-digit by 1-digit numbers), including computation in simple contextual problems. 4. Solve problems involving odd and even numbers, multiples and factors of numbers, rounding numbers (up to the nearest ten thousand), and making estimates. 5. Combine two or more properties of numbers or operations to solve problems in context.

GRADE 4 Number

Expressions, 1. Find the missing number or operation in a number sentence (e.g., 17 + w = 29). Simple 2. Identify or write expressions or number sentences to represent problem situations that Equamay involve unknowns. tions, and 3. Identify and use relationships in a well-defined pattern (e.g., describe the relationship Relationbetween adjacent terms and generate pairs of whole numbers given a rule). ships Fractions and Decimals

Measurement and Geometry

1. Recognize fractions as parts of wholes or collections; represent fractions using words, numbers, or models; compare and order simple fractions; add and subtract simple fractions, including those set in problem situations. (Fractions may have denominators of 2, 3, 4, 5, 6, 8, 10, 12, or 100.) 2. Demonstrate knowledge of decimal place value including representing decimals using words, numbers, or models; compare, order, and round decimals; add and subtract decimals, including those set in problem situations. (Decimals may have one or two decimal places, allowing for computations with money.)

Measurement1. Measure and estimate lengths (millimeters, centimeters, meters, kilometers); solve problems involving lengths. 2. Solve problems involving mass (gram and kilogram), volume (milliliter and liter), and time (minutes and hours); identify appropriate types and sizes of units and read scales. 3. Solve problems involving perimeters of polygons, areas of rectangles, areas of shapes covered with squares or partial squares, and volumes filled with cubes. Geometry

1. Identify and draw parallel and perpendicular lines; identify and draw right angles and angles smaller or larger than a right angle; compare angles by size. 2. Use elementary properties, including line and rotational symmetry, to describe, compare, and create common two-dimensional shapes (circles, triangles, quadrilaterals, and other polygons). 3. Use elementary properties to describe and compare three-dimensional shapes (cubes, rectangular solids, cones, cylinders, and spheres) and relate these with their two-dimensional representations.

Data

Reading, Interpreting, and Representing Data

1. Read and interpret data from tables, pictographs, bar graphs, line graphs, and pie charts. 2. Organize and represent data to help answer questions.

Data

Using Data to Solve Problems

Use data to answer questions that go beyond directly reading data displays (e.g., solve problems and perform computations using data, combine data from two or more sources, draw conclusions based on data).

Table 9: Content Areas & Learning Goals. TIMSS mathematical content areas and learning goals in Grade 4.

Content Area

Topic

Learning Goal

Integers

1. Demonstrate understanding of properties of numbers and operations; find and use multiples and factors, identify prime numbers, evaluate positive integer powers of numbers, evaluate square roots of perfect squares up to 144, and solve problems involving square roots of whole numbers. 2. Compute and solve problems with positive and negative numbers, including through movement on the number line or various models (e.g., losses and gains, thermometers).

Fractions and Decimals

1. Using various models and representations, compare and order fractions and decimals, and identify equivalent fractions and decimals. 2. Compute with fractions and decimals, including those set in problem situations.

Ratio, Proportion, and Percent

1. Identify and find equivalent ratios; model a given situation by using a ratio; divide a quantity according to a given ratio. 2. Solve problems involving proportions or percents, including converting between percents and fractions or decimals.

GRADE 8 Number

Algebra

Expressions, 1. Find the value of an expression or a formula given values of the variables. Opera2. Simplify algebraic expressions involving sums, products, and powers; compare tions, and expressions to determine if they are equivalent. Equations 3. Write expressions, equations, or inequalities to represent problem situations. 4. Solve linear equations, linear inequalities, and simultaneous linear equations in two variables, including those that model real life situations. Relationships 1. Interpret, relate and generate representations of linear functions in tables, graphs, or and words; identify properties of linear functions including slope and intercepts. Functions 2. Interpret, relate and generate representations of simple non-linear functions (e.g., quadratic) in tables, graphs, or words; generalize pattern relationships in a sequence using numbers, words, or algebraic expressions.

Geometry

Geometric Shapes and Measurements

1. Identify and draw types of angles and pairs of lines and use the relationships between angles on lines and in geometric figures to solve problems, including those involving the measures of angles and line segments; solve problems involving points in the Cartesian plane. 2. Identify two-dimensional shapes and use their geometric properties to solve problems, including those involving perimeter, circumference, area, and the Pythagorean Theorem. 3. Recognize and draw images of geometric transformations (translations, reflections, and rotations) in the plane; identify congruent and similar triangles and rectangles and solve related problems. 4. Identify three-dimensional shapes and use their geometric properties to solve problems, including those involving surface area and volume; relate three-dimensional shapes with their two-dimensional representations.

Data and Probability

Data

1. Read and interpret data from one or more sources to solve problems (e.g., interpolate and extrapolate, make comparisons, draw conclusions). 2. Identify appropriate procedures for collecting data; organize and represent data to help answer questions. 3. Calculate, use, or interpret statistics (i.e., mean, median, mode, range) summarizing data distributions; recognize the effect of spread and outliers.

Probability

For simple and compound events: a) determine theoretical probability (based on equally likely outcomes, e.g., rolling a fair die) or b) estimate the empirical probability (based on experimental outcomes).

Table 10: Content Areas & Learning Goals. TIMSS mathematical content areas and learning goals in Grade 8.

Record · ID 192420 · SHA-256 c5402511604d5b4c
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.