Tell Me Why: Designing an Explainable LLM-based Dialogue System for Student Problem Behavior Diagnosis Zhilin Fan1 , Deliang Wang3 , Penghe Chen1,2 , and Yu Lu1,2(B) School of Educational Technology, Beijing Normal University, Beijing, China 2 Advanced Innovation Center for Future Education, Beijing Normal University, Beijing, China [email protected] 3 Faculty of Education, The University of Hong Kong, Hong Kong SAR, China
arXiv:2604.22237v1 [cs.CL] 24 Apr 2026
1
Abstract. Diagnosing student problem behaviors requires teachers to synthesize multifaceted information, identify behavioral categories, and plan intervention strategies. Although fine-tuned large language models (LLMs) can support this process through multi-turn dialogue, they rarely explain why a strategy is recommended, limiting transparency and teachers’ trust. To address this issue, we present an explainable dialogue system built on a fine-tuned LLM. The system uses a hierarchical attribution method based on explainable AI (xAI) to identify dialogue evidence for each recommendation and generate a natural-language explanation based on that evidence. In technical evaluation, the method outperformed baseline approaches in identifying supporting evidence. In a preliminary user study with 22 pre-service teachers, participants who received explanations reported higher trust in the system. These findings suggest a promising direction for improving LLM explainability in educational dialogue systems. Keywords: Explainable AI · Large Language Models · Student Problem Behavior · Dialogue System · Teacher Trust
1
Introduction
Student problem behaviors, such as aggression and rule breaking, can negatively affect students’ psychological well-being and academic development [10]. Diagnosing these behaviors requires teachers to synthesize information about the student, family, and school context, identify behavioral categories, and plan interventions [1]. This process is challenging because it draws on knowledge from multiple domains, including psychology and pedagogy [2]. Recent studies suggest that fine-tuned large language models (LLMs) can support student problem behavior diagnosis through multi-turn dialogue and generate intervention strategies more consistent with expert practice [2,1]. However, these systems typically recommend strategies without explaining why they
2
Z. Fan et al.
are appropriate. Because teachers are professionally accountable for the strategies they choose [5], this lack of explanation may limit their trust in and use of the recommendations. To address this gap, we present an explainable dialogue system built on a fine-tuned LLM. For each recommended strategy, the system identifies teacherprovided dialogue evidence and generates a natural-language explanation grounded in that evidence. Building on recent explainable AI (xAI) attribution methods for LLMs [9,3], we adopt a hierarchical attribution method that first locates the most influential dialogue turn and then identifies the teacher sentence within that turn. We address two research questions (RQs): RQ1. To what extent can the hierarchical attribution method identify dialogue evidence that supports recommended strategies? RQ2. How do explanations influence teachers’ trust in the explainable diagnostic dialogue system?
2
System Design and Implementation
2.1
System Design
Figure 1 summarizes the explainable diagnostic dialogue system, which consists of a dialogue module for multi-turn diagnosis and intervention strategy recommendation, and an explanation module for identifying dialogue evidence and generating natural-language explanations.
Teacher
Dialogue History
Recommended Strategies Supporting Evidence User Interface Evidence
Explanations
Dialogue History Fine-tuned LLM
Recommended Strategies Dialogue Module
Hierarchical Attribution
Evidence Supporting Evidence
LLM-based Generation
Explanations
Explanation Module
Fig. 1: Overview of the explainable diagnostic dialogue system.
The system is shaped by two design decisions. First, because general-purpose LLMs lack the proactive diagnostic inquiry capabilities required for student problem behavior diagnosis [1], the dialogue module is built on a fine-tuned LLM that
Explainable LLM-based Dialogue System for Student Problem Behavior
3
can proactively collect and synthesize multifaceted student information from teachers. Second, because self-generated LLM explanations may sound plausible without faithfully reflecting the basis of a recommendation [11], the explanation module first identifies dialogue evidence via xAI attribution and then prompts the fine-tuned LLM to generate a natural-language explanation. 2.2
Dialogue Module Implementation
We fine-tuned Qwen2.5-3B-Instruct on the training set of the expert-annotated diagnostic dialogue corpus (3,636 examples) from Chen et al. [1] using LoRA1 , with hyperparameters selected via five-fold cross-validation. On the corresponding test set (409 examples) [1], the fine-tuned LLM achieved a macro-F1 of 0.71 for problem behavior type identification and a BERTScore of 0.79 for generated intervention strategies compared with authentic practices, providing a reasonable basis for the explanation module. Additional implementation details are available on GitHub2 . 2.3
Explanation Module Implementation
Turn-level attribution. Drawing on temporal likelihood dynamics from prior work [9], we first identify the most relevant dialogue turn to efficiently narrow down the evidence search in long multi-turn dialogues. Let rl denote the recommended intervention strategy and Ci the dialogue prefix up to turn i. We quantify the contribution of each turn by comparing the model’s support toward rl with and without the turn: gi = log Pθ (rl | Ci ) − log Pθ (rl | Ci−1 ), {z } | {z } | with turn i
i∗ = arg max gi . i
(1)
without turn i
A large gi indicates that adding turn i substantially increases the model’s support for the recommended intervention strategy. We therefore select the turn with the largest gi , denoted i∗ , as the most influential turn for subsequent sentence-level attribution. Sentence-level attribution. Within this turn, we focus on teacher utterances, i∗ which serve as the core diagnostic context, denoted ui∗ = {si∗ ,j }nj=1 . Following recent context attribution work [4,3], we score each sentence using two complementary signals: necessity and sufficiency. The first signal, Drop, captures necessity via a leave-one-out (LOO) ablation [4]. If removing a sentence from the teacher-provided context U lowers the likelihood of the recommended intervention strategy, the model is likely relying on that sentence as supporting evidence: Drop(si∗ ,j ) = log Pθ (rl | U ) − log Pθ (rl | U \ si∗ ,j ), {z } | | {z } full context
1 2
without sentence j
https://huggingface.co/docs/peft https://github.com/zhilinfan/AIED2026-Explainable-Dialogue-System
(2)
4
Z. Fan et al.
where U denotes the teacher-provided context in the selected turn. The second signal, Hold, captures sufficiency. If a sentence on its own still preserves support for the recommended intervention strategy, it contains standalone evidence rather than relying only on surrounding context: Hold(si∗ ,j ) = log Pθ (rl | si∗ ,j ) − log Pθ (rl | U ) . {z } {z } | |
(3)
full context
sentence j alone
Following [3], we combine the two signals by addition so that sentences are favored when they are both necessary in context and informative on their own: ϕi∗ ,j = Drop(si∗ ,j ) + Hold(si∗ ,j ) = log Pθ (rl | si∗ ,j ) − log Pθ (rl | U \ si∗ ,j ) . (4) | {z } | {z } sentence j alone
without sentence j
A sentence therefore receives a high score when it supports the recommended intervention strategy better on its own than the remaining context does without it. We select the top-ranked sentence as supporting evidence for the following explanation generation for teachers. 1 3
2
4
Fig. 2: System interface showing (1) the dialogue history, (2) recommended strategies, (3) the identified supporting evidence sentence, and (4) the naturallanguage explanation.
Because raw attribution scores are difficult for teachers to interpret, we convert the selected evidence into a natural-language explanation. Specifically, the system prompts the same fine-tuned LLM to generate an explanation conditioned
Explainable LLM-based Dialogue System for Student Problem Behavior
5
on the recommended intervention strategy and the selected evidence sentence. Fig. 2 shows a snapshot of the user interface.
3
Evaluation and Results
3.1
RQ1: Identifying Supporting Dialogue Evidence
We constructed an evaluation benchmark on the test set from Chen et al. [1]. Two educational experts annotated teacher-uttered sentences that support each recommended strategy, achieving substantial inter-annotator agreement (Cohen’s κ = 0.81). Disagreements were resolved through discussion, yielding a reliable ground truth. We compared the hierarchical attribution method against four baselines that directly score all teacher utterances across the dialogue: Drop+Hold, Leave-oneout [4], GradNorm, and Similarity. Following prior work [9], we report Hit@1, Hit@3, Hit@5, and MRR. As shown in Table 1, the hierarchical attribution
Table 1: Comparison of attribution methods for identifying dialogue evidence. Method
Hit@1
Hit@3
Hit@5
MRR
Hierarchical (Ours) Drop+Hold Leave-one-out GradNorm Similarity
0.778 0.736 0.641 0.578 0.511
0.856 0.825 0.724 0.701 0.670
0.945 0.911 0.850 0.804 0.738
0.803 0.766 0.719 0.686 0.639
method outperforms all baselines across metrics. Compared to using Drop+Hold alone without turn-level attribution, it improves Hit@1 from 0.736 to 0.778, suggesting the value of narrowing the search to the most influential turn. Drop+Hold also exceeds Leave-one-out, indicating the benefit of incorporating the sufficiency signal. The Hit@1 of 0.778 indicates that the top-ranked sentence matches the ground-truth evidence in over 77% of cases, supporting our design choice of selecting the top-1 sentence for explanation generation (Section 2.3). 3.2
RQ2: Preliminary Trust Study
To examine how explanations relate to teachers’ trust, we conducted a preliminary study with 22 pre-service teachers (mean age = 23.9, SD = 1.4), randomly assigned to a treatment group (n = 12) or a control group (n = 10). All participants first used the baseline system without explanations to diagnose five student cases and then completed a six-item trust scale in a five-point Likert format [8]. Next, both groups diagnosed five comparable cases: the treatment group used the explainable system, whereas the control group continued to use the baseline
6
Z. Fan et al.
system. The trust questionnaire showed good internal consistency (Cronbach’s α = 0.81 at pre-test and 0.77 at post-test). Because the trust scores were ordinal and the sample was small, we used non-parametric tests: the Wilcoxon signedrank test for within-group pre–post comparisons and the Mann-Whitney U test for between-group comparisons. Descriptively, the treatment group’s median trust increased from 17 (IQR 17–18) to 19 (IQR 18–20), whereas the control group remained at 18 (IQR 17– 19 to 16–19). Baseline trust did not differ significantly between groups (U = 50.0, p = .514). In the treatment group, trust increased from pre- to posttest (W = 3.5, p = .012), whereas the control group showed no significant change (W = 10.0, p > .05). A Mann-Whitney test on change scores indicated a between-group difference (U = 93.5, p = .026, r = .56), with larger trust gains in the explanation condition.
4
Conclusions and Future Directions
This study explores how to make LLM-based dialogue systems more explainable for teachers in the context of student problem behavior diagnosis. We present an explainable dialogue system, combining a dialogue module built on a finetuned LLM with an explanation module based on an xAI attribution method. For each recommended intervention strategy, the explanation module identifies the most influential dialogue turn, ranks teacher-uttered sentences within that turn by necessity and sufficiency, and finally uses the fine-tuned LLM to generate a natural-language explanation grounded in the selected evidence. We evaluate the approach through both a technical evaluation and a preliminary user study with 22 pre-service teachers. Results suggest that the hierarchical attribution method performs better than baseline approaches in identifying dialogue evidence associated with recommended strategies. Findings from the user study further suggest that providing explanations may improve teachers’ trust in the system. This pattern is broadly consistent with prior work on xAI-based explanations for teachers [12,6,13]. Together, these findings suggest that grounding explanations in dialogue evidence using xAI attribution methods may be a promising direction for developing more trustworthy LLM-based educational dialogue systems. However, several limitations should be noted. The user study involved a small sample of pre-service teachers and measured only self-reported trust, so the findings do not establish whether explanations improve downstream decision making or intervention quality. In addition, the current system explains only recommended intervention strategies, and the attribution method focuses on a single most influential dialogue turn, which may overlook evidence distributed across multiple turns. Future work should therefore evaluate the system with larger and more diverse teacher populations, extend explanation coverage to other diagnostic components, explore multi-turn and richer explanation forms such as counterfactual explanations [7], and examine generalizability to the full corpus and larger LLMs.
Explainable LLM-based Dialogue System for Student Problem Behavior
7
Acknowledgments This research is supported by the National Natural Science Foundation of China (No.62477003, No.62177009).
References 1. Chen, P., Fan, Z., Lu, Y.: Knowstu: Diagnosing students’ problem behaviors using fine-tuned llm and rag. IEEE Transactions on Learning Technologies 18, 1–15 (2025) 2. Chen, P., Fan, Z., Lu, Y., Xu, Q.: Pbchat: Enhance student’s problem behavior diagnosis with large language model. In: International Conference on Artificial Intelligence in Education. pp. 32–45. Springer (2024) 3. Chuang, Y.S., Cohen-Wang, B., Shen, Z., Wu, Z., Xu, H., Lin, X.V., Glass, J.R., Li, S.W., Yih, W.t.: Selfcite: Self-supervised alignment for context attribution in large language models. In: Forty-second International Conference on Machine Learning 4. Cohen-Wang, B., Shah, H., Georgiev, K., Madry, A.: Contextcite: Attributing model generation to context. arXiv preprint arXiv:2409.00729 (2024) 5. Fan, Z., Chen, P., Lu, Y.: Why did the ai suggest that? designing an explainable educational counseling system. In: International Conference on Artificial Intelligence in Education. pp. 321–335. Springer (2025) 6. Feldman-Maggor, Y., Cukurova, M., Kent, C., Alexandron, G.: The impact of explainable ai on teachers’ trust and acceptance of ai edtech recommendations: The power of domain-specific explanations. International Journal of Artificial Intelligence in Education 35(5), 2889–2922 (2025) 7. Khosravi, H., Shum, S.B., Chen, G., Conati, C., Tsai, Y.S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., Gašević, D.: Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence 3, 100074 (2022) 8. Merritt, S.M.: Affective processes in human–automation interactions. Human Factors 53(4), 356–370 (2011). https://doi.org/10.1177/0018720811411912 9. Qian, C., Wang, P., Liu, D., Yang, J., Guo, D., Tang, L., Mei, J., Ren, Q., Shao, S., Liu, Y., Fu, J., Shao, J., Hu, X.: The why behind the action: Unveiling internal drivers via agentic attribution (2026), https://arxiv.org/abs/2601.15075 10. Sutherland, K., Conroy, M., McLeod, B., Granger, K., Broda, M., Kunemund, R.: Preliminary study of the effects of best in class–elementary on outcomes of elementary students with problem behavior. Journal of Positive Behavior Interventions 22(4), 220–233 (2020) 11. Turpin, M., Michael, J., Perez, E., Bowman, S.: Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems 36, 74952–74965 (2023) 12. Wang, D., Bian, C., Chen, G.: Using explainable ai to unravel classroom dialogue analysis: Effects of explanations on teachers’ trust, technology acceptance and cognitive load. British Journal of Educational Technology 55(6), 2530–2556 (2024) 13. Wang, D., Chen, G.: Making ai accessible for stem teachers: Using explainable ai for unpacking classroom discourse analysis. IEEE Transactions on Education 67(6), 907–918 (2024)