arXiv CSopen access

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

Empirical Software Engineering manuscript No. (will be inserted by the editor)

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

arXiv:2606.22683v1 [cs.SE] 21 Jun 2026

Matthew Minish · Matthias Galster · Fabian Gilson

Received: date / Accepted: date

Abstract Context: Reflection is a fundamental skill in software engineering education, particularly in project-based courses where students learn through extended group work and need to develop their ability to reflect iteratively throughout their work. For students to benefit from reflection, their written reflections need to be assessed so that feedback can guide and improve their reflective practice. However, manually assessing written reflections to guide reflections is time-consuming. Furthermore, assessment of reflections often results in broad, non-specific feedback for a student to improve. Objective: This study builds on reflective writing frameworks to produce an eight-indicator scheme for assessing student reflections in software engineering. Furthermore, this study validates an automated classifier for assessing reflections against the framework, enabling scalable and structured feedback whilst reducing instructor workload. Method: We adapted existing reflection frameworks through iterative refinement to create our eight-indicator framework. Three annotators labelled student reflection texts, establishing moderate to reliable inter-rater agreement. We then trained and evaluated multiple encoder-only transformer modMatthew Minish University of Canterbury Christchurch, New Zealand E-mail: [email protected] Matthias Galster University of Bamberg Bamberg, Germany University of Canterbury Christchurch, New Zealand E-mail: [email protected] Fabian Gilson University of Canterbury Christchurch, New Zealand E-mail: [email protected]

2

Matthew Minish et al.

els and compared them with decoder-only large language models using zeroshot prompting. Results: The fine-tuned RoBERTa model achieved the strongest performance, substantially outperforming decoder-only models in both accuracy and speed. The classifier demonstrated human-level agreement on most indicators whilst enabling near-instantaneous classification. We provide two model variants optimised for different assessment priorities. Conclusions: Our fine-tuned encoder-only models enable efficient automated assessment of reflective writing. The framework and automated classifier offer a means to provide timely, structured feedback on student reflections in software engineering. Keywords Software engineering education · Self-reflection · Automated Classification · Assessment

1 Introduction 1.1 Problem and Motivation In software engineering, reflecting on one’s own behaviour is a widely used and established element of professional learning and continuous improvement (Babb et al. 2014; Guo et al. 2020; Rogers 2001). Through reflection, software engineers leverage experiences, knowledge, and emotions to gain further insights for future situations (Moon 2004). In educational settings, reflections are often part of experiential learning such as software engineering project courses (Morales-Trujillo et al. 2022). In industry, reflections are essential for continuous improvement and adapting how practitioners work (Babb et al. 2014). Indeed, reflection is not only one of the 12 principles of the agile software development (Beck 2001), but is also made explicit through everyday practices such as the daily stand-up meeting (Schwaber and Sutherland 2011) and retrospectives (Derby et al. 2006). Schön (1983) captures this in the concept of the ‘reflective practitioner’. Reflections cover various aspects, such as reviewing the outcomes of individuals’ (or teams’) processes and practices, but also considering the effects of behaviours or skills, and how these may be altered for different future outcomes (Moon 2004; Zimmerman 2002). Reflection itself is a skill that can be trained (Moon 2004). Indeed, teaching how to reflect, reason, and develop problem-solving skills is a key outcome in software engineering education and training (Turns et al. 2014; Chng 2018). While reflection can occur in many forms—such as through group discussions or reflection workshops—written reflections are most common in project-based software engineering education (Shekar 2007; Hazzan and Tomayko 2005; Prior et al. 2016). These typically take the form of journaling or formal reflection reports, though they may also be produced through specialised digital tools such as Microsoft Viva or GitHub’s Good Day Project (Kalliamvakou 2021), or through survey-based approaches (A. N. Meyer et al. 2021). To help students

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

3

improve the quality of their reflection in such writing, it is important to assess reflections and give feedback on reflections (ACM/IEEE 2014). However, such assessment is time-consuming, and the feedback tends to be broad and non-specific (Poldner et al. 2014)—for instance, commenting on the overall quality of a reflection without identifying which particular dimensions (such as reasoning, consideration of others’ perspectives, or future planning) a student has addressed or neglected. More targeted feedback that identifies specific dimensions of reflection to improve has been shown to better support the development of reflection skills (Ullmann 2019). More analytical approaches— where specific dimensions of a student’s reflection are identified and fed back individually—can support the development of reflection skills by directing students’ attention to dimensions they may have overlooked (Wulff et al. 2021; Ullmann 2019). For example, rather than receiving feedback such as “your reflection could be deeper,” a student might instead be prompted to consider the perspectives of their team members, or to articulate concrete reasons for why an experience unfolded as it did. Such feedback requires first identifying which dimensions of reflection (e.g., perspectives, reasoning) are present or absent in a student’s writing—a classification task that is the focus of this study. In this study, we focus on structured written reflections produced weekly by students during a software engineering project course. Students respond to three guiding prompts that ask them to consider what went well, what did not, and what they intend to do differently, reflecting on their own work, their interactions with their team, and their progress throughout the project. This format is described in detail in Section 4.2, and the data collection context is described in Section 5.1.1. To support more analytical assessment of written reflections, a number of reflective writing assessment frameworks have been proposed (see Section 2.3), while further studies have evaluated the feasibility of different approaches for automating the assessment of written reflections (see Section 2.4). These frameworks and automated assessment approaches have been evaluated in a number of different fields outside software engineering, e.g., health / dental students (Jung and Wise 2020; Jung et al. 2022), pharmacy students (Li et al. 2023; Liu et al. 2019a) and teacher education (Nehyba and Štefánik 2023; Solopova et al. 2023; Wulff et al. 2021, 2023). While recent research with computer science students may be closely related to software engineering (Alrashidi et al. 2020a, 2023; Chng 2018), the typically more individualised and shortterm nature of computer science education may differ from the unique context of software engineering project-based learning, such that existing assessment frameworks and automated approaches may not transfer to the collaborative, iterative, and professional practice dimensions central to software engineering reflection. In software engineering, students typically engage in extended, group-based practical work resembling real-world development contexts, where reflection occurs iteratively throughout the project lifecycle (Morales-Trujillo et al. 2022; Groeneveld et al. 2020). In such settings, the timeliness of providing feedback is critical as students continue working after reflecting—unlike the one-off assignments or final reflections common in other contexts where

4

Matthew Minish et al.

delayed or no feedback may be acceptable (Demmans Epp et al. 2019; Subramanian and Budhrani 2020). To date, no study has developed or evaluated an automated approach for classifying reflective writing specifically within a software engineering project context. Regarding software engineering practitioners, some works have investigated how to improve common practices involving reflection such as daily stand-up meetings (Stray et al. 2013, 2020) and retrospectives (Derby et al. 2006; Y. Dubinsky and O. Hazzan 2012). Prior studies have explored how developers benefit from self-reflection in their day-to-day work (Kalliamvakou 2021; A. N. Meyer et al. 2021), finding that purposeful self-reflection can increase developers’ awareness of their work habits and support self-improvement. However, these studies focus on improving the effectiveness of existing reflection practices, or understanding their effects on developers, rather than assessing the quality of reflective outputs, which is the focus of our work, situated in an educational context. 1.2 Paper Goals and Contributions This study investigates how to support the assessment of student reflections in software engineering and in particular project-based learning. Building on prior work in reflective writing assessment in other educational contexts (Chng 2018; Nehyba and Štefánik 2023; Alrashidi et al. 2023; Ullmann 2019), we adapt existing reflection indicators for the software engineering context and develop an automated classifier to identify indicators in reflection text. We address two problems: (1) The lack of consistent and relevant feedback on reflective writing, which often results in broad or vague feedback that is not actionable; (2) The effort to provide detailed, individualised assessments. By extending existing reflection frameworks (see Section 3) and leveraging recent advancements in natural language processing, we automate a classification of reflective writing, enabling scalable and structured feedback while reducing instructor workload. This study makes four key contributions: 1. An adaptation of existing reflective writing frameworks for software engineering project courses, with eight indicators that capture facets of reflection (e.g., reasoning or feelings). The adapted framework has been validated by human instructors who used the framework to identify indicators in reflection text from software engineering students. 2. An automated classifier built on fine-tuned transformer models to identify reflection indicators in reflection text. The classifier achieves human-level performance on reflection assessment based on the indicators in the reflective writing framework. 3. A comparison of encoder-only models (e.g., BERT variants) versus decoderonly models (e.g., LLMs) for this task, demonstrating the trade-offs between classification accuracy and computational efficiency. This is to show the practical applicability of the automated classifier.

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

5

4. Practical deployment guidelines for reflection classification, including two model variants optimised for different use cases based on the weight of indicators. The remainder of this paper is organised as follows. Section 2 provides background on reflection in software engineering education and reviews related work on automated assessment of reflective writing. Section 3 describes our methodology for annotating student reflections and developing automated classifiers, including both encoder-only and decoder-only models. In Section 4 we present our reflective writing framework with eight indicators adapted for software engineering project-based learning. In Section 6 we present the performance of our classifiers and their agreement with human annotators. Finally, in Section 7 we discuss the implications of our findings, limitations of the work, and provide recommendations for educators and directions for future research. We conclude in Section 8.

2 Background 2.1 Reflection as a Skill for Software Engineers In software engineering practice, reflection is part of professional development and continuous improvement (Babb et al. 2014; Galster et al. 2023). The concept of the ‘reflective practitioner’ (Schön 1983; Hazzan 2002) describes how professionals in practice use reflection to learn from experience, adapt, and improve their work. Reflection is also one of the twelve principles of the Manifesto of Agile Software Development (Beck 2001) and is embedded within agile software development through several established practices. Daily stand-up meetings (Schwaber and Sutherland 2011; Stray et al. 2020) provide opportunities for teams to reflect on progress and impediments, whilst retrospectives (Derby et al. 2006) prompt teams to reflect on their processes and identify improvements.

2.2 Reflection in Software Engineering Training In software engineering training, reflection is an important element of learning, e.g., in experiential learning such as software engineering project courses (Shekar 2007; Hazzan 2002; Hazzan and Tomayko 2005). Learning how to reflect meaningfully is non-trivial and at a tertiary level, students commonly lack depth in their reflections (Lim et al. 2022; Menekse 2020). To help students improve reflection skills, it is important to assess their reflective processes to provide feedback (Prior et al. 2016; ACM/IEEE 2014). Furthermore, improvements in reflection quality are significantly correlated with better learning gains and student performance (Embo et al. 2015; Mäeots et al. 2016; Chou and Chang 2011; Menekse 2020).

6

Matthew Minish et al.

The current understanding of reflective practice is largely built upon the work of four different theorists (Dewey (1910), Schön (1983), Kolb (2014), and Mezirow (1991)). Although each has presented their own theories for reflection, their ideas align in that reflection is a process of “intentionally making meaning of experiences in service of future action” (Turns et al. 2014). Despite this general consensus over what reflection is, and the processes by which it occurs, there exists no similar standard for how the quality of a student’s reflection might be assessed within a software engineering context.

2.3 Assessment Frameworks for Reflective Writing We can consider the assessment of reflection through two lenses (Moon 2004): firstly, by evaluating the effects or outcomes—typically on learning—observed as a result of the reflection (emphasizes reflection as an element of the wider learning process); secondly, by inspecting the process of reflection itself, and analysing the processes or artefacts (such as reflective writings) produced therein (supporting the development of reflection as a skill). An example of evaluating the effects or outcomes (learning) may be: if a student is asked to keep a learning journal in which they reflect on their ability to work with others, educators may assess reflection by whether the student’s teamwork skills (the learning outcome) improve, rather than by inspecting the contents of the learning journal (the process of reflection) itself. Alternatively, when assessing the processes and artefacts of reflection directly—commonly by considering reflective writing such as essays, reflection questionnaires, or learning journals—these artefacts are typically processed in relation to taxonomies for learning or cognition, such as the Structure of the Observed Learning Outcome (SOLO) taxonomy (Biggs and Collis 2014), or Bloom’s (Revised) Taxonomy (Krathwohl 2002). Additionally, some novel frameworks have been proposed for assessment of reflective writing within the context of computer science education more specifically (Ullmann 2015; Alrashidi et al. 2020b), having been adapted from established theories of reflection and learning, and designed to better support automation of assessment with machine learning approaches (Ullmann 2019). Such approaches supporting automated assessment are important as they may enable real-time and scalable feedback systems for students. Real-time feedback is normally not feasible in real educational settings given that assessment of reflection artefacts by humans is time consuming, and as a result any feedback that is given may be too holistic in nature, lacking specific guidance for each student (Poldner et al. 2014).

2.4 Automated Approaches for Assessing Reflective Writing Existing approaches to automatic reflection analysis can be broadly categorized into keyword-based and machine learning-based methods (Alrashidi and Joy 2020). Keyword-based methods, such as those proposed by Ullmann (2011,

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

7

2017), identify specific keywords in the text as indicators of reflection, e.g., words like ‘I’ and ‘Feel’ may be indicative of a student discussing their feelings, while words like ‘because’ and ‘but’ might be more indicative a student highlighting difficulties they experienced. Other approaches have been proposed which use Linguistic Inquiry and Word Count (LIWC) to identify words linked to conceptual reflective statement types—e.g., affective terms like ‘happy’ or ‘anxiety’, cognitive terms like ‘understand’, or, perceptual terms like ‘heard’ (Lin et al. 2016). Machine learning-based methods have also been developed which use classification algorithms trained on annotated datasets of reflection text to detect reflective indicators. Alrashidi et al. (2023) constructed a classifier for their Reflective Writing Framework (Alrashidi et al. 2020b) tailored for computer science education, applying natural language processing techniques like n-grams and part-of-speech n-grams, along with a Random Forest classifier. Nehyba and Štefánik (2023) applied multiple techniques, including both shallow classifiers such as Random Forest, and pre-trained deep learning models such as XLM-RoBERTa, for reflective writing analysis using Ullmann’s assessment framework (Ullmann 2019). Pre-trained deep learning models such as XLM-RoBERTa (and other BERT variants) use a transformer architecture (Devlin et al. 2019) and are encoder-only models, which typically need to be fine-tuned on task-specific data to perform text classification. Recently, decoder-only large language models (LLMs) such as GPT and Qwen have emerged as an alternative approach to encoder-only models (Naveed et al. 2025), demonstrating strong performance across various natural language processing tasks, including text classification (Galke et al. 2025; Kostina et al. 2025). Unlike encoder-only models that require task-specific fine-tuning, decoderonly models can perform classification through prompting approaches, where the model is provided with instructions and examples within the input prompt itself (Brown et al. 2020). This zero-shot or few-shot capability offers flexibility, as models can be adapted to new tasks without gradient updates or retraining (Wei et al. 2023). However, the computational demands of these larger models—which typically have billions of parameters—can limit their practical deployment, particularly when real-time responses are required. Recent comparisons suggest that whilst decoder-only models show promise for classification tasks, fine-tuned encoder-only models often achieve superior performance when sufficient labelled data is available for training (Galke et al. 2025; Wang et al. 2024).

3 Overview of Research Method This section provides a brief overview of our research method, which spans Sections 4 (the reflective writing framework) and 5.1 onwards (the annotation and classifier development). The detailed methodological descriptions follow in the subsequent sections. In this study we employ a multi-stage approach to de-

8

Matthew Minish et al.

velop and evaluate an automated reflection classifier for software engineering, shown in Figure 1. Our method comprises four main stages: 1. First, we developed and refined a reflective framework with eight indicators through iterative review by the authors (Section 4). This framework was developed specifically considering the nature of reflections in software engineering projects. 2. Second, we annotated a dataset of 1,518 student responses from a software engineering context using this framework, establishing inter-rater reliability through multiple rounds of annotation and guideline refinement (Section 5.1). 3. Third, we trained and evaluated multiple machine learning classifiers, comparing both encoder-only models (e.g., BERT, RoBERTa) and decoderonly models (GPT, Qwen) for automated classification (Section 5.2). By comparing multiple model architectures, we aimed to identify the most suitable approach for practical deployment in educational settings, where both classification accuracy and computational efficiency are important considerations. 4. Finally, we assessed the best-performing classifier’s agreement with human annotators to determine whether it achieved human-level performance (Section 6.3). The iterative nature of framework development and annotation ensured that our indicators could be applied consistently by human raters before attempting automated classification.

4 Reflective Writing Framework for Software Engineering 4.1 Software Engineering Context of Reflections The assessment of reflections depends on the context in which reflections are done, why, and how reflections are done. This context also influences the design of the reflective writing framework.

4.2 Context of Student Reflection In our context, reflections follow a structured format inspired by the questions commonly used in a Scrum Retrospective (Schwaber and Sutherland 2011), but are performed weekly during each sprint while the sprint work is still in progress. Unlike a Scrum retrospective, which occurs at the end of a sprint, or a final reflection submitted after a project has concluded, these reflections allow students to adapt their behaviour within the same sprint. In this sense, they function as a form of reflection-in-action (Schön 1983): 1. What did you do well this week? 2. What didn’t you do so well this week?

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

9

Fig. 1 Overview of research method in four stages

10

Matthew Minish et al.

3. What will you do differently in future? These questions attempt to target distinct aspects of the reflective process. In the first question, we encourage students to acknowledge their successes and consider how their practical experiences connect with theoretical knowledge—a component we view as important in project-based learning. Drawing on Zimmerman’s (Zimmerman 2002) account of self-regulated learning—where self-satisfaction with one’s own behaviour is identified as a driver of motivation and engagement—we aim for students to consider both technical and personal / social aspects of their software development work, with the intention of fostering such feelings of self-satisfaction. In the second question, we encourage students to reflect on the negative aspects of their experiences, with the intention of developing students’ willingness to reflect openly on failures and shortcomings. This question guides students’ reactions towards being more adaptive—willing to change and learn— rather than defensive, where students might withdraw from learning opportunities. The third question attempts to focus on future planning, where we ask students to set concrete goals and justify their chosen actions. This approach aligns with what we understand to be the ‘goal setting’ and ‘strategic planning’ aspects of the forethought phase of self-regulated learning (Zimmerman 2002). We envision reflections of varying lengths in a journaling approach, with responses ranging from a couple of brief sentences, to more comprehensive analyses over 2–3 paragraphs. 4.3 Design of Reflective Writing Framework The reflective writing framework (RWF) we use for assessing reflective writing of software engineering students was adapted from earlier works by Ullmann et al. (2012); Ullmann (2015, 2019), and Alrashidi et al. (2020b, 2023). These frameworks were created for assessing reflective writing in computer science education (Alrashidi et al. 2020b), and we have adapted them to more closely suit our specific software engineering project-based context, with a reflection format constructed of three parts (see Section 4.2). Ullmann et al. (2012) initially proposed five ‘indicators of reflection’, whose definitions have been refined and changed slightly since their initial proposal. Ullmann’s original elements were: Description of an experience, Personal experience, Critical analysis, Taking perspectives into account, and Outcome (Ullmann et al. 2012). More recently, Alrashidi et al. have built upon this work by proposing a new framework with seven indicators of reflection—Descriptive, Understanding, Feelings, Reasoning, Perspective, New Learning, and Future Action—in addition to defining three ‘reflection levels’—1 (non-reflective), 2 (reflective), or 3 (critically reflective)—consistent with the levels of reflection defined by Wong et al. (Wong et al. 1995). In this study we adapt the indicators proposed by Ullmann et al. (2012); Ullmann (2015, 2019) and Alrashidi et al. (2020b, 2023) for the software engi-

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

11

neering project context, with annotation criteria developed iteratively through multiple rounds of author review and refinement, as shown in Figure 1. In Section 4.4 we describe the indicators of reflection present in this framework, including—where appropriate—how they deviate from the indicators proposed by Alrashidi et al. and Ullmann. We note that the indicators in our framework—both in their original forms and in our adaptation—describe dimensions of reflective writing rather than properties specific to software engineering. The framework’s value in our context therefore lies not in encoding software-engineering-specific concepts, but in providing a structured means to identify which facets of reflection a student has addressed, so that feedback can be targeted to facets they may have overlooked. The practical use of this targeted feedback is discussed in Section 7.4.3. While the indicators describe facets of reflective writing rather than properties specific to software engineering, they align with established themes in the software engineering literature on reflection. Critical analysis of one’s actions and their consequences (Reasoning, Hindsight) is central to the “reflective practitioner” framing in software engineering (Schön 1983; Hazzan 2002); reflection on perspectives within a team (Perspective) is a foundational practice in agile retrospectives (Derby et al. 2006) and daily stand-ups (Stray et al. 2020); and the formulation of intentions for future practice (Future Intention) underpins continuous improvement in agile software development (Babb et al. 2014). The indicators thus support analysis of reflective writing that is consistent with how reflection is understood and practised in software engineering contexts.

4.4 Indicators of Reflection Each of the eight indicators described below represents a specific facet of reflection. The most significant change in our proposed indicators when compared to the most recent framework by Alrashidi et al. (2023) is that we have broken down the ‘Future Action’ indicator into two separate indicators. The ‘Future Action’ in (Alrashidi et al. 2023) considers not just future intention by the writer—e.g., “Next week I will...”—but also a retroactive analysis where “the writer would, given the same circumstances again, intentionally do something differently”. In our context, students answer three questions, two of which are explicitly retrospective in nature, and the third of which is explicitly prospective. Beyond this structural alignment, the split reflects a conceptual distinction that is particularly important in our context: because reflections are performed weekly during an ongoing project, students have the opportunity to act on their reflections while work is still in progress. Distinguishing between retrospective analysis (how the student would have approached a past situation differently) and prospective planning (what the student intends to do next) allows us to separately identify a student’s capacity for critical selfevaluation and their ability to translate that evaluation into concrete plans

12

Matthew Minish et al.

for action. For this reason, we split the ‘Future Action’ indicator into both its forward looking component—which we call Future Intention to avoid confusion with the original term—and its retrospective component—which we call Hindsight. Below we list the eight indicators of our framework, each with examples drawn from real student reflections. Further examples can be found alongside annotation guidelines as supplementary material (Minish et al. 2025): 1. Description. The writer provides a factual recounting of events they have witnessed or experienced. This does not require any degree of analysis or deeper understanding, and as such is not indicative of reflection, but it does provide the context needed for reflection. For example: “I fixed a bug before submission”; or, “We ended up having major merge conflicts”. 2. Understanding. The writer shows understanding of a goal, concept or personal experience, by interpreting it subjectively. This involves evaluating its significance, considering how it aligns with their desires or assumptions, and exploring what it means for them. The focus is on personal insights and high level evaluations, such as whether an experience was positive or negative, and why it happened, without delving into specific, concrete details. For example: “Teamwork was great, especially teammates did not hesitate to help each other”; or, “Didn’t code fast enough because I take so long doing research”. 3. Feelings. The writer identifies their feelings and emotions. This may include the students reported self-satisfaction with their behaviour (or lack thereof), in addition to experienced motivation or frustration. Students may discuss feelings like pride or happiness about their, or their team’s, successes. Similarly, students may express frustration, annoyance, or disappointment arising from their, or the team’s, actions. Such feelings may also arise from the emotional component of empathy (i.e., when emotions may be ‘contagious’ between individuals (de Waal 2008)), but this must still be categorised by the individual describing their own feelings or emotions. The other major component of empathy, cognitive empathy—where individuals can understand others’ perspectives or mental states without experiencing their emotions for themselves (Rogers et al. 2007)—corresponds to the ‘perspective’ indicator below. For example: “I feel a lot more motivated”; or, “I feel like I have let down my team”. 4. Reasoning. The writer makes an in-depth analysis, leading to a deeper understanding of the experience by explaining in terms of concrete causes, effects, or consequences. This is distinct from the Understanding indicator where the writer subjectively considers the experience from their own perspective, as this indicator emerges through a critical and objective analysis of concrete actions, events, or results. Similarly, this is distinct from the Hindsight and Future Intention indicators, in that this indicator extends only to analysing concrete causes for why something happened, and does not include the drawing of any conclusions from the experience. Such conclusions that take the form of considering how the situation could have

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

13

been better approached would fall under Hindsight, while specific actions the writer intends to take in the future correspond to Future Intention. For example: “I also took a dive into learning how thymeleaf fragments work which made for a lot less boilerplate code”; or, “If I plan out my time better in the future I believe I’d be able to achieve more when it comes to tasks”. 5. Perspective. The writer acknowledges, considers, or discusses alternative perspectives. These perspectives may belong to their peers, team members, or instructors, and may have arisen through observation, discussion, or feedback. The perspectives can relate to a variety of topics, such as problems, experiences, solutions, emotions or challenges faced by others. For example, the writer might reflect on how a teammate struggled with a task, how a proposed solution was received by others, or how their actions impacted team morale. This indicator may arise as the result of the cognitive component of empathy (Rogers et al. 2007), where the writer understands another’s perspective. However, any emotions that the writer experiences themselves as a result of this empathy, would correspond to the ‘feelings’ indicator. The perspectives must originate from someone other than the writer (e.g., a peer’s opinion, a team member’s difficulty, or an instructor’s feedback), but they do not need to be in disagreement with the writer’s own views. This indicator captures the writer’s ability to step outside their own viewpoint and consider how others perceive or are affected by a situation, decision, or outcome. For example: “After talking to other members in my team who agreed with my solution I see my mistake”; or, “She had a hard time understanding as she hadn’t yet worked with this part of the code base”. 6. New Learning. The writer describes what they have learned from the experience, in terms of either personal or professional skills. This may involve a better understanding of theory by putting it into practice, acquisition of new skills or abilities, or changes in perspective brought about by analysis of their experiences. For example: “This gave me more knowledge about the database”; or, “I’ve learned to communicate with my team in an effective way”. 7. Hindsight. The writer considers a past experience, and describes how— based on their new understanding as a result of reflection on the experience— they could, should, or would have approached the situation differently. This is commonly expressed through phrases such as “should(n’t) have”, or “would(n’t) have”. For example: “I feel I should have helped more with the integration/unit tests”; or, “I could’ve been more vocal in saying my opinions more right away instead of waiting for a moment where everyone is quiet”. 8. Future Intention. The writer indicates that they intend to do something and plans for the future. Here, “something” can be any future action, for example: trying a new process or changing their behaviour; talking to a peer; or, working on certain tasks. For example: “In the future, before sprint planning, I will spend extra time reading over the product backlog”;

14

Matthew Minish et al.

or, “I should manage my time better, and when I complete a task, I should pick up another or check if the team needs any help”. Indicators are not mutually exclusive, and reflection texts may exhibit any combination of indicators. Consider the following example snippet from a reflection: I am proud of the work I did on the project. For the button to add images, I used CSS for the first time to correctly position and make the button transparent. Here we can see two indicators of reflection present. In the first sentence “I am proud of...” the student expresses pride in their work, meaning that indicator 3 – ‘Feelings’ is present. In the second sentence, the student simply reports something that they did, meaning that indicator 1 – ‘Description’ is present. In this example the indicators apply to one sentence each, but indicators may also apply multiple times within a single sentence. Consider the following example: This week I learned how to mock dependencies in unit tests, which is something I really should have done in the tests I wrote last week, but now that I know how to I’ll include it when testing the feature I am currently working on. In this example we can see that the student begins by referencing some new learning that has taken place, meaning that indicator 6 – ‘New Learning’ is present. The student then includes both indicators 7 – ‘Hindsight’, and 8 – ‘Future intention’, when talking about something they should have been doing previously, and something they intended to do, respectively.

5 Development and Evaluation of Classifier 5.1 Annotation of Training Data for Reflection Classifier To begin training an automated classifier for reflection indicators, we manually annotated a set of 1,518 responses (totalling 6,704 sentences). Each of the 1,518 annotated texts corresponds to a single response to one of the three reflection prompts described in Section 4.2, rather than a complete weekly reflection (which contains three such responses). The data collection context is described in Section 5.1.1. Only blank reflections were filtered out; no further filtering was performed on the dataset to ensure that our annotation captured the reality of reflections produced by students in a real course. Summary statistics for the annotated dataset—including word counts, indicator frequencies, and the distribution of indicators per response—are presented in Table 1. Following guidelines for qualitative content analysis (Campbell et al. 2013), the first author began by creating annotation guidelines for each of the eight indicators of reflection, to be applied to each of the three answers (full text)

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

15

Table 1 Summary statistics for the annotated dataset. Word counts and indicators per response are reported as mean (SD). Indicator frequencies show the number (and percentage) of responses in which each indicator was identified. The distribution of indicators per response shows how many responses contained 0, 1, 2, etc. indicators. Q1 (did well)

Q2 (did not do well)

Response characteristics Words per response 88.3 (73.7) Indicators per response 2.94 (1.50) Indicator frequencies (% of responses) Description 68.6% Understanding 81.6% Feelings 52.0% Reasoning 26.8% Perspective 23.1% New Learning 39.0% Hindsight 0.7% Future Intention 2.3% Distribution of indicators per response 0 indicators 6.9% 1 indicator 11.2% 2 indicators 19.5% 3 indicators 25.0% 4 indicators 20.7% 5 indicators 14.2% 6 indicators 2.5% 7 indicators 0.0% 8 indicators 0.0%

Q3 (do differently)

Overall

81.7 (75.2) 2.70 (1.57)

74.1 (78.2) 2.19 (1.30)

82.3 (75.6) 2.66 (1.51)

42.8% 81.0% 20.9% 41.5% 21.2% 5.4% 36.1% 20.7%

5.1% 63.2% 7.5% 33.6% 14.9% 2.7% 3.5% 88.5%

43.1% 76.8% 29.1% 34.1% 20.4% 17.2% 14.9% 30.6%

8.5% 13.1% 25.7% 24.3% 15.7% 8.1% 3.1% 1.2% 0.2%

9.6% 20.3% 30.9% 26.4% 8.3% 2.9% 1.3% 0.3% 0.0%

8.2% 14.2% 24.7% 25.1% 15.7% 9.1% 2.4% 0.5% 0.1%

in a reflection, including: the criteria annotators use to determine whether each indicator was present in a piece of reflection text; exemplar statements of each indicator taken from real reflections; common terms or words which typically do correspond with the presence of an indicator; and confounder terms or words which superficially may appear to correspond to an indicator, but actually do not. These annotation guidelines are included as supplementary material (Minish et al. 2025). The initial set of guidelines produced by the first author was reviewed by the second and third authors, and over several iterations the definitions, exemplars, and common / confounder phrases were refined to remove subjectivity and ambiguity between indicators where possible. This iterative refinement followed standard procedures for qualitative coding, as recommended by McDonald et al. (2019). Inter-rater agreement was then calculated by having three annotators (the first author, and two teaching staff of the project course the reflections were performed in) annotate 100 reflections texts using the criteria, and calculating the Krippendorff’s alpha scores (Krippendorff 2019) separately for each indicator, as this is a multi-label classification setting where indicators are completely independent. After the first round of annotation, all disagreements were gathered and discussed by the three annotators. For each disagreement, a final decision was agreed upon, and the annotation guidelines for indicators with lower agreement

16

Matthew Minish et al.

scores were further refined to remove ambiguities and improve consistency. A second round of 100 annotations was then conducted to re-calculate agreement after these refinements. Following this process, annotators worked independently on different datasets to annotate a total of 1,518 responses for training and evaluating the classifier. As a reliability check to verify annotation quality remained acceptable while annotators worked independently, one of every five such items was reannotated by the first author and Krippendorff’s alpha calculated. Inter-rater agreement scores for the reflection annotation task are presented in Table 2. A Krippendorff’s alpha greater than 0.8 typically indicates reliable rating, while 0.67–0.79 is often treated as a lower bound for tentative conclusions (Krippendorff 2019). Values below 0.67 are indicative of poor agreement, suggesting that the coding scheme is flawed, or is not being applied consistently by raters, while a value of 0 indicates no agreement better than chance (Krippendorff 2019). After the second round of agreement calculation, all Krippendorff’s alpha scores were above the lower bound, and many were in the reliable range of 0.8+. Between human annotation rounds, inter-rater agreement improved for all indicators except future intention, which decreased from 0.96 to 0.89 (which still indicates high agreement). This decrease likely reflects natural variance in the annotation process where agreement could go up or down slightly when human annotators are involved (Krippendorff 2004). Still, the inter-rater agreement for future intention remains adequate, with the highest agreement score among all indicators. The agreement scores calculated from the reliability checks on independently labelled similarly increased, possibly as a result of annotators gaining more practice in applying the framework.

Table 2 Inter-rater agreement for indicators by Krippendorff’s alpha scores. Version 1 refers to the initial annotation guidelines after iterative refinement by the authors, while version 2 of the annotation guidelines includes changes made when reconciling disagreements after the first round of annotation using version 1. Also included are the agreement scores from the reliability check of independently labelled data, as described in Section 5.1. Indicator Description Understanding Feelings Reasoning Perspective New Learning Hindsight Future Intention

Version 1

Version 2

0.69 0.74 0.62 0.64 0.33 0.69 0.71 0.96

0.87 0.84 0.82 0.72 0.68 0.79 0.85 0.89

Version 2 (independent) 0.88 0.88 0.93 0.80 0.76 0.87 0.90 0.88

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

17

5.1.1 Data Collection Context The reflections were produced by 91 students at the University of Canterbury, New Zealand, taking a third-year software engineering project course in 2024. In the course, students worked in teams of six to eight on a year-long project following Scrum practices, with the project spanning six sprints of 2–4 weeks each over two semesters. Students were required to attend formal sessions in person on campus, though they were free to perform their development work either on campus or from home. Reflections were submitted weekly through a dedicated “Weekly Self-Reflections” interface in the ScrumBoard project management tool (Minish et al. 2024) used by the course. The timing of submission within a given week was at students’ discretion, though the typical pattern was to complete reflections on the Friday or over the weekend. Students received basic written guidance on how to write reflections through the course’s student guide, but no further training (such as workshops or worked examples) was provided. Reflections were graded summatively at the end of each sprint, where teaching staff assigned a single traffic light grade (Red, Yellow, or Green) for the overall quality of a student’s reflections within that sprint. Lateness or missing reflections could negatively affect this grade. As the unit of analysis for our annotation was the individual response to a single prompt rather than the sprint-level aggregate, these grades were not used in the present study. Across the 91 students, the number of annotated responses per student ranged from 11 to 30, with a mean of 16.7 (SD = 3.8, median = 16). Some variation is expected, as students could submit reflections of varying lengths and did not always answer all three prompts in every weekly reflection.

5.2 Reflection Classifier For automated classification of reflections we explored two types of pre-trained language models: encoder-only models such as BERT and its variants; and, decoder-only large language models (LLMs) such as GPT or Qwen. Encoder-only models like BERT have demonstrated promising performance on text classification tasks when fine-tuned with task-specific data (Sun et al. 2020), with studies showing the fine-tuning allows such models to achieve stateof-the-art results across various NLP benchmarks (Sun et al. 2020). In software engineering contexts, encoder-only models have been successfully applied to classify software requirements (Hey et al. 2020; Kaur and Kaur 2023), identify bug report priorities (Izadi et al. 2022), and detect code smells (Alazba et al. 2024). However, those models have not been used to classify reflective content in a software engineering context—see Section 2.4 for their uses in reflection classification in other contexts. Meanwhile, decoder-only models like GPT have shown excellent performance on few-shot learning tasks, scoring highly on many NLP benchmarks without gradient updates or fine-tuning (Brown et al. 2020). In software en-

18

Matthew Minish et al.

gineering, LLMs have been used for classification of software requirements (Binkhonain and Alfayez 2025), software vulnerabilities (Lu et al. 2024), and issues in open source projects (Aracena et al. 2025). While LLMs remain largely unexplored as a means of classifying or assessing reflective texts, recent work in a computer science education context has suggested that LLM-guided reflections may be comparable to conventional reflections methods such as reflection questionnaires (Kumar et al. 2024). Recent works comparing the two model architectures suggest that while fine-tuned encoder-only models generally outperform the prompting of decoderonly models when sufficient labelled data is available, generative decoder models like GPT can still match or approach a fine-tuned encoder’s performance, particularly in low-data scenarios (Wang et al. 2024). To experiment with different models, we used the following: – The encoder-only models were first downloaded from their public Huggingface1 repositories, and then fine-tuned on our annotated dataset with PyTorch2 . – The decoder-only models were also downloaded from Huggingface, and hosted locally via Ollama3 . This was all performed on the same machine, running Linux Mint 22, with 32GB of RAM, an Intel i7-14700 CPU, and Nvidia RTX 4070 12GB GPU. More information, including code artefacts used for training, hyper-parameter tuning, and automating the decoder-only models’ classification, are included as supplementary material (Minish et al. 2025). 5.2.1 Encoder-only Models For encoder-only models, we explore BERT and its variants (e.g., RoBERTa, ALBERT, DeBERTa). These models are pre-trained on massive corpora and achieve strong performance on text classification tasks after fine-tuning (Devlin et al. 2019; Liu et al. 2019b; He et al. 2021). Our data set (see Section 5.1) consists of 1,518 annotated responses (totalling 6,704 sentences), comparable to related works: 5,080 sentences across 77 essays (Ullmann 2019); 7,128 sentences from 300 journals (Nehyba and Štefánik 2023); 1,113 sentences from 174 students (Alrashidi et al. 2023). Rather than training a new model from scratch—which would require learning both general language understanding and our specific classification task from our limited dataset—we leverage models already trained on general natural language and fine-tune them to focus specifically on our indicators. This approach has been used successfully in related works for classifying similar indicators of reflective writing in similarly sized training datasets (Cavalcanti et al. 2023; Nehyba and Štefánik 2023). 1

https://huggingface.co/ https://pytorch.org/ 3 https://ollama.com/ 2

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

19

Since each reflection text can exhibit multiple reflection indicators simultaneously (see Section 4.4), we frame this as a multi-label classification problem, and used a Binary Cross Entropy (BCE) loss function (Tsoumakas et al. 2010). This treats each indicator as an independent binary prediction—consistent with how our annotators assessed each indicator separately during labelling (see Section 5.1). To handle class imbalance of indicators in our dataset, we applied inverse frequency weighting to our loss function. Specifically, each class was weighted inversely proportional to its frequency in the training data, such that rarer indicators received proportionally higher weights during training. For example, if some indicator A appeared in 100 training samples, while indicator B appears in only 50 samples, indicator B would be weighted twice as strongly in the loss calculation. This ensures the model does not simply learn to predict only the most common indicators while ignoring rarer ones (King and Zeng 2001). To evaluate the classifier’s performance, we employed 10-fold cross-validation (Kohavi 1995). The complete dataset of 1,518 annotated responses was randomly divided into 10 equal folds. For each fold, we trained the model on 9 folds and evaluated on the remaining fold, repeating this process 10 times. This ensured that every reflection text was used for both training and testing, reducing the impact of any particular train-test split. We compared the performance of five encoder-only models (BERT, DistilBERT, ALBERT, DeBERTa, and RoBERTa) using identical training configurations and cross-validation folds to ensure fair comparison. For all five models we used the base-sized variants available on Huggingface (bert-baseuncased, roberta-base, distilbert-base-uncased, albert-base-v2, and microsoft/deberta-base), to keep the comparison consistent across architectures and within the computational budget available for fine-tuning and hyperparameter tuning. These five specific models were chosen for their widespread usage in text classification settings, beginning with the foundational model BERT (Devlin et al. 2019), and including its common variants with optimisations to: training approach (Liu et al. 2019b); computational efficiency (Sanh et al. 2020; Lan et al. 2020); and, architecture improvements (He et al. 2021). Based on the results of this comparison (detailed in Section 6.1), we selected the best-performing model and conducted hyperparameter tuning using the Optuna framework (Akiba et al. 2019) to further optimise classification performance. During hyperparameter tuning we again used 10-fold cross-validation for evaluation during the tuning process. We optimised five key parameters: learning rate (log-uniform distribution between 1 × 10−6 and 1 × 10−4 ), training epochs (2-6), batch size (4, 8, 12, 16, 24, 32, 36, 48, or 64), positive class weight scaling (0.0-1.0), and classification threshold (0.3-0.7). The positive class weight scaling parameter controls the degree to which the inverse frequency weights (described above with regards to handling class imbalance) are applied: a value of 0.0 applies no class weighting (treating all classes equally), while 1.0 applies the full inverse frequency weighting, with intermediate values allowing for partial weighting adjustments.

20

Matthew Minish et al.

These parameter ranges encompass standard configurations used by the original BERT authors (Devlin et al. 2019), and extend them with both higher and lower values to allow the tuning framework (Akiba et al. 2019) to empirically find the highest performing configurations for our context. The best hyperparameters from 1,000 trials were selected based on cross-validation performance, these results are reported in Section 6.1.2. 5.2.2 Decoder-only Models In addition to encoder-only models, we explored decoder-only LLMs for reflection classification. Given the sensitive nature of student self-reflections and associated privacy and ethical considerations, we ran these models locally using Ollama (Ollama Team 2024) and publicly available models, rather than relying on third-party cloud services. We evaluated two open-source decoder-only models: OpenAI’s gpt-oss 20B model (OpenAI et al. 2025); and, Alibaba’s Qwen3 14B model (Yang et al. 2025). These models were selected for their strong performance on natural language understanding benchmarks, their manageable computational requirements for local deployment, and their open-source availability (OpenAI 2025; Qwen Team 2025). For both models, we employed a zero-shot prompting approach to evaluate their capability to classify reflections using only the annotation guidelines, without requiring task-specific fine-tuning. We did not employ few-shot prompting (i.e., including labelled examples within the prompt), as our aim was to evaluate the decoder-only models’ baseline capability to apply the annotation guidelines, mirroring the conditions under which human annotators initially approached the task. The potential effect of few-shot prompting on classification performance is discussed in Section 7.5.2. The complete prompt and code artefacts used to automate the classification are available as supplementary material (Minish et al. 2025). This prompt contained all information given to human annotators (as described in Section 5.1), including example phrases and commonly related terms. The prompt explicitly instructed the model to treat each indicator independently, and to only consider was explicitly written in the reflection text without inferring anything. To evaluate the models, we prompted them to classify all 1,518 responses by appending each text from the standardised prompt between the designated [START CLASSIFICATION TEXT] and [END CLASSIFICATION TEXT] delimiters. The models were instructed to return their classifications using a provided JSON object template with boolean values for each of the eight indicators, which we then parsed programmatically. Importantly, we enabled extended reasoning (or “thinking”) modes for both models 4 to improve classification quality. Doing so causes the models to generate intermediate reasoning steps—effectively, thinking aloud for a time— before producing their final classification, which has been shown to improve performance on complex reasoning tasks (Wei et al. 2023). 4 Via a flag in the model, see https://huggingface.co/Qwen/Qwen3-14B# switching-between-thinking-and-non-thinking-mode

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

21

We evaluated the decoder-only models using the same metrics as the encoderonly models to enable direct comparison: – General performance was assessed using Hamming loss and exact match accuracy. – Per-indicator performance was measured using accuracy, precision, recall, and F1 score. The results of the evaluations are presented in Section 6.2.

6 Results In this section we present: (1) a comparison of different transformer models fine-tuned on the annotated data; (2) evaluations of two final models after hyperparameter-tuning; and (3) the agreement of the final classifier with human annotators.

6.1 Encoder-only Model Comparison 6.1.1 Comparison of Five Models Results Summary: RoBERTa achieved the strongest overall performance among the five encoder-only models, with the lowest Hamming loss and consistently high F1-scores on the most challenging indicators (Perspective, Feelings, Reasoning). It was selected for hyperparameter tuning on this basis. Table 3 summarises the F1-scores across all eight indicators for the five encoder-only models evaluated using 10-fold cross-validation. Full per-indicator results including precision, recall, and accuracy are provided in Appendix A (Table 10). All models showed strong performance on indicators with high human inter-rater agreement, with F1-scores generally exceeding 0.85 for Understanding and Future Intention across most models. However, performance varied substantially on the more challenging indicators— Perspective, Feelings, and Reasoning—which also had lower inter-rater agreement between human raters (see Table 2). RoBERTa consistently achieved the highest or near-highest F1scores on these challenging indicators. Given that all models performed satisfactorily on the high-agreement indicators, we prioritised the model that performed best where there was more room for improvement, and selected RoBERTa for hyperparameter tuning.

22

Matthew Minish et al.

Table 3 Encoder-only model F1-scores by indicator (mean ± std). Full results including precision, recall, and accuracy are provided in Appendix A, Table 10. Indicator Description Understanding Feelings Reasoning Perspective New Learning Hindsight Future Intention Macro F1 Micro F1 Exact Match Acc. Hamming Loss

ALBERT 0.724±0.060 0.863±0.145 0.663±0.070 0.625±0.069 0.470±0.083 0.643±0.086 0.791±0.138 0.798±0.219 0.697±0.093 0.726±0.099 0.242±0.070 0.184±0.038

BERT 0.771±0.027 0.916±0.022 0.688±0.036 0.671±0.031 0.492±0.062 0.701±0.076 0.810±0.048 0.889±0.040 0.742±0.016 0.776±0.016 0.283±0.023 0.157±0.015

DistilBERT 0.777±0.027 0.916±0.018 0.695±0.059 0.666±0.043 0.505±0.087 0.735±0.053 0.824±0.053 0.882±0.026 0.750±0.018 0.782±0.016 0.302±0.043 0.152±0.010

DeBERTa 0.780±0.024 0.914±0.022 0.707±0.059 0.693±0.043 0.524±0.080 0.764±0.054 0.846±0.073 0.897±0.027 0.766±0.018 0.791±0.014 0.303±0.041 0.148±0.012

RoBERTa 0.781±0.029 0.914±0.020 0.724±0.059 0.689±0.050 0.537±0.133 0.812±0.063 0.847±0.056 0.909±0.029 0.777±0.029 0.802±0.015 0.325±0.042 0.136±0.010

Note: Best values for each metric are highlighted in bold.

6.1.2 Final Model Performance Results Summary: After hyperparameter tuning, the F1-optimised RoBERTa model improved exact match accuracy from 0.325 to 0.380. We provide two model variants: one optimised for overall accuracy (F1-optimised), suitable for general formative feedback where maximising feedback coverage is the priority; and one that favours recall, suitable for contexts where educators wish to minimise the risk of giving misleading feedback to students. Following hyperparameter optimisation of the RoBERTa model, Table 4 summarises the F1-scores for two model variants. Full per-indicator results including precision, recall, and accuracy are provided in Appendix A (Table 11). Here we show the performance of two models: – Firstly, we present a standard model with the highest mean F1-Score across all labels. – Secondly, we present a model based on the highest mean score using the following heuristic which “prefers” recall whilst maintaining classification balance: 0.6 × recall + 0.4 × F1. This configuration provides a “safer” alternative that educators might prefer if they are hesitant to rely on automated assessments of reflections when giving feedback to students, as it is more likely to give students the benefit of the doubt in ambiguous cases. We do this because feedback should only be given to students when it is relevant to their learning (Nicol and Macfarlane-Dick 2006), and poor quality or misleading feedback—that may arise from a misclassification by an automated assessment tool—can negatively affect students’ self-esteem and performance (Black and Wiliam 1998).

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

23

Table 4 RoBERTa F1-scores by indicator after hyperparameter tuning (mean ± std). Full results including precision, recall, and accuracy are provided in Appendix A, Table 11. Indicator Description Understanding Feelings Reasoning Perspective New Learning Hindsight Future Intention Macro F1 Micro F1 Exact Match Acc. Hamming Loss

RoBERTa (before tuning) 0.781±0.029 0.914±0.020 0.724±0.059 0.689±0.050 0.537±0.133 0.812±0.063 0.847±0.056 0.909±0.029 0.777±0.029 0.802±0.015 0.325±0.042 0.136±0.010

RoBERTa (F1 Optimised) 0.805±0.040 0.930±0.014 0.800±0.051 0.732±0.043 0.595±0.048 0.844±0.034 0.864±0.038 0.903±0.014 0.809±0.013 0.831±0.011 0.380±0.040 0.119±0.011

RoBERTa (Recall Pref.) 0.802±0.026 0.931±0.013 0.745±0.053 0.691±0.050 0.541±0.041 0.749±0.049 0.813±0.070 0.886±0.037 0.770±0.025 0.793±0.022 0.285±0.052 0.162±0.026

Note: Best values for each metric are highlighted in bold.

Table 5 shows confusion matrices for each indicator of the two final models. Here we can see that the model prioritising F1 score has the highest rate of true positive (top left cells) and true negative classifications (bottom left cells), while the recall weighted model sacrifices these metrics somewhat to minimise the rate of false negatives (top right cells). See Section 7.4.1 for an evaluation of what these values mean in practice.

Table 5 Confusion Matrices for RoBERTa Model Variants after Hyperparameter Tuning Category Description Understanding Feelings Reasoning Perspective New Learning Hindsight Future Intention

Predicted True False True False True False True False True False True False True False True False

Recall Preference Actual T F 613 275 41 589 1137 149 29 203 393 248 49 828 471 435 46 566 266 405 43 804 243 108 18 1149 207 59 19 1233 446 95 19 958

F1 Optimised Actual T F 570 213 84 651 1136 137 30 215 376 147 66 929 440 293 77 708 209 258 100 951 233 71 28 1186 198 42 28 1250 438 67 27 986

Note: For each category, T = True, F = False. Each 2×2 block shows True Positives, False Positives (left column) and False Negatives, True Negatives (right column).

24

Matthew Minish et al.

6.2 Decoder-only Model Comparison Results Summary: Both decoder-only models substantially underperformed the fine-tuned RoBERTa model across all indicators, whilst requiring processing times two to three orders of magnitude longer (20–45 seconds vs. under 0.1 seconds per classification). Qwen3:14b outperformed GPT-OSS:20b overall, but neither model is practical for real-time feedback applications. Table 6 presents the performance of two decoder-only LLMs evaluated on the complete dataset of 1,518 annotated responses using zero-shot prompting with extended reasoning enabled (as described in Section 5.2.2). 6.2.1 GPT-OSS Performance GPT-OSS:20b achieved an exact match accuracy of 0.189 and a Hamming loss of 0.197. The model demonstrated highly variable performance across indicators. Performance was strongest on indicators where human annotators also showed high agreement: Hindsight (F1: 0.802, accuracy: 0.948), Future Intention (F1: 0.886, accuracy: 0.924), and New Learning (F1: 0.807, accuracy: 0.920). However, the model struggled significantly on some indicators. Perspective, also the indicator with the lowest inter-rater agreement (α = 0.68), showed the weakest performance (F1: 0.473, accuracy: 0.811), with particularly low recall (0.384) indicating the model frequently failed to identify this indicator when present. Understanding also proved challenging, despite strong agreement between human raters (α = 0.84), with the model achieving high precision (0.918) but relatively low recall (0.656), suggesting it was overly conservative in identifying this indicator. 6.2.2 Qwen3 Performance Qwen3:14b achieved an exact match accuracy of 0.229 and a Hamming loss of 0.175, representing improved overall performance compared to GPT-OSS:20b across both metrics. The model demonstrated strong performance on several indicators with high human inter-rater agreement. Future Intention achieved the highest F1score (0.860, accuracy: 0.899), followed by Description (F1: 0.819, accuracy: 0.822), Hindsight (F1: 0.802, accuracy: 0.932), and New Learning (F1: 0.788, accuracy: 0.913). However, challenges remained with certain indicators. Perspective, the indicator with the lowest inter-rater agreement (α = 0.68), showed the weakest performance (F1: 0.508, accuracy: 0.822), though this represented an improvement over GPT-OSS’s performance on the same indicator. The model achieved

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

25

Table 6 Decoder-only Model Performance Comparison Metric Overall Performance Exact Match Accuracy Hamming Loss Macro Precision Macro Recall Macro F1 Micro Precision Micro Recall Micro F1 Description Accuracy Precision Recall F1-Score Understanding Accuracy Precision Recall F1-Score Feelings Accuracy Precision Recall F1-Score Reasoning Accuracy Precision Recall F1-Score Perspective Accuracy Precision Recall F1-Score New Learning Accuracy Precision Recall F1-Score Hindsight Accuracy Precision Recall F1-Score Future Intention Accuracy Precision Recall F1-Score

GPT-OSS:20b

Qwen3:14b

0.189 0.197 0.731 0.785 0.734 0.697 0.790 0.741

0.229 0.175 0.723 0.795 0.745 0.737 0.785 0.760

0.690 0.607 0.993 0.754

0.822 0.767 0.878 0.819

0.671 0.918 0.656 0.766

0.707 0.948 0.677 0.790

0.814 0.694 0.739 0.716

0.811 0.658 0.800 0.722

0.650 0.508 0.971 0.667

0.696 0.550 0.868 0.673

0.811 0.615 0.384 0.473

0.822 0.633 0.424 0.508

0.920 0.747 0.878 0.807

0.913 0.707 0.889 0.788

0.948 0.930 0.705 0.802

0.932 0.743 0.872 0.802

0.924 0.827 0.953 0.886

0.899 0.780 0.957 0.860

Note: Best values for each metric are highlighted in bold.

26

Matthew Minish et al.

low recall (0.424), suggesting it remained conservative in identifying alternative perspectives. Understanding also struggled, despite strong agreement between human raters (α = 0.84), with the model achieving high precision (0.948) but lower recall (0.677), indicating a tendency towards conservative classification similar to GPT-OSS but with somewhat improved recall performance. 6.2.3 Comparison of Decoder-only Models Qwen3:14b performed better than GPT-OSS:20b across most metrics and indicators, achieving higher exact match accuracy (0.229 versus 0.189) and lower Hamming loss (0.175 versus 0.197). The performance differences were most pronounced on certain indicators: Qwen3:14b achieved notably higher F1-scores for Description (0.819 versus 0.754) and Understanding (0.790 versus 0.766), whilst both models struggled similarly with Perspective, achieving F1-scores of 0.508 and 0.473 respectively. For indicators with high human inter-rater agreement, both models performed well, with F1-scores near to or exceeding 0.8 for Hindsight (0.802 for both models), Future Intention (0.860 and 0.886), and New Learning (0.788 and 0.807). 6.2.4 Comparison with Encoder-only Models Comparing the decoder-only models with the fine-tuned encoder-only models reveals clear differences in their performance for this task. The F1-optimised RoBERTa model achieved substantially better overall performance, with an exact match accuracy of 0.380 compared to 0.229 for Qwen3:14b and 0.189 for GPT-OSS:20b. Similarly, RoBERTa’s Hamming loss of 0.119 was considerably lower than both Qwen3:14b (0.175) and GPT-OSS:20b (0.197), indicating more accurate multi-label predictions. The performance gap between encoder-only and decoder-only models varied substantially across indicators. RoBERTa outperformed both decoder-only models on every single indicator when comparing F1-scores, which are derived from both precision and recall to provide a balanced measure of classification performance. For indicators with high human inter-rater agreement—such as Understanding, Hindsight, and Future Intention—both model types performed well, though RoBERTa consistently achieved higher F1-scores: Understanding (0.930 versus 0.790 and 0.766), Hindsight (0.864 versus 0.802 for both), and Future Intention (0.903 versus 0.860 and 0.886). The differences were more pronounced for indicators where human annotators showed lower agreement. For Perspective, the indicator with the lowest human inter-rater agreement (α = 0.68), RoBERTa achieved an F1-score of 0.595 compared to 0.508 for Qwen3:14b and 0.473 for GPT-OSS:20b. Similarly, for Reasoning (α = 0.72), RoBERTa’s F1-score of 0.732 exceeded both Qwen3:14b (0.673) and GPT-OSS:20b (0.667). These patterns suggest that fine-tuning on task-specific annotated data enabled the encoder-only models to better capture the nuanced distinctions that even human annotators find

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

27

challenging, whilst zero-shot decoder-only models struggle more with these ambiguous cases despite their extended reasoning capabilities.

6.2.5 Computational Efficiency for Practical Deployment Results Summary: The fine-tuned RoBERTa model classifies a single reflection in under 0.1 seconds on consumer-grade hardware, making real-time feedback feasible. Decoder-only models required 20–45 seconds per classification, precluding real-time use. For practical deployment in educational settings, computational efficiency is an important consideration alongside classification accuracy. Table 7 compares the processing times of the decoder-only models, both run on the same consumer-grade hardware as the encoder-only models (see Section 5.2.1).

Table 7 Decoder-only Model Timing Comparison (mean ± std). “Input processing” refers to the time the model spends reading the annotation guidelines plus the reflection text to be classified. “Response generation” refers to the time the model spends generating its output (including extended reasoning). Metric Total Time per Sample Time (seconds) Input Processing Time (seconds) Response Generation Time (seconds) Tokens

GPT-OSS:20b

Qwen3:14b

44.6±28.1

20.7±8.0

3.2±4.4

0.4±0.6

36.1±19.8 917±511

20.3±8.0 840±323

Qwen3:14b averaged 20.7 seconds per classification, compared to GPTOSS:20b’s 44.6 seconds—a 54% reduction in processing time. This difference was primarily driven by the response generation phase, where Qwen3:14b averaged 20.3 seconds compared to GPT-OSS:20b’s 36.1 seconds. The high standard deviations in processing times (8.0 seconds for Qwen3:14b; 19.8 seconds for GPT-OSS:20b) reflect the varying complexity of reflection texts, which ranged from single sentences to multiple paragraphs. In contrast, the fine-tuned RoBERTa model classifies a single reflection text in 0.01–0.05 seconds—approximately three orders of magnitude faster than the decoder-only models. This difference has direct implications for deployment: encoder-only models can be used in real-time to provide feedback to students as they write or submit their reflections, whereas decoder-only models would introduce delays that may be impractical in an interactive educational setting.

28

Matthew Minish et al.

6.3 Classifier Agreement with Human Annotations Results Summary: When treated as a fourth annotator, the F1-optimised RoBERTa classifier maintained or exceeded human-level agreement on six of eight indicators. Description and Understanding showed lower agreement with human annotators than the other indicators. To evaluate whether the F1-optimised RoBERTa classifier was able to approach human-level performance in applying the reflection assessment framework, we calculated Krippendorff’s alpha treating the classifier as an additional rater alongside the three human annotators. This approach allows direct comparison between human-to-human agreement and classifier-to-human agreement on the same metric. Table 8 presents these agreement scores across two datasets: the 100-sample set used in the final round of inter-rater agreement calculation (see Section 5.1); and, the complete dataset of 1,518 samples. Table 8 Inter-rater agreement for indicators by Krippendorff’s alpha scores. The second and third columns show agreement on the final 100-sample set used for determining interrater agreement (see Table 2), comparing human-only agreement with agreement when including the F1-optimised RoBERTa classifier as a fourth rater. The fourth column shows agreement across all 1,518 annotated samples. Indicator Description Understanding Feelings Reasoning Perspective New Learning Hindsight Future Intention

Humans only (per Table 2) 0.872 0.835 0.822 0.719 0.677 0.787 0.849 0.889

Humans and F1 RoBERTa (n=100) 0.773 0.715 0.828 0.720 0.727 0.820 0.886 0.913

Humans and F1 RoBERTa (n=1,518) 0.787 0.702 0.922 0.843 0.843 0.916 0.955 0.959

On the 100-sample set used in the final round of inter-rater agreement, when the classifier was included as a fourth rater, six of the eight indicators (Feelings, Reasoning, Perspective, New Learning, Hindsight, and Future Intention) maintained or exceeded the human-only agreement levels, with Perspective showing the most notable increase from 0.677 to 0.727. However, two indicators—Description and Understanding—showed decreases in agreement, dropping from 0.872 to 0.773 and 0.835 to 0.715 respectively. These decreases suggest that whilst the classifier performs well in absolute terms—achieving high accuracy, precision, and recall (see Table 11)—its judgements diverge more from human annotators on these indicators than on others. When including the F1-optimised RoBERTa model as a fourth rater in the complete dataset of 1,518 samples, Krippendorff’s alpha values ranged from α = 0.702 to α = 0.959. The highest agreement was observed for Future In-

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

29

tention (α = 0.959), Hindsight (α = 0.955), and Feelings (α = 0.922), while Understanding showed the lowest agreement at α = 0.702. Compared to the human-only agreement on the calibration set, most indicators maintained similar or higher alpha values on the full dataset. Specifically, Feelings (0.828 to 0.922), Reasoning (0.720 to 0.843), Perspective (0.727 to 0.843), New Learning (0.820 to 0.916), Hindsight (0.886 to 0.955), and Future Intention (0.913 to 0.959) all showed increases, while Description (0.787) and Understanding (0.702) remained lower than their respective human-only baselines (0.872 and 0.835). 7 Discussion 7.1 Reliability of the Reflective Writing Framework Key Takeaway: The reflective writing framework can be applied reliably in practice: six of eight indicators achieved strong inter-rater agreement (α > 0.8), whilst the remaining two exceeded the lower bound for tentative conclusions. The framework’s modular design allows practitioners to include or exclude individual indicators based on their reliability requirements and pedagogical context. The inter-rater agreement results presented in Table 2 demonstrate that most indicators in our reflective writing framework achieved acceptable to strong levels of reliability after iterative refinement of the annotation guidelines. Six of the eight indicators exceeded the recommended threshold of α = 0.8 for reliable coding (Krippendorff 2019), whilst the remaining two indicators (Reasoning and Perspective) achieved values above the lower bound of α = 0.67 for tentative conclusions (Krippendorff 2019). This means that it is indeed possible to identify the indicators from our framework in reflection text. This confirms the practical applicability of the reflection framework and the indicators. 7.1.1 Conceptual Validity of Indicators with Lower Agreement The two indicators with lowest inter-rater agreement after the second round of joint annotation—Reasoning (α = 0.72) and Perspective (α = 0.68)— warranted careful consideration regarding their inclusion in the framework. For Reasoning, the primary challenge we encountered when defining annotation guidelines lay in distinguishing it conceptually from the Understanding indicator. While iterating on the guidelines, we clarified that Reasoning requires concrete cause-and-effect relationships—where the writer identifies that a specific, concrete action or event led to a particular, concrete outcome— whilst Understanding involves more subjective interpretation or high-level evaluation of experiences. For example, a student stating “the merge conflicts

30

Matthew Minish et al.

occurred because we didn’t communicate about which files we were working on” demonstrates Reasoning through concrete causation, whilst “I think communication is important for teamwork” represents Understanding through subjective interpretation, often delivered by the writer as opinions without explicit links to concrete events. This distinction was refined through guideline development, and as annotators became more practised in applying the framework, we saw the inter-rater agreement rise to meet the ‘reliable’ standard with a Krippendorff’s α = 0.80. Conceptually, the ability to distinguish between subjective (Understanding) and objective (Reasoning) analysis of one’s experiences is key to separating between low levels of reflection, and critical reflection (Wong et al. 1995; Boud et al. 2013). For this reason, and due to the increased agreement as annotation continued, we chose to retain Reasoning and Understanding as distinct indicators in the framework. Perspective presented different challenges. As an infrequently occurring indicator in our dataset (present in only 20.4% of texts; see Table 1), annotators had fewer opportunities to practice their judgements, which may have contributed to lower agreement. Through refining the criteria for the indicator to be less reliant on personal interpretation by the annotator, and clarifying that the ‘perspective’ in question must pertain to someone other than the author of the reflection, we were able to achieve a much higher—though still moderate—agreement between the first and second round of agreement checks. As annotation continued and annotators encountered more examples of this indicator, we saw from our reliability checks on the independently annotated data that agreement on the perspective indicator continued to improve, almost to the ‘reliable’ standard. This, combined with the particular importance in a software engineering context that students reflect on the social and interpersonal aspects of their experiences, has led to us also retaining the Perspective indicator in our framework. 7.1.2 Trade-offs between Comprehensiveness and Reliability We have opted to include indicators achieving only moderate reliability after the second round of inter-rater agreement. This was done for a few reasons: – A framework restricted only to indicators with very high agreement (α > 0.8) may be more reliable, but may also be less useful in real educational contexts due to the omission of important dimensions of reflection identified in established theories (Dewey 1910; Schön 1983; Kolb 2014; Mezirow 1991). Both Reasoning and Perspective align with core aspects of reflective practice: critical analysis of experiences and consideration of alternative viewpoints are widely recognised as essential components of meaningful reflection (Moon 2004). – The observed agreement levels for Reasoning (α = 0.72) and Perspective (α = 0.68), whilst lower than for other indicators, remain above the threshold of α = 0.67 typically considered acceptable for drawing tentative conclusions (Krippendorff 2019). Moreover, the reliability checks on independently annotated data showed continued improvement, with agreement

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

31

reaching α = 0.80 for Reasoning and α = 0.76 for Perspective (see Table 2). This progression may show that as annotators gained experience applying the framework, their judgements became more consistent—with Reasoning ultimately achieving the threshold for reliable coding and Perspective approaching it. Given the inherently subjective nature of assessing reflective writing—where even expert annotators must make interpretive judgements about students’ underlying thought processes—these agreement levels may be considered reasonable. The complexity of human reflection means that some dimensions will inevitably be more ambiguous to identify than others. – As the agreement levels suggest that only tentative conclusions should be drawn from these indicators, they should be treated with appropriate caution in practical applications. For instance, if the framework is used to inform feedback to students, instructors might prioritise the high-agreement indicators (Description, Understanding, Feelings, New Learning, Hindsight, and Future Intention) whilst treating assessments of Reasoning and Perspective more tentatively. Rather than assuming these classifications are definitively correct, they might be used to identify potential opportunities for feedback that warrant closer inspection. Importantly, our framework was designed such that all indicators are treated as completely independent. This means that practitioners who wish to exclude Reasoning or Perspective—whether due to concerns about reliability or because they are less relevant to their specific context—can do so without invalidating the assessment of the remaining indicators. 7.1.3 Comparison with Related Work Annotation Methods Our approach to establishing inter-rater agreement differs from related work in this domain, with implications for interpreting and comparing reliability metrics. We calculated Krippendorff’s alpha across all annotated texts without pre-filtering. Ullmann (2019), when developing machine learning models to automatically detect reflection indicators in writing, pre-filtered the dataset of 5080 sentences from 76 student reflective essays across health, business, and engineering disciplines before calculating inter-rater agreement. The filtering retained only sentences where a majority of annotators agreed on the classification. Their reported Cohen’s kappa values ranged from 0.48 to 0.86 across indicators. These inter-rater reliability values were calculated by randomly splitting crowdsourced annotations into groups, and aggregating each group’s classifications through majority voting (simple, two-thirds, or four-fifths majority), then calculating Cohen’s kappa between the aggregated group classifications. This process excludes sentences lacking sufficient annotator agreement and uses a metric designed for only two raters rather than the multiple raters actually involved (Ullmann 2019). Whilst this aggregation approach may be common in crowdsourcing contexts, the pre-filtering of low-agreement sentences may yield higher reliability estimates than what was actually observed during an-

32

Matthew Minish et al.

notation. Additionally, metrics designed for multiple raters—such as Krippendorff’s alpha or Fleiss’ kappa—may be more appropriate than Cohen’s kappa when aggregating judgements from many annotators (Krippendorff 2019). Nehyba and Štefánik (2023), when developing automated classification of indicators for reflective writing by Czech student teachers, encountered similar annotation difficulties in their initial round, obtaining Cohen’s kappa values ranging from only 0.10 to 0.51 across all annotator pairs from a dataset of 7,128 sentences from reflective journals. In response to these low agreement rates, they selected only sentences where the majority of annotators agreed, then requiring their two most consistent annotators to discuss and resolve the remaining 5.86% of disagreements through consensus. This process ultimately achieved final kappa values between 0.35 and 0.67, calculated only on the filtered and consensus-reached subset. The restructuring required—from multiple independent annotators to filtering by majority agreement and ultimately requiring discussion to achieve consensus—illustrates the difficulty in achieving consistent categorisation of reflective writing. Alrashidi et al. (2023), when developing machine learning classifiers to detect seven reflection indicators in computer science students’ final-year project reports, employed three human annotators to code 1,113 sentences from 74 student essays, calculating agreement when all annotators agreed on the same sentence classification. They reported Cohen’s kappa values ranging from κ = 0.46 to κ = 0.75 across the reflection indicators, representing moderate to substantial agreement, though individual kappa values for each indicator were not provided. Whilst their study reports agreement levels for the final coded dataset used to train their machine learning classifiers, the paper does not explicitly describe whether any pre-filtering or iterative refinement of annotations occurred before calculating these agreement statistics. Additionally, with three annotators, metrics designed for multiple raters such as Krippendorff’s alpha or Fleiss’ kappa may have been more appropriate than Cohen’s kappa. Agreement metrics calculated on pre-filtered data (as in Ullmann (2019) and Nehyba and Štefánik (2023)) represent only the subset of cases where annotators already showed some consensus, excluding the most difficult or ambiguous cases where the framework may be hardest to apply consistently. These pre-filtering processes also mean that reported indicator frequencies in these studies may not reflect the true occurrence rates in the original data. Indicators that are genuinely present but difficult to identify consistently would be systematically under-represented in the filtered datasets, making it difficult to compare indicator frequency distributions across studies. Our more conservative approach—calculating Krippendorff’s alpha on the complete unfiltered dataset—provides a more realistic assessment of how consistently the framework can be applied in practice. The strong agreement we achieved on most indicators (α > 0.8 for six of eight indicators) demonstrates that our framework can be applied reliably even when including the most challenging cases. The moderate agreement on Reasoning (α = 0.72) and Perspective (α = 0.68), whilst lower than other indicators, still exceeds the threshold

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

33

for tentative conclusions and represents genuine inter-rater reliability across the full spectrum of reflection texts, not just the easier cases. 7.1.4 Framework Suitability for Real-world Use The high-agreement indicators (Description, Understanding, Feelings, New Learning, Hindsight, and Future Intention) can be classified with substantial confidence, enabling reliable automated feedback on these dimensions. The moderate-agreement indicators (Reasoning and Perspective) may require more cautious interpretation, with automated classifications highlighting potential opportunities for feedback, rather than definitive assessments. This tiered approach to reliability is appropriate for the intended use case of providing formative feedback in educational settings. The goal is not to replace instructor assessment, but to enable scalable, timely feedback that can guide students’ ongoing reflection throughout a course. In this context, a framework that captures important but difficult to assess dimensions of reflection—even with moderate reliability—may be more valuable than a narrower but more reliable framework that misses key aspects of reflective practice. For example, educators may integrate automated feedback to students as they are writing their reflections, but only show feedback relating to the higher reliability indicators at first. Similarly, by knowing which indicators are more and less reliable, educators can tailor the tone of feedback messages: e.g., “You have not discussed X” for high reliability indicators, but “Our system find you may not have discussed X” for low reliability indicators.

7.2 Dataset Characteristics in Context Key Takeaway: Our dataset is drawn from real weekly reflections in a software engineering project course, differing from prior work in both context (project-based SE versus essays or general computer science) and unit of analysis (whole responses to structured questions versus individual sentences). This grounds our results in authentic student reflection produced under real course conditions. Our dataset comprises 1,518 annotated responses from third-year software engineering students at the University of Canterbury, New Zealand. These were weekly self-reflections in a group project course, structured around questions (see Section 4.2). The dataset contains 6,704 sentences with a mean length of 82 words per response (min=1, max=665, SD=76). Table 9 compares our dataset characteristics with previous work in reflective writing classification. Below we summarize key observations: – Reflection context: Our semi-structured three-question format (see Section 4.2) differs from the reflective essays used by Ullmann (2019), and the

34

Matthew Minish et al.

Table 9 Comparison of datasets used in reflection classification studies, including assessment frameworks, units of analysis, and text characteristics. Framework Context Unit Size Length

Framework

Context Unit Size Length

Framework Context Unit Size Length

Framework Context Unit Size Length

This study 8 boolean indicators: Description, Understanding, Feelings, Reasoning, Perspective, New Learning, Hindsight, Future Intention Weekly reflections from third-year software engineering students in project course Whole response to 1 of 3 questions 1,518 (6,704 sentences) Mean 82 words (min=1, max=665, SD=76) Ullmann (2019) Ordinal reflection score (1–3); 8 boolean indicators: Description of an experience, Feelings, Personal belief, Awareness of difficulties, Perspective, Outcome: Lessons learned, Outcome: Future intention 77 reflective essays (67 from BAWE corpus, 10 from cited literature) Sentence 5,080 sentences 116,633 words total (mean: 23 words per sentence) Nehyba and Štefánik (2023) Same as Ullmann (2019) Reflective journals from 300 Czech pre-service teachers Sentence 7,128 sentences (of 33,859 total) Mean 28.65 sentences per journal (SD=14.45); words per sentence not reported Alrashidi et al. (2023) 7 boolean indicators: Description of an experience, Understanding, Feelings, Reasoning, Perspective, Future action, New learning Reflective writing from 74 third- and fourth-year CS students from a UK university Sentence 1,113 sentences Not reported

free-form journals used by Nehyba and Štefánik (2023). These three questions directly influence the nature of indicators present in responses. For example, Question 3 solicits the Future Intention indicator, whilst Questions 1 and 2 prompt more retrospective content, such as the Description or Hindsight indicators. The context of our reflections differs slightly from Alrashidi et al.’s third- and fourth-year computer science students in the UK (Alrashidi et al. 2023), and substantially from Nehyba and Štefánik’s Czech pre-service teachers (Nehyba and Štefánik 2023) and Ullmann’s essay corpus (Ullmann 2019). Software engineering students reflecting on project work may emphasise technical problem-solving and team dynamics differently than other populations. – Unit of analysis: We chose to classify complete responses to reflection prompts, whilst Ullmann (2019) classified individual sentences, with Nehyba and Štefánik (2023), and Alrashidi et al. (2023) following this same approach. Our reasoning for analysing whole responses, rather than indi-

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

35

vidual sentences, is as follows. First, our reflection format is already split into distinct themes via the three-questions used (see Section 4.2). Second, treating sentences in isolation loses important context for identifying reflective elements— Nehyba and Štefánik (2023) found that classification performance improved when sentences were analysed alongside their surrounding context. Finally, as human educators we consider the full context of a reflection—not isolated sentences—when giving feedback, so we chose to train our classifier in the same way.

7.3 Model Performance and Human Agreement Key Takeaway: Classifier performance closely mirrors human inter-rater agreement: indicators that humans classify consistently are also classified well by the models, whilst indicators with lower human agreement remain challenging for automated approaches. The fine-tuned RoBERTa encoder-only model substantially outperforms decoder-only models in both accuracy and speed, making it suitable for real-time deployment.

7.3.1 Classifier Performance Varies by Indicator Difficulty The performance of our classifiers varied substantially across indicators in patterns that closely mirror human inter-rater agreement, suggesting that indicators which humans find difficult to classify consistently also present challenges for automated approaches. Indicators with high human inter-rater agreement—including Description, Understanding, Feelings, New Learning, Hindsight, and Future Intention— achieved strong classification performance across all model types, with the F1-optimised RoBERTa model achieving F1-scores of 0.80 or higher for all six of these indicators (see Table 11). These results suggest that when human annotators can apply the framework consistently, machine learning models can learn to replicate those judgements. Conversely, the two indicators with lowest human agreement—Reasoning and Perspective—proved most challenging for classifiers (see Table 11). These patterns are not surprising: if expert human annotators equipped with detailed guidelines struggle to consistently identify them, we should not expect machine learning models to resolve these ambiguities definitively. The moderate performance on these indicators likely reflects genuine ambiguity in student writing, or the assessment framework itself, rather than classifier limitations.

36

Matthew Minish et al.

7.3.2 Encoder-only Models Enable Real-time Deployment The stark differences in computational efficiency between encoder-only and decoder-only models have direct implications for practical deployment in educational settings. Our fine-tuned RoBERTa encoder-only model classifies individual reflection texts in 0.01–0.05 seconds on standard, consumer-grade hardware—making real-time feedback feasible. A student’s reflection could be analysed, and feedback given without wait. This aligns with formative assessment principles, where timely feedback is preferred (Nicol and Macfarlane-Dick 2006). In contrast, the decoder-only models—whilst requiring no task-specific finetuning—averaged 20.7 seconds (Qwen3:14b) to 44.6 seconds (GPT-OSS:20b) per classification. These processing times, whilst acceptable for small-scale analysis or research purposes, preclude real-time feedback applications. This performance gap suggests different use cases for the two approaches. Encoder-only models are appropriate for production deployment where realtime or near-real-time feedback is desired and where sufficient annotated data exists for fine-tuning. Decoder-only models may be more suitable for prototyping new frameworks, analysing small datasets where fine-tuning is impractical, or in contexts where zero-shot flexibility outweighs processing time. The zero-shot capability of decoder-only models does offer advantages in adaptability. Changes to indicator definitions or addition of new indicators require only prompt modifications rather than retraining, potentially reducing the iteration time when refining assessment frameworks. However, our results suggest this flexibility comes at the cost of both classification accuracy and computational efficiency. 7.3.3 Classifier Approaches Human-level Agreement on Most Indicators To evaluate whether our classifier approaches human-level performance, we calculated Krippendorff’s alpha treating the F1-optimised RoBERTa model as an additional annotator (see Table 8). This allows direct comparison between classifier-to-human agreement and human-to-human agreement. For several indicators, classifier-to-human agreement approached or exceeded human-to-human agreement levels (see Table 8). These results suggest that for high-agreement indicators, the classifier performs comparably to an expert human annotator. The two indicators with lowest human-only agreement showed interesting patterns when the classifier was included. For Perspective, agreement improved notably: from α = 0.68 (human-only) to α = 0.73 when including the classifier on the final agreement set, and further to α = 0.84 across the complete dataset. For Reasoning, human-only agreement remained relatively stable at α = 0.72 on both the final agreement set and when including the classifier, then improved to α = 0.84 across the complete dataset. Whilst these values exceed the threshold for tentative conclusions, the initially moderate agreement indicates these remain the most challenging indicators to classify consistently.

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

37

The pattern for Description and Understanding indicators is noteworthy. When including the classifier on the final agreement set, agreement decreased slightly for Description and Understanding (see Table 8). This suggests that whilst the classifier performs well in absolute terms—achieving high accuracy, precision, and recall (see Table 11)—its judgements diverge more from human annotators on these indicators than on others. The classifier may have learned patterns in the training data that, whilst consistent and predictive, do not fully align with how expert humans apply the framework for these particular dimensions.

7.4 Readiness for Real Educational Applications Key Takeaway: The classifier enables a shift from broad, holistic feedback on reflections to structured, indicator-level feedback that identifies specific dimensions a student may have overlooked. Educators can determine which indicators they expect for a given context, and deliver feedback on missing indicators—for example, as nudges that guide students towards deeper reflection without prescribing what to write. Two model variants support different pedagogical priorities, and the framework’s independence of indicators allows educators to tailor which dimensions are assessed and how feedback is delivered. As discussed in Sections 7.1.3 and 6.3, our framework not only achieved strong inter-rater agreement without pre-filtering data, but classifier models demonstrated agreement with human annotators comparable to expert-level performance across most indicators. 7.4.1 Two-Model Approach for Different Use Cases Given the difficulty of robustly identifying indicators in reflective writing, where even human experts do not show perfect agreement, we provide two classifier models for educators to choose from. The F1-optimised model maximises overall accuracy to most closely match classification by a human expert, while the recall-optimised model gives a more conservative approach, useful in cases where educators may strongly wish to avoid giving misleading feedback (i.e., suggesting a student discuss an indicator of reflection that they have already addressed, but was missed during classification). In a practical deployment for generating formative feedback—where a missing indicator would trigger a prompt to the student—the classification outcomes shown in Table 5 have the following implications: – True Positives (Top left in cell): The student has addressed the indicator; no feedback is given. (Correct decision)

38

Matthew Minish et al.

– True Negatives (Bottom right in cell): The student has not addressed the indicator; feedback is given. (Correct decision) – False Positives (Bottom left in cell): The student has not addressed the indicator, but no feedback is given. (Missed opportunity) – False Negatives (Top right in cell): The student has addressed the indicator, but receives feedback anyway. (Potentially confusing feedback) The recall-preference model (Table 5) correctly identifies 6,330 opportunities for formative feedback (true negatives summed across all indicators), at the cost of 264 instances of potentially confusing feedback (false negatives). The F1-optimised model identifies 6,876 feedback opportunities (true negatives summed across all indicators)—546 more than the recall-preference model—but produces 440 false negatives, 176 more instances of potentially confusing feedback. This represents a trade-off between feedback coverage and precision: the F1-optimised model provides more opportunities for feedback, but with an increased risk of potentially confusing students, whilst the recallpreference model is more conservative but may miss opportunities for feedback. 7.4.2 Guidance for Software Engineering Training Our assessment framework and automated classifier are designed for formative assessment—supporting ongoing learning rather than determining grades. The classification identifies concepts students may have missed in their reflections, but instructors retain control over pedagogical decisions, including which indicators to include, how they prioritised, and how any guidance is presented to students, e.g., immediate feedback messages; periodic summary reports; or, flagging reflections for instructor review. The approach offers practical advantages for formative assessment: fast classification on every-day computer hardware enables feedback in-action rather than retrospectively; automated classification allows integration with large cohorts without prohibitive instructor workload; and the indicators provide a straightforward framework that educators can build on to suit their specific pedagogical context. This approach should not be used for summative assessment. It was not designed to support summative assessment, and we feel using automated classification to determine marks would be inappropriate given the inherent subjectivity in assessing reflection and the potential for students to game the system if classifications directly affect grades. 7.4.3 Enabling Actionable Feedback Through Indicator-level Classification A central motivation of this work is addressing the limitation that feedback on reflective writing tends to be broad and non-specific (see Section 1). Our framework and classifier address this by enabling feedback at the level of individual reflection indicators, rather than at the level of overall reflection quality.

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

39

In practice, this operates as follows. Educators first determine which indicators of reflection they consider appropriate for a given context—for instance, they may expect responses to Question 3 (“What will you do differently in future?”) to exhibit the Future Intention indicator, and potentially Reasoning to justify the intended change. The classifier then identifies which of the expected indicators are absent from a student’s response. Feedback can be generated for each missing indicator, directing the student’s attention to specific dimensions of the reflection they may have overlooked. One approach to delivering such feedback is through nudges, drawing on the principles of nudge theory and choice architecture (Thaler and Sunstein 2008). In this approach, the learning environment is structured to guide students towards deeper reflection without restricting their options or prescribing what they should write (Dimitrova and Mitrovic 2022). For example, a student who has described what they will do differently, but has not provided reasoning, might receive a prompt to consider why they believe this change will be effective. Such nudges have been applied in computing education to support the quality of reflective writing (Mohammadhassan and Mitrovic 2022; Mohammadhassan et al. 2022), and are well-suited to reflection in software engineering project-based learning, where reflection is an inherently personal and self-directed activity that may be undermined by overly prescriptive feedback. Alternatively, educators may prefer to use the classifications to generate periodic summary reports highlighting patterns across multiple reflections, or to flag specific reflections for manual review. This indicator-level approach represents a qualitative shift from the status quo. Rather than receiving feedback such as “your reflection could be deeper”—which leaves students uncertain about what to improve—a student might instead receive a prompt to consider the perspectives of their team members, or to articulate concrete reasons for why an experience unfolded as it did. The framework thus translates the abstract goal of “deeper reflection” into concrete, identifiable dimensions that students can act upon.

7.5 Limitations Our study has some limitations to consider when interpreting the results and applying the framework in practice. These limitations are reported in line with the framework presented in Wohlin et al. (2012). 7.5.1 Conclusion Validity Two indicators—Reasoning (α = 0.72) and Perspective (α = 0.68)—showed only moderate inter-rater agreement amongst human annotators. Whilst these values exceed the threshold for tentative conclusions, and match or exceed the agreement reported in related works, classifications for these indicators should be interpreted more cautiously.

40

Matthew Minish et al.

7.5.2 Internal Validity Our study is subject to several confounding factors. The prompts given to decoder-only models (GPT, Qwen) may have influenced their performance. Whilst we provided the same annotation guidelines to both decoder-only models and human annotators, different prompt structures or phrasings might yield different results. In particular, we used zero-shot prompting (providing only the annotation guidelines, without labelled examples), as this allowed us to evaluate baseline model capability without confounding the comparison with example selection effects. Few-shot prompting—where a small number of labelled examples are included in the prompt—has been shown to improve performance on similar classification tasks (Brown et al. 2020), and may have produced stronger results for the decoder-only models. We did not explore this systematically here, but flag it as a consideration when interpreting our results. Additionally, we used versions of these models (GPT-OSS:20B and Qwen3:14B) that could run on consumer-grade hardware, whereas larger variants (e.g., GPT-OSS:120B, Qwen3:235B) might achieve better classification performance, though would require significantly greater computing resources. During annotation guideline refinement, discussions between annotators may have established implicit criteria that were not fully captured in the final written guidelines. Whilst this tacit knowledge would be shared between human annotators, it remains unavailable to the decoder-only models that relied on the documented guidelines, potentially limiting their performance when compared to human annotators or the encoder-only models.

7.5.3 External Validity Our classifier was trained and evaluated exclusively on reflections from students at a single software engineering project course at the University of Canterbury, New Zealand. This limits generalisability in several ways. First, our three-question reflection format (see Section 4.2) may influence which indicators appear in student responses. Whilst the classifier was trained on responses to all three questions, its performance on fully free-form reflections or differently structured prompts has not been assessed. Second, differences in student populations (e.g., cultural backgrounds, prior education, English language proficiency) or institutional practices (e.g., teaching and assessment methods) could affect classifier performance. The classifier’s generalisability to different educational contexts—such as alternative reflection formats, different student populations, or non-project-based courses— remains to be evaluated. Finally, our dataset size of 1,518 responses, whilst comparable to related work (see Table 9), may limit the model’s ability to transfer to other reflective writing styles. Validation on larger and more diverse datasets would strengthen confidence in the classifier’s generalisability.

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

41

7.5.4 Construct Validity Whilst our indicators are derived from established reflection theories, it remains to be seen whether they capture all important elements of reflection in software engineering education contexts. The framework may miss disciplinespecific aspects of reflection that are particularly relevant to software development practice such as reflection on technical decision-making and trade-offs, learning from code reviews or debugging experiences, or consideration of software quality attributes and their impact on design choices. We classify complete responses to individual reflection questions rather than individual sentences. This approach suits our structured reflection format, and is suited to providing holistic feedback as a human educator typically would, but may not be suitable for contexts where per-sentence analysis is desired. 7.6 Future Work Several directions for future research could extend and validate this work. 7.6.1 Real-world Deployment and Evaluation The most immediate next step is deploying the classifier in an active software engineering project course. Such a deployment requires the design of a feedback delivery system that translates classifier outputs into pedagogically appropriate messages for students. This is a non-trivial undertaking, involving decisions about which indicators to assess for each reflection question, how feedback should be worded and timed, and what form it should take—for instance, whether as real-time nudges during writing, post-submission summaries, or flags for instructor review. These design decisions carry pedagogical implications that warrant careful consideration and evaluation in their own right. This would enable evaluation of how students respond to automated feedback, whether feedback supports improvement in reflection quality over time, and provide a practical guide for educators to follow in their own courses. 7.6.2 Framework Refinement The moderate agreement for Reasoning and Perspective suggests opportunities for refining these indicator definitions. Further investigation could explore whether splitting or combining certain indicators would improve both human agreement and automated classification. Additionally, examining whether all indicators are truly independent—as we currently assume—would inform whether the multi-label classification approach is optimal; co-occurrence analysis of indicators within responses would be one approach to this. Furthermore, an investigation for elements of reflection that may exist but are not covered by our indicators that could inform further opportunities for feedback, particularly in different disciplines, or with industry practitioners.

42

Matthew Minish et al.

7.6.3 Validation Across Other Datasets Our classifier was developed and evaluated on a single institutional dataset. Future work should evaluate its performance on reflections from different institutions, countries, and student populations to establish generalisability. This includes assessing transfer performance of the classifier models presented in this study, determining whether human validation is needed before deployment in new contexts, and investigating whether fine-tuning using annotated data from the new context (i.e., reflection texts labelled with indicators) is needed to maintain performance in other settings. 7.6.4 Generalisation to Other Contexts Evaluating the classifier’s performance on reflections from different formats (e.g., free-form essays, different prompt structures) and different educational contexts (e.g., short-term projects, industry internships) would establish the framework’s broader applicability. This may require collecting additional annotated datasets or exploring transfer learning approaches, or adapting the classifier to operate at different granularities (e.g., sentence-level versus fullresponse classification). 7.6.5 Analysis of Indicator Distribution Across Questions Whilst our three-question reflection format was designed to elicit specific elements of reflection (e.g., retrospective analysis in Question 1 and Question 2, but future planning in Question 3), we have not analysed whether associated indicators appear in the expected questions. Examining the relationship between question structure and indicator presence, in association with pedagogy of what should appear for each question, could inform more targeted prompt design and enable question-specific feedback. 7.6.6 Longitudinal Analysis The framework currently assesses which indicators are present in individual reflections but does not capture temporal patterns. Extending the framework to analyse temporal patterns in student reflections could provide insights into how reflection skills develop over time. This might involve tracking individual students’ progression across multiple reflection submissions or identifying common developmental trajectories. Such approaches would allow for more intelligent, tailored feedback to individual students. 7.6.7 Application to Software Engineering Practice Whilst this study focused on software engineering training, the framework and classifier could potentially support reflection in professional software engineering practice. Retrospectives and post-incident reviews could benefit from

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

43

automated analysis to identify missing or underexplored elements of reflection, for example as a preparation for retrospectives. Future work could investigate whether the indicators transfer to professional contexts, how they might need adapting for team-level versus individual reflections, and whether automated analysis could benefit existing agile practices.

8 Conclusion This study presents a reflective writing framework and automated classifier models for assessing student reflections in software engineering education. Our framework achieved inter-rater agreement comparable to or exceeding related work. The fine-tuned RoBERTa encoder-only models demonstrated strong classification performance. We compared encoder-only and decoder-only models for classifying reflection indicators. Whilst decoder-only models such as GPT and Qwen performed reasonably well when provided with the same annotation guidelines as human annotators, the fine-tuned encoder-only models—particularly after hyperparameter tuning—achieved superior classification performance. Beyond accuracy, the computational efficiency differences were substantial: encoder-only models classified reflections in under 0.1 seconds compared to 20–45 seconds for decoder-only models. This performance gap suggests distinct use cases: encoder-only models are better suited for production deployment where realtime feedback is desired, whilst decoder-only models may be more appropriate for prototyping new frameworks or contexts where annotated training data is unavailable. The framework provides a foundation for educators to support students’ reflection skills through scalable, structured feedback in project-based software engineering courses. By automating the identification of reflection indicators, the classifier can reduce instructor workload whilst enabling more consistent and timely feedback than manual assessment alone. However, the moderate agreement on some indicators, along with the single-institution dataset, highlight areas for further refinement and validation.

Declarations Funding: This work did not receive external funding. Ethical approval: The collection and usage of data have been approved by the University of Canterbury’s Human Research Ethics Committee. Informed consent: Informed consent was obtained from all individual participants included in the study. Author Contributions:

44

Matthew Minish et al.

Matthew Minish: Conceptualisation, Methodology, Investigation, Formal analysis, Data curation, Software, Writing - original draft, Writing - review & editing, Validation, Visualisation. Matthias Galster: Conceptualisation, Methodology, Investigation, Writing - review & editing, Supervision. Fabian Gilson: Conceptualisation, Methodology, Investigation, Writing - review & editing, Supervision. Data availability: Our code artefacts, annotation guidelines, and other supplementary material are available publicly: https://doi.org/10.5281/zenodo. 15770043 Conflict of interest: The authors have no competing interests to declare that are relevant to the content of this article. Clinical Trial Number: Not applicable.

References A N Meyer, G C Murphy, T Zimmermann, T Fritz (2021) Enabling Good Work Habits in Software Developers through Reflective Goal-Setting. IEEE Transactions on Software Engineering 47(9):1872–1885, DOI 10.1109/TSE.2019.2938525 ACM/IEEE (2014) ACM/IEEE joint task force on computing curricula: Software engineering 2014: Curriculum guidelines for undergraduate degree programs in software engineering. https://ieeecs-media.computer.org/assets/pdf/se2014.pdf Akiba T, Sano S, Yanase T, Ohta T, Koyama M (2019) Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Alazba A, Aljamaan H, Alshayeb M (2024) CoRT: Transformer-based code representations with self-supervision by predicting reserved words for code smell detection. Empirical Software Engineering 29(3):59, DOI 10.1007/s10664-024-10445-9 Alrashidi H, Joy M (2020) Reflective Writing Analysis Approach Based on Semantic Concepts: An Evaluation of WordNet Affect Efficiency. In: Bi Y, Bhatia R, Kapoor S (eds) Intelligent Systems and Applications, vol 1038, Springer International Publishing, Cham, pp 321–333, DOI 10.1007/978-3-030-29513-4 23 Alrashidi H, Joy M, Ullmann T, Almujally N (2020a) Educators’ validation on a reflective writing framework (RWF) for assessing reflective writing in computer science education. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12149 LNCS:316–322, DOI 10.1007/978-3-030-49663-0 37 Alrashidi H, Ullmann T, Ghounaim S, Joy M (2020b) A Framework for Assessing Reflective Writing Produced Within the Context of Computer Science Education. In: Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20, Frankfurt, Germany Alrashidi H, Almujally N, Kadhum M, Daniel Ullmann T, Joy M (2023) Evaluating an Automated Analysis Using Machine Learning and Natural Language Processing Approaches to Classify Computer Science Students’ Reflective Writing. In: Ranganathan G, Bestak R, Fernando X (eds) Pervasive Computing and Social Networking, Springer Nature, Singapore, pp 463–477, DOI 10.1007/978-981-19-2840-6 36 Aracena G, Luster K, Santos F, Steinmacher I, Gerosa MA (2025) Applying large language models to issue classification: Revisiting with extended data and new models. Science of Computer Programming 246:103333, DOI 10.1016/j.scico.2025.103333

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

45

Babb J, Hoda R, Nørbjerg J (2014) Embedding Reflection and Learning into Agile Software Development. IEEE Software 31(4):51–57, DOI 10.1109/MS.2014.54 Beck K (2001) ”Manifesto for Agile Software Development”. Manifesto for Agile Software Development Biggs JB, Collis KF (2014) Evaluating the Quality of Learning: The SOLO Taxonomy (Structure of the Observed Learning Outcome). Academic Press Binkhonain M, Alfayez R (2025) Are prompts all you need? Evaluating prompt-based Large Language Models (LLM)s for software requirements classification. Requirements Engineering 30(4):423–443, DOI 10.1007/s00766-025-00451-8 Black P, Wiliam D (1998) Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice 5(1):7–74, DOI 10.1080/0969595980050102 Boud D, Keogh R, Walker D (2013) Promoting reflection in learning: A model. In: Reflect.: Turn. Exp. into Learn., Taylor and Francis, pp 18–40 Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler DM, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D (2020) Language Models are Few-Shot Learners. DOI 10.48550/arXiv.2005.14165, 2005.14165 Campbell JL, Quincy C, Osserman J, Pedersen OK (2013) Coding In-depth Semistructured Interviews: Problems of Unitization and Intercoder Reliability and Agreement. Sociological Methods & Research 42(3):294–320, DOI 10.1177/0049124113500475 Cavalcanti A, Mello R, Gašević D, Freitas F (2023) Towards Explainable Prediction Feedback Messages Using BERT. International Journal of Artificial Intelligence in Education DOI 10.1007/s40593-023-00375-w Chng S (2018) Incorporating reflection into computing classes: Models and challenges. Reflective Practice 19(3):358–375, DOI 10.1080/14623943.2018.1479686 Chou PN, Chang CC (2011) Effects of Reflection Category and Reflection Quality on Learning Outcomes during Web-Based Portfolio Assessment Process: A Case Study of High School Students in Computer Application Course. Turkish Online Journal of Educational Technology - TOJET 10(3):101–114 Demmans Epp C, Akcayir G, Phirangee K (2019) Think twice: Exploring the effect of reflective practices with peer review on reflective writing and writing quality in computer-science education. Reflective Practice 20(4):533–547, DOI 10.1080/14623943. 2019.1642189 Derby E, Schwaber K, Larsen D (2006) Agile Retrospectives : Making Good Teams Great. The Pragmatic Bookshelf Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, pp 4171–4186, DOI 10.18653/v1/N19-1423 Dewey J (1910) How We Think. How We Think., D C Heath, Lexington, MA, US, DOI 10.1037/10903-000 Dimitrova V, Mitrovic A (2022) Choice Architecture for Nudges to Support Constructive Learning in Active Video Watching. International Journal of Artificial Intelligence in Education 32(4):892–930, DOI 10.1007/s40593-021-00263-1 Embo M, Driessen E, Valcke M, van der Vleuten CPM (2015) Relationship between reflection ability and clinical performance: A cross-sectional and retrospective-longitudinal correlational cohort study in midwifery. Midwifery 31(1):90–94, DOI 10.1016/j.midw. 2014.06.006 Galke L, Scherp A, Diera A, Karl F, Lin BX, Khera B, Meuser T, Singhal T (2025) Are We Really Making Much Progress in Text Classification? A Comparative Review. DOI 10.48550/arXiv.2204.03954, 2204.03954 Galster M, Mitrovic A, Malinen S, Holland J, Peiris P (2023) Soft skills required from software professionals in New Zealand. Information and Software Technology 160, DOI 10.1016/j.infsof.2023.107232

46

Matthew Minish et al.

Groeneveld W, Vennekens J, Aerts K (2020) Software engineering education beyond the technical a systematic literature review. In: SEFI Annu. Conf.: Var. Delect.... Complex. New Norm., Proc., European Society for Engineering Education (SEFI), pp 1607–1621 Guo P, Saab N, Post LS, Admiraal W (2020) A review of project-based learning in higher education: Student outcomes and measures. International Journal of Educational Research 102:101586, DOI 10.1016/j.ijer.2020.101586 Hazzan O (2002) The reflective practitioner perspective in software engineering education. Journal of Systems and Software 63(3):161–171, DOI 10.1016/S0164-1212(02)00012-2 Hazzan O, Tomayko J (2005) Reflection and abstraction in learning software engineering’s human aspects. Computer 38:39–45, DOI 10.1109/MC.2005.200 He P, Liu X, Gao J, Chen W (2021) DeBERTa: Decoding-enhanced BERT with Disentangled Attention. DOI 10.48550/arXiv.2006.03654, 2006.03654 Hey T, Keim J, Koziolek A, Tichy WF (2020) NoRBERT: Transfer Learning for Requirements Classification. In: 2020 IEEE 28th International Requirements Engineering Conference (RE), pp 169–179, DOI 10.1109/RE48521.2020.00028 Izadi M, Akbari K, Heydarnoori A (2022) Predicting the objective and priority of issue reports in software repositories. Empirical Software Engineering 27(2):50, DOI 10.1007/ s10664-021-10085-3 Jung Y, Wise A (2020) How and how well do students reflect?: Multi-dimensional automated reflection assessment in health professions education. In: ACM International Conference Proceeding Series, pp 595–604, DOI 10.1145/3375462.3375528 Jung Y, Wise A, Allen K (2022) Using theory-informed data science methods to trace the quality of dental student reflections over time. Advances in Health Sciences Education 27(1):23–48, DOI 10.1007/s10459-021-10067-6 Kalliamvakou E (2021) Octoverse Spotlight 2021: The Good Day Project—Personal analytics to make your work days better Kaur K, Kaur P (2023) MNoR-BERT: Multi-label classification of non-functional requirements using BERT. Neural Computing and Applications 35(30):22487–22509, DOI 10.1007/s00521-023-08833-1 King G, Zeng L (2001) Logistic regression in rare events data. Political analysis 9(2):137–163 Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 2, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, IJCAI’95, pp 1137–1143 Kolb D (2014) Experiential learning: Experience as the source of learning and development. Experiential Learning: Experience as the Source of Learning and Development Kostina A, Dikaiakos MD, Stefanidis D, Pallis G (2025) Large Language Models For Text Classification: Case Study And Comprehensive Review. DOI 10.48550/arXiv.2501. 08457, 2501.08457 Krathwohl DR (2002) A Revision of Bloom’s Taxonomy: An Overview. Theory Into Practice 41(4):212–218, DOI 10.1207/s15430421tip4104 2 Krippendorff K (2004) Measuring the Reliability of Qualitative Text Analysis Data. Quality and Quantity 38(6):787–800, DOI 10.1007/s11135-004-8107-7 Krippendorff K (2019) Content Analysis: An Introduction to Its Methodology. SAGE Publications, Inc., DOI 10.4135/9781071878781 Kumar H, Xiao R, Lawson B, Musabirov I, Shi J, Wang X, Luo H, Williams JJ, Rafferty AN, Stamper J, Liut M (2024) Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms. In: Proceedings of the Eleventh ACM Conference on Learning @ Scale, Association for Computing Machinery, New York, NY, USA, L@S ’24, pp 86–97, DOI 10.1145/3657604.3662042 Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2020) ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. DOI 10.48550/arXiv. 1909.11942, 1909.11942 Li Y, Raković M, Dai W, Lin J, Khosravi H, Galbraith K, Lyons K, Gašević D, Chen G (2023) Are deeper reflectors better goal-setters? AI-empowered analytics of reflective writing in pharmaceutical education. Computers and Education: Artificial Intelligence 5, DOI 10.1016/j.caeai.2023.100157

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

47

Lim RBT, Hoe KWB, Zheng H (2022) A Systematic Review of the Outcomes, Level, Facilitators, and Barriers to Deep Self-Reflection in Public Health Higher Education: MetaAnalysis and Meta-Synthesis. Frontiers in Education 7, DOI 10.3389/feduc.2022.938224 Lin CW, Lin MJ, Wen CC, Chu SY (2016) A word-count approach to analyze linguistic patterns in the reflective writings of medical students. Medical Education Online 21:10.3402/meo.v21.29522, DOI 10.3402/meo.v21.29522 Liu M, Shum S, Mantzourani E, Lucas C (2019a) Evaluating machine learning approaches to classify pharmacy students’ reflective statements. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11625 LNAI:220–230, DOI 10.1007/978-3-030-23204-7 19 Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019b) RoBERTa: A Robustly Optimized BERT Pretraining Approach. DOI 10. 48550/arXiv.1907.11692, 1907.11692 Lu G, Ju X, Chen X, Pei W, Cai Z (2024) GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learning. Journal of Systems and Software 212:112031, DOI 10.1016/j.jss.2024.112031 Mäeots M, Siiman L, Kori K, Pedaste M (2016) Relation between students’ reflection levels and their inquiry learning outcomes. In: 8th International Conference on Education and New Learning Technologies (EDULEARN2016) McDonald N, Schoenebeck S, Forte A (2019) Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proc ACM HumComput Interact 3(CSCW):72:1–72:23, DOI 10.1145/3359174 Menekse M (2020) The Reflection-Informed Learning and Instruction to Improve Students’ Academic Success in Undergraduate Classrooms. The Journal of Experimental Education 88(2):183–199, DOI 10.1080/00220973.2019.1620159 Mezirow J (1991) Transformative Dimensions of Adult Learning. Jossey-Bass, 350 Sansome Street, San Francisco, CA 94104-1310 ($27 Minish M, Gilson F, Galster M (2024) ScrumBoard: A Project Management Tool PurposeBuilt for Software Engineering Education. In: 2024 36th International Conference on Software Engineering Education and Training (CSEE&T), pp 1–5, DOI 10.1109/ CSEET62301.2024.10663034 Minish M, Galster M, Gilson F (2025) Assessment of self-reflections in software engineering (supplementary material). DOI 10.5281/zenodo.15770043 Mohammadhassan N, Mitrovic A (2022) Discovering Differences in Learning Behaviours During Active Video Watching Using Epistemic Network Analysis. In: Wasson B, Zörgő S (eds) Advances in Quantitative Ethnography, Springer International Publishing, Cham, pp 362–377, DOI 10.1007/978-3-030-93859-8 24 Mohammadhassan N, Mitrovic A, Neshatian K (2022) Investigating the effect of nudges for improving comment quality in active video watching. Computers and Education 176, DOI 10.1016/j.compedu.2021.104340 Moon JA (2004) A Handbook of Reflective and Experiential Learning: Theory and Practice. Routledge, London, DOI 10.4324/9780203416150 Morales-Trujillo ME, Galster M, Gilson F, Mathews M (2022) A Three-Year Study on Peer Evaluation in a Software Engineering Project Course. IEEE Transactions on Education 65(3):409–418, DOI 10.1109/TE.2021.3123682 Naveed H, Khan AU, Qiu S, Saqib M, Anwar S, Usman M, Akhtar N, Barnes N, Mian A (2025) A Comprehensive Overview of Large Language Models. ACM Trans Intell Syst Technol 16(5):106:1–106:72, DOI 10.1145/3744746 Nehyba J, Štefánik M (2023) Applications of deep language models for reflective writings. Education and Information Technologies 28(3):2961–2999, DOI 10.1007/ s10639-022-11254-7 Nicol DJ, Macfarlane-Dick D (2006) Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education 31(2):199–218, DOI 10.1080/03075070600572090 Ollama Team (2024) Ollama: Get up and running with large language models OpenAI (2025) Openai/gpt-oss-20b · Hugging Face. https://huggingface.co/openai/gpt-oss20b

48

Matthew Minish et al.

OpenAI, Agarwal S, Ahmad L, Ai J, Altman S, Applebaum A, Arbus E, Arora RK, Bai Y, Baker B, Bao H, Barak B, Bennett A, Bertao T, Brett N, Brevdo E, Brockman G, Bubeck S, Chang C, Chen K, Chen M, Cheung E, Clark A, Cook D, Dukhan M, Dvorak C, Fives K, Fomenko V, Garipov T, Georgiev K, Glaese M, Gogineni T, Goucher A, Gross L, Guzman KG, Hallman J, Hehir J, Heidecke J, Helyar A, Hu H, Huet R, Huh J, Jain S, Johnson Z, Koch C, Kofman I, Kundel D, Kwon J, Kyrylov V, Le EY, Leclerc G, Lennon JP, Lessans S, Lezcano-Casado M, Li Y, Li Z, Lin J, Liss J, Lily, Liu, Liu J, Lu K, Lu C, Martinovic Z, McCallum L, McGrath J, McKinney S, McLaughlin A, Mei S, Mostovoy S, Mu T, Myles G, Neitz A, Nichol A, Pachocki J, Paino A, Palmie D, Pantuliano A, Parascandolo G, Park J, Pathak L, Paz C, Peran L, Pimenov D, Pokrass M, Proehl E, Qiu H, Raila G, Raso F, Ren H, Richardson K, Robinson D, Rotsted B, Salman H, Sanjeev S, Schwarzer M, Sculley D, Sikchi H, Simon K, Singhal K, Song Y, Stuckey D, Sun Z, Tillet P, Toizer S, Tsimpourlas F, Vyas N, Wallace E, Wang X, Wang M, Watkins O, Weil K, Wendling A, Whinnery K, Whitney C, Wong H, Yang L, Yang Y, Yasunaga M, Ying K, Zaremba W, Zhan W, Zhang C, Zhang B, Zhang E, Zhao S (2025) Gpt-oss-120b & gpt-oss-20b Model Card. DOI 10.48550/arXiv.2508.10925, 2508.10925 Poldner E, Van der Schaaf M, Simons PRJ, Van Tartwijk J, Wijngaards G (2014) Assessing student teachers’ reflective writing through quantitative content analysis. European Journal of Teacher Education 37(3):348–373, DOI 10.1080/02619768.2014.892479 Prior J, Ferguson S, Leaney J (2016) Reflection is hard: Teaching and learning reflective practice in a software studio. In: Proceedings of the Australasian Computer Science Week Multiconference, Association for Computing Machinery, New York, NY, USA, ACSW ’16, DOI 10.1145/2843043.2843346 Qwen Team (2025) Qwen/Qwen3-14B · Hugging Face. https://huggingface.co/Qwen/Qwen3-14B Rogers K, Dziobek I, Hassenstab J, Wolf OT, Convit A (2007) Who cares? Revisiting empathy in Asperger syndrome. Journal of Autism and Developmental Disorders 37(4):709– 715, DOI 10.1007/s10803-006-0197-8 Rogers RR (2001) Reflection in Higher Education: A Concept Analysis. Innovative Higher Education 26(1):37–57, DOI 10.1023/A:1010986404527 Sanh V, Debut L, Chaumond J, Wolf T (2020) DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. DOI 10.48550/arXiv.1910.01108, 1910.01108 Schön D (1983) The Reflective Practitioner: How Professionals Think in Action. Ashgate Schwaber K, Sutherland J (2011) The scrum guide. Scrum Alliance 21(1):1–38 Shekar A (2007) Active learning and reflection in product development engineering education. European Journal of Engineering Education 32(2):125–133, DOI 10.1080/ 03043790601118705 Solopova V, Rostom E, Cremer F, Gruszczynski A, Witte S, Zhang C, López F, Plößl L, Hofmann F, Romeike R, Gläser-Zikuda M, Benzmüller C, Landgraf T (2023) PapagAI: Automated Feedback for Reflective Essays. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 14236 LNAI, pp 198–206, DOI 10.1007/978-3-031-42608-7 16 Stray V, Moe NB, Sjøberg DIK (2020) Daily Stand-Up Meetings: Start Breaking the Rules. IEEE Software 37(3):70–77, DOI 10.1109/MS.2018.2875988, 1808.07650 Stray VG, Lindsjørn Y, Sjøberg DI (2013) Obstacles to Efficient Daily Meetings in Agile Development Projects: A Case Study. In: 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, pp 95–102, DOI 10.1109/ESEM. 2013.30 Subramanian K, Budhrani K (2020) Influence of course design on student engagement and motivation in an online course. In: Proceedings of the 51st ACM Technical Symposium on Computer Science Education, Association for Computing Machinery, New York, NY, USA, SIGCSE ’20, pp 303–308, DOI 10.1145/3328778.3366828 Sun C, Qiu X, Xu Y, Huang X (2020) How to Fine-Tune BERT for Text Classification? DOI 10.48550/arXiv.1905.05583, 1905.05583 Thaler RH, Sunstein CR (2008) Nudge: Improving Decisions about Health, Wealth, and Happiness. Yale University Press Tsoumakas G, Katakis I, Vlahavas I (2010) Mining Multi-label Data. In: Maimon O, Rokach L (eds) Data Mining and Knowledge Discovery Handbook, Springer US, Boston, MA,

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

49

pp 667–685, DOI 10.1007/978-0-387-09823-4 34 Turns J, Sattler B, Yasuhara K, Borgford-Parnell J, Atman C (2014) Integrating reflection into engineering education. In: ASEE Annu. Conf. Expos. Conf. Proc., American Society for Engineering Education Ullmann TD (2011) An architecture for the automated detection of textual indicators of reflection. In: 1st European Workshop on Awareness and Reflection in Learning Networks Held in Conjunction with the 6th European Conference on Technology Enhanced Learning: Towards Ubiquitous Learning 2011, Palermo, Italy Ullmann TD (2015) Automated detection of reflection in texts. A machine learning based approach. PhD thesis, The Open University Ullmann TD (2017) Reflective writing analytics: Empirically determined keywords of written reflection. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, Association for Computing Machinery, New York, NY, USA, LAK ’17, pp 163–167, DOI 10.1145/3027385.3027394 Ullmann TD (2019) Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches. International Journal of Artificial Intelligence in Education 29(2):217– 257, DOI 10.1007/s40593-019-00174-2 Ullmann TD, Wild F, Scott P (2012) Comparing automatically detected reflective texts with human judgements. In: 2nd Workshop on Awareness and Reflection in TechnologyEnhanced Learning, Saarbrucken, Germany, pp 101–116 de Waal FBM (2008) Putting the Altruism Back into Altruism: The Evolution of Empathy. Annual Review of Psychology 59(Volume 59, 2008):279–300, DOI 10.1146/annurev. psych.59.103006.093625 Wang Y, Qu W, Ye X (2024) Selecting Between BERT and GPT for Text Classification in Political Science Research. DOI 10.48550/arXiv.2411.05050, 2411.05050 Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi E, Le Q, Zhou D (2023) Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. DOI 10. 48550/arXiv.2201.11903, 2201.11903 Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, Wesslén A (2012) Experimentation in Software Engineering, Experimentation in Software Engineering, vol 9783642290442. Springer-Verlag Berlin Heidelberg, DOI 10.1007/978-3-642-29044-2 Wong FK, Kember D, Chung LY, Yan L (1995) Assessing the level of student reflection from reflective journals. Journal of Advanced Nursing 22(1):48–57, DOI 10.1046/j.1365-2648. 1995.22010048.x Wulff P, Buschhüter D, Westphal A, Nowak A, Becker L, Robalino H, Stede M, Borowski A (2021) Computer-Based Classification of Preservice Physics Teachers’ Written Reflections. Journal of Science Education and Technology 30(1):1–15, DOI 10.1007/ s10956-020-09865-1 Wulff P, Mientus L, Nowak A, Borowski A (2023) Utilizing a Pretrained Language Model (BERT) to Classify Preservice Physics Teachers’ Written Reflections. International Journal of Artificial Intelligence in Education 33(3):439–466, DOI 10.1007/ s40593-022-00290-6 Y Dubinsky, O Hazzan (2012) Software Governance Using Retrospectives: A Case Study. In: 2012 IEEE International Conference on Software Science, Technology and Engineering, pp 40–45, DOI 10.1109/SWSTE.2012.11 Yang A, Li A, Yang B, Zhang B, Hui B, Zheng B, Yu B, Gao C, Huang C, Lv C, Zheng C, Liu D, Zhou F, Huang F, Hu F, Ge H, Wei H, Lin H, Tang J, Yang J, Tu J, Zhang J, Yang J, Yang J, Zhou J, Zhou J, Lin J, Dang K, Bao K, Yang K, Yu L, Deng L, Li M, Xue M, Li M, Zhang P, Wang P, Zhu Q, Men R, Gao R, Liu S, Luo S, Li T, Tang T, Yin W, Ren X, Wang X, Zhang X, Ren X, Fan Y, Su Y, Zhang Y, Zhang Y, Wan Y, Liu Y, Wang Z, Cui Z, Zhang Z, Zhou Z, Qiu Z (2025) Qwen3 Technical Report. DOI 10.48550/arXiv.2505.09388, 2505.09388 Zimmerman BJ (2002) Becoming a Self-Regulated Learner: An Overview. Theory Into Practice 41(2):64–70, DOI 10.1207/s15430421tip4102 2

50

Matthew Minish et al.

A Full Model Performance Results Table 10 presents the full per-indicator results for the five encoder-only models evaluated using 10-fold cross-validation, and Table 11 presents the full per-indicator results for the two final RoBERTa model variants after hyperparameter tuning.

Table 10 Multi-label Text Classifier Model Performance Comparison (mean ± std) Metric Overall Performance Exact Match Accuracy Hamming Loss Macro Precision Macro Recall Macro F1 Micro Precision Micro Recall Micro F1 Description Accuracy Precision Recall F1-Score Understanding Accuracy Precision Recall F1-Score Feelings Accuracy Precision Recall F1-Score Reasoning Accuracy Precision Recall F1-Score Perspective Accuracy Precision Recall F1-Score New Learning Accuracy Precision Recall F1-Score Hindsight Accuracy Precision Recall F1-Score Future Intention Accuracy Precision Recall F1-Score

ALBERT

BERT

DistilBERT

DeBERTa

RoBERTa

0.242±0.070 0.184±0.038 0.696±0.046 0.753±0.158 0.697±0.093 0.712±0.049 0.770±0.166 0.726±0.099

0.283±0.023 0.157±0.015 0.711±0.033 0.794±0.043 0.742±0.016 0.739±0.037 0.821±0.032 0.776±0.016

0.302±0.043 0.152±0.010 0.721±0.022 0.793±0.033 0.750±0.018 0.748±0.026 0.822±0.028 0.782±0.016

0.303±0.041 0.148±0.012 0.733±0.030 0.820±0.048 0.766±0.018 0.749±0.032 0.842±0.040 0.791±0.014

0.325±0.042 0.136±0.010 0.759±0.030 0.817±0.054 0.777±0.029 0.776±0.033 0.833±0.043 0.802±0.015

0.763±0.029 0.733±0.068 0.742±0.128 0.724±0.060

0.792±0.032 0.740±0.056 0.810±0.040 0.771±0.027

0.799±0.027 0.748±0.054 0.814±0.051 0.777±0.027

0.803±0.025 0.754±0.039 0.811±0.046 0.780±0.024

0.812±0.029 0.784±0.041 0.781±0.049 0.781±0.029

0.816±0.137 0.896±0.043 0.873±0.205 0.863±0.145

0.868±0.033 0.895±0.029 0.938±0.023 0.916±0.022

0.867±0.026 0.888±0.029 0.946±0.029 0.916±0.018

0.866±0.033 0.897±0.033 0.934±0.037 0.914±0.022

0.868±0.030 0.905±0.030 0.926±0.043 0.914±0.020

0.788±0.038 0.625±0.088 0.729±0.123 0.663±0.070

0.803±0.028 0.644±0.073 0.750±0.061 0.688±0.036

0.817±0.037 0.682±0.086 0.724±0.087 0.695±0.059

0.829±0.029 0.709±0.073 0.728±0.130 0.707±0.059

0.843±0.020 0.744±0.079 0.731±0.126 0.724±0.059

0.735±0.037 0.630±0.117 0.689±0.188 0.625±0.069

0.767±0.024 0.649±0.066 0.711±0.091 0.671±0.031

0.763±0.038 0.644±0.065 0.702±0.089 0.666±0.043

0.767±0.031 0.630±0.073 0.784±0.075 0.693±0.043

0.775±0.034 0.655±0.084 0.744±0.087 0.689±0.050

0.734±0.065 0.421±0.114 0.601±0.201 0.470±0.083

0.756±0.041 0.434±0.071 0.593±0.124 0.492±0.062

0.763±0.053 0.447±0.087 0.597±0.115 0.505±0.087

0.757±0.066 0.454±0.098 0.653±0.126 0.524±0.080

0.785±0.056 0.505±0.116 0.652±0.199 0.537±0.133

0.852±0.042 0.560±0.091 0.786±0.142 0.643±0.086

0.879±0.037 0.618±0.104 0.829±0.070 0.701±0.076

0.900±0.014 0.674±0.086 0.820±0.066 0.735±0.053

0.910±0.028 0.720±0.134 0.845±0.080 0.764±0.054

0.931±0.024 0.781±0.132 0.869±0.060 0.812±0.063

0.941±0.030 0.800±0.089 0.825±0.194 0.791±0.138

0.943±0.014 0.806±0.092 0.833±0.103 0.810±0.048

0.947±0.015 0.799±0.073 0.858±0.080 0.824±0.053

0.951±0.027 0.830±0.112 0.873±0.060 0.846±0.073

0.953±0.019 0.820±0.088 0.884±0.065 0.847±0.056

0.898±0.088 0.901±0.041 0.781±0.263 0.798±0.219

0.933±0.024 0.900±0.062 0.886±0.076 0.889±0.040

0.928±0.015 0.886±0.058 0.885±0.064 0.882±0.026

0.935±0.016 0.866±0.058 0.935±0.039 0.897±0.027

0.942±0.020 0.876±0.049 0.948±0.036 0.909±0.029

Note: Best values for each metric are highlighted in bold.

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

51

Table 11 RoBERTa Model Tuning Performance Comparison (mean ± std) Metric Overall Performance Exact Match Accuracy Hamming Loss Macro Precision Macro Recall Macro F1 Micro Precision Micro Recall Micro F1 Description Accuracy Precision Recall F1-Score Understanding Accuracy Precision Recall F1-Score Feelings Accuracy Precision Recall F1-Score Reasoning Accuracy Precision Recall F1-Score Perspective Accuracy Precision Recall F1-Score New Learning Accuracy Precision Recall F1-Score Hindsight Accuracy Precision Recall F1-Score Future Intention Accuracy Precision Recall F1-Score

RoBERTa (before tuning)

RoBERTa (F1 Optimised)

RoBERTa (Recall Preference)

0.325±0.042 0.136±0.010 0.759±0.030 0.817±0.054 0.777±0.029 0.776±0.033 0.833±0.043 0.802±0.015

0.380±0.040 0.119±0.011 0.774±0.024 0.859±0.027 0.809±0.013 0.789±0.024 0.880±0.025 0.831±0.011

0.285±0.052 0.162±0.026 0.679±0.045 0.915±0.021 0.770±0.025 0.693±0.042 0.929±0.018 0.793±0.022

0.812±0.029 0.784±0.041 0.781±0.049 0.781±0.029

0.829±0.029 0.792±0.036 0.827±0.087 0.805±0.040

0.802±0.033 0.710±0.057 0.929±0.040 0.802±0.026

0.868±0.030 0.905±0.030 0.926±0.043 0.914±0.020

0.888±0.020 0.894±0.021 0.969±0.025 0.930±0.014

0.888±0.022 0.889±0.027 0.977±0.016 0.931±0.013

0.843±0.020 0.744±0.079 0.731±0.126 0.724±0.059

0.877±0.028 0.758±0.077 0.854±0.058 0.800±0.051

0.822±0.048 0.656±0.102 0.884±0.057 0.745±0.053

0.775±0.034 0.655±0.084 0.744±0.087 0.689±0.050

0.792±0.047 0.661±0.092 0.839±0.057 0.732±0.043

0.725±0.060 0.567±0.077 0.902±0.052 0.691±0.050

0.785±0.056 0.505±0.116 0.652±0.199 0.537±0.133

0.823±0.032 0.572±0.080 0.649±0.126 0.595±0.048

0.712±0.053 0.405±0.051 0.835±0.073 0.541±0.041

0.931±0.024 0.781±0.132 0.869±0.060 0.812±0.063

0.943±0.013 0.789±0.056 0.912±0.050 0.844±0.034

0.893±0.033 0.638±0.084 0.920±0.040 0.749±0.049

0.953±0.019 0.820±0.088 0.884±0.065 0.847±0.056

0.960±0.013 0.866±0.070 0.869±0.050 0.864±0.038

0.935±0.030 0.742±0.120 0.918±0.042 0.813±0.070

0.942±0.020 0.876±0.049 0.948±0.036 0.909±0.029

0.937±0.013 0.859±0.029 0.955±0.023 0.903±0.014

0.923±0.029 0.826±0.063 0.960±0.022 0.886±0.037

Note: Best values for each metric are highlighted in bold.

Related documents

• Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation#965480
• A11yLTLNav: Automatic Detection of Accessibility Navigation Failures#965481
• Detecting Argument-Swap Bugs Using Context-Enhanced Code Representations#965482
• A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents#965483
• CARE: A Responsibility-Oriented Architecture for Domain-Scoped Resolution in Modular and Upgradeable Smart Contracts#965484