SLRMentor: An LLM-Based Tool Supporting Learning of SLR in Software Engineering Rodolfo Gil-Pereira1 , Ronnie de Souza Santos1 , Italo Santos2 , and Cleyton Magalhães3 University of Calgary, Canada University of Hawai‘i at Mānoa, USA 3 Universidade Federal Rural de Pernambuco (UFRPE), Brazil [email protected], [email protected], [email protected], [email protected] 1
arXiv:2606.07831v1 [cs.SE] 5 Jun 2026
2
Abstract. This paper presents SLRMentor, a conversational assistant designed to support both learning about the systematic literature review process and the execution of planning activities in software engineering. The tool offers general guidance on SLR methodology and supports key planning tasks, including search string construction and reasoning about inclusion and exclusion criteria, with explanations grounded in established SLR guidelines. A pilot validation with graduate students suggests that SLRMentor helps clarify the SLR process and planning decisions, lowers initial barriers for novice researchers, and supports learning while still requiring active methodological judgment. Keywords: Systematic Literature Review · Literature Review · Chatbot · Research Tool
1
Introduction
Evidence-based software engineering (EBSE) advocates the systematic use of empirical evidence to inform software engineering research and practice [11]. Within EBSE, systematic literature reviews (SLRs) are defined as secondary studies that identify, evaluate, and synthesize primary research through explicit, structured, and repeatable procedures [11, 10, 14]. SLRs are widely used in software engineering to consolidate empirical evidence, reduce reliance on anecdotal knowledge, and support theory building and research agenda setting, particularly in a field characterized by diverse study contexts, research methods, and terminology [10, 12, 23]. Conducting an SLR involves a sequence of well-defined activities, including planning the review, executing the search, selecting relevant studies, extracting data, and synthesizing findings [10, 12, 14]. Among these activities, the planning phase is consistently identified as central to review quality and credibility, as it determines the research questions, inclusion and exclusion criteria, and search strategies that guide subsequent steps [23, 22]. Prior work reports that search strategy construction is time-consuming and error prone due
2
R. Gil-Pereira et al.
to terminology variation, database specific syntax, and trade-offs between recall and precision, and that planning decisions directly affect the completeness, reproducibility, and updatability of SLRs [23, 10, 22]. Recent research has investigated the use of artificial intelligence techniques to support specific activities within the SLR process [1, 20]. Existing studies report that large language models (LLMs) have been explored primarily for supporting search related tasks and study selection activities, including keyword suggestion and classification based on titles and abstracts [1, 6]. Empirical evaluations indicate that such AI-based support exhibits limitations in accuracy and reliability and should therefore be used as an assistive aid rather than as a substitute for human judgment [6]. The reviewed literature further shows that current AI based approaches concentrate on improving efficiency in executing review tasks, particularly during study selection, while providing limited support for methodological understanding or learning of the SLR process itself [1, 6]. Despite established guidelines and the increasing availability of tool support, prior educational research indicates that teaching evidence-based software engineering and systematic literature review methods remains challenging, particularly for novice researchers [2, 15, 17]. Studies report that effective learning of systematic reviews requires more than procedural guidance, with students benefiting from instructional support that makes methodological reasoning and decision making explicit and supports iterative engagement across planning and protocol development activities [2, 15, 13, 16]. Motivated by these observations, this research investigates how a conversational assistant can support learning of SLR planning activities in software engineering and poses the following research question (RQ): RQ. How can a conversational assistant support researchers in learning how to conduct systematic literature reviews in software engineering? To address this question, we present SLR Mentor, a LLM-based conversational assistant designed to support learning of SLR practices with a focus on planning activities, providing explanations, and structured guidance grounded in established SLR guidelines [10].
2
Background
This section provides background on two strands of prior work that inform this study: research on the teaching and learning of systematic literature reviews in software engineering and the use of conversational agents in educational contexts. Teaching Systematic Literature Reviews in Software Engineering. Research on evidence-based software engineering reports that systematic literature reviews and systematic mapping studies are commonly taught in graduate and advanced undergraduate software engineering courses [2, 8]. Instruction typically combines
SLRMentor
3
theoretical introductions to EBSE principles with project-based activities in which students conduct secondary studies on selected topics, often grounded in established SLR guidelines [2, 16]. Prior work indicates an emphasis on hands-on engagement with the review process, supported by problem-based and projectbased learning approaches that connect methodological concepts to concrete research questions and domains [2]. The literature also indicates that learning to conduct SLRs presents challenges for novice researchers across multiple stages of the process, including research question formulation, search strategy construction, inclusion and exclusion criteria definition, quality assessment, and synthesis [15]. These difficulties have been attributed to limited experience with secondary research methods and domain knowledge, and may persist even when guidelines are followed [15]. Educational studies suggest that without explicit instructional support, students may focus on producing required artifacts rather than engaging with underlying methodological reasoning, motivating calls for teaching approaches that support reflection, iteration, and explicit discussion of planning decisions [2, 13, 16]. Conversational Agents for Learning Support in Software Engineering. Conversational agents and chatbots are commonly described as software systems that interact with users through natural language to provide information, guidance, or automated assistance [4, 21]. In educational contexts, they are characterized as interfaces that mediate access to learning resources, support engagement through dialogue-based interaction, and assist learners during instructional activities, including responding to inquiries, supporting self-paced learning, and providing feedback [4, 3]. Prior research also reports limitations related to transparency, consistency of responses, and alignment with pedagogical goals [4, 3]. More recent work indicates that conversational agents are increasingly implemented using LLMs, expanding expressive capacity while raising concerns related to reliability, student reliance, instructional control, and academic integrity [9]. Within software engineering education, these systems have been investigated as support mechanisms for programming practice, guidance on software engineering processes, and interaction with educational or development platforms, with studies reporting potential value for explanation and assistance alongside concerns about variability in learning experiences and an emphasis on task completion over conceptual understanding [21, 18, 7, 9]. Across this literature, conversational agents are reported as complementary tools whose role depends on their integration with instructional design and learning objectives rather than as substitutes for teaching or learning activities [21, 9].
3
Method
This study follows the engineering research method, also referred to as Design Science [14]. Our contribution is a software artifact, SLRMentor, designed to support novice software engineering researchers in planning systematic literature reviews. The tool provides learning-oriented guidance during the early stages of
4
R. Gil-Pereira et al.
SLRs, with a focus on supporting understanding and methodological reasoning rather than automating review tasks. Tool Development. The development of SLRMentor was informed by prior work on conversational agents and bots in software engineering and educational contexts, which characterizes such systems as interfaces for providing guidance, information, and learning support through natural language interaction [21]. The tool employs a natural language conversational interface to support incremental, dialogue-based interaction during SLR planning, reflecting how conversational agents have been used to assist users in complex tasks and learning activities [21]. Two general-purpose LLMs (OpenAI and Google Gemini) are used to respond to general questions about the SLR method and to support specific planning activities, such as the construction of search strings and the definition of inclusion and exclusion criteria. Generated outputs are accompanied by explanations intended to support methodological reasoning rather than direct adoption, consistent with prior work emphasizing the assistive rather than autonomous use of such systems in educational settings [7, 9]. The system supports iterative refinement by allowing users to revise goals and artifacts across multiple interactions, aligning with descriptions of how SLR protocols are developed and refined in practice [10]. To support reliability, SLR Mentor integrates retrieval-augmented generation [19, 5], grounding responses in curated SLR guidelines and methodological documentation to improve consistency with established practices and increase transparency, including but not limited to [12, 14]. Task Definition. The functionality of SLRMentor is structured around three learning oriented tasks aligned with the planning phase of systematic literature reviews. First, the Mentor Chat supports conceptual understanding of the SLR process by allowing users to ask questions about review stages, methodological considerations, documentation practices, and common sources of bias, with the aim of supporting orientation and sense-making, particularly for novice reviewers. Second, the Search String Chat assists users in constructing search strategies by guiding the translation of a research goal into a structured search string, proposing keywords and related terms, explaining Boolean operators, and encouraging reflection on alternative formulations, with an emphasis on understanding systematic search design rather than simply producing a final string. Third, the Criteria Chat supports the definition of inclusion and exclusion criteria by helping users derive criteria from their study goals and explaining how these criteria operationalize scope and relevance, thereby supporting reasoning about study selection decisions and their implications for the review. Validation. The pilot validation of SLRMentor adopted an exploratory and formative approach focused on educational value and perceived reliability when used by novice researchers, consistent with the early stage of the artifact and an Engineering Research perspective [14]. The evaluation took place in the context of a graduate software engineering course with eight students, comprising
SLRMentor
5
four doctoral and four master’s students, whose final course assignment required them to conduct either a mapping study or a rapid review, both of which are types of systematic literature reviews, on their own research topics following established SLR guidelines. All students were novice reviewers, conducting a secondary study for the first or second time. After course completion and grading, students were invited to voluntarily and anonymously use SLR Mentor and provide feedback through a structured questionnaire, with no impact on course assessment; four students chose to participate. Participants were instructed to interact with all three components of the system and to apply them to the same study they had previously conducted manually, enabling reflection and comparison between manual and tool-supported planning. Data collection combined Likert scale items and open-ended questions addressing clarity, adequacy, consistency with known SLR practices, and perceived support for learning and reflection. Analysis emphasized descriptive summaries and qualitative interpretation rather than statistical inference, with reliability considered indirectly through participants’ comparisons between tool-generated artifacts and their manually produced counterparts, focusing on perceived methodological alignment, transparency, and support for critical reasoning. The validation data is available at https://figshare.com/s/e332968f559de828cbab.
4
Results
In this paper, we focus on the educational role of SLRMentor, describing how its features can be used to support SRL teaching and learning and how novice researchers experienced these features. Figure 1 shows an overview of SLRMentor, and the live version is available at https://slrmentor.ca/
Fig. 1. SLRMentor Interactive Interface in Use
6
R. Gil-Pereira et al.
Educational Use of SLRMentor. SLR Mentor supports learning during the general understanding and planning phases of systematic literature reviews through three complementary components: – Mentor Chat. The Mentor Chat provides general guidance on systematic literature reviews, supporting conceptual understanding of review stages, methodological principles, and common decisions involved in conducting an SLR. This component is intended to help students clarify doubts about the review process and reflect on methodological choices. – Search String Chat. The Search String Chat supports the construction of search strings by guiding students in translating a research goal or question into searchable terms. This feature provides explanations about keyword selection, use of synonyms, and application of Boolean operators to help students understand how search strategies are systematically designed. – Criteria Chat. The Criteria Chat assists students in defining inclusion and exclusion criteria based on their study goal or research question. This feature supports reasoning about study selection by proposing criteria and explaining how they relate to the scope and focus of the review. Across these components, SLR Mentor is designed to support understanding and reflection on SLR planning practices through conversational interaction. Students can engage in dialogue with the tool, building their understanding incrementally from one question to the next while practicing planning activities. Through the use of retrieval-augmented generation, responses are grounded in curated SLR guidelines and methodological documentation, ensuring that explanations remain aligned with established practices rather than relying on uncontextualized model generation. Validation. Across the three components, participant responses suggest that SLR Mentor is experienced primarily as a learning oriented support rather than as a tool for producing definitive planning artifacts. For the Mentor Chat (Table 1), most participants indicate that the explanations provided are clear and generally consistent with SLR practices they already know, while perceptions of whether the answers support learning beyond providing conclusions vary across participants. For the Search String Chat (Table 2), responses indicate that participants generally perceive the component as supporting understanding of the reasoning behind search string construction and the inclusion or exclusion of terms and operators, with stronger agreement regarding its potential usefulness for students who are learning to design search strategies. In contrast, responses for the Criteria Chat (Table 3) show greater variability, with some participants reporting support for understanding and reflecting on inclusion and exclusion criteria, while others report lower comparability between tool-generated criteria and those defined manually. Participants’ narratives describe SLR Mentor as a learning-oriented support that assists with understanding the SLR process while still requiring active judgement and refinement. Several participants emphasized the clarity and adequacy of the Mentor Chat for orienting first-time reviewers, noting that responses
SLRMentor
7
were “clear and aligned to the response” (P04) and that they “really help answer the question in full” (P03). In terms of efficiency, participants reported that the tool helped structure their work when engaging with complex planning tasks. One participant described it as “generally helpful for parsing through complicated systematic literature review requirements” (P01), while another characterized it as a “beginner-friendly scaffold” that helped them move through the SLR workflow in a structured way (P02). The Search String Chat was described as educational rather than prescriptive, with participants highlighting that it “does a good job explaining how and why things are done” (P01) and that it is “useful to show how one might be constructed” (P04), while still requiring further refinement by the student. Experiences with the Criteria Chat were more mixed, but several participants highlighted its role as an initial learning aid. One participant noted that it helped illustrate “common categories used to define inclusion and exclusion criteria” (P02), while another described it as “extremely useful” for first time use, with the expectation that criteria would be refined iteratively (P03). Across components, participants framed the tool as supporting learning through explanation, structure, and comparison with their own work, rather than as a system that replaces methodological decision-making. Table 1. Participant responses for the Mentor Chat Participant
P01 P02 P03 P04
The explanations pro- The information pro- The answers supported vided by the Mentor vided was consistent learning rather than Chat were clear and easy with SLR practices you simply providing concluto follow. already knew sions. Agree Agree Neutral Strongly agree Strongly agree Agree Strongly agree Agree Strongly agree Strongly agree Strongly agree Disagree
Table 2. Participant responses for the Search String Chat Participant Helped me under- Helped me under- Helped me reflect Could support stustand search string stand why terms on my manually dents learning to construction and operators were constructed search build search strings included or ex- string for the first time cluded P01 Agree Agree Disagree Strongly agree P02 Agree Agree Strongly agree Strongly agree P03 Strongly agree Agree Strongly agree Strongly agree P04 Agree Neutral Agree Strongly agree
5
Discussion
Our results align with aspects of prior work on teaching systematic literature reviews in software engineering, which reports that novice researchers encounter
8
R. Gil-Pereira et al. Table 3. Participant responses for the Criteria Chat Participant Helped me under- Criteria were com- Helped me reflect Could support stustand how inclusion parable to my man- on the clarity and dents defining inand exclusion crite- ually defined crite- precision of my cri- clusion and excluria are derived ria teria sion criteria for the first time P01 Disagree Strongly disagree Neutral Neutral P02 Agree Neutral Agree Agree P03 Agree Strongly agree Strongly agree Strongly agree P04 Strongly agree Agree Agree Strongly agree
difficulties when learning how to reason about planning decisions, which should include search strategies and inclusion and exclusion criteria [8, 2, 15]. In our study, participants’ interactions with SLRMentor indicate that the tool was mainly used to clarify concepts, revisit methodological choices, and reflect on planning decisions they had already made manually, rather than to simply follow procedural steps or adopt generated artifacts. This is compatible with earlier educational studies that emphasize the value of explanation and reflection for supporting understanding of secondary study methods [2]. At the same time, our results extend existing work by illustrating how conversational, guidelinegrounded assistance can be embedded directly into SLR planning activities, providing learners with on-demand explanations and opportunities for iterative sense-making during practice, rather than relying solely on instructor-led feedback or static training materials. Answering our research question (RQ. How can a conversational assistant support learning SLR processes for software engineering researchers? ), our results indicate that a conversational assistant can support learning by making methodological reasoning more explicit during planning activities and by enabling comparison between tool-supported guidance and researchers’ own decisions. Participants’ accounts also point to clear limits of this support, as meaningful use of the tool required critical judgment and domain knowledge, suggesting that conversational assistants may complement, but do not replace, established approaches to teaching systematic literature reviews in software engineering. 5.1
Implications for Research and Education
The results suggest implications for both research and education in software engineering. From a research perspective, the findings indicate that conversational assistants grounded in established guidelines can support learning by making methodological reasoning explicit during systematic literature review planning, rather than by producing finalized artifacts. This points to the need for further investigation of such tools as supports for reflection, comparison, and sense making, including how they influence the development of independent methodological judgment over time and across levels of research experience. From an educational perspective, the results indicate that SLRMentor can function as a complementary scaffold that supports understanding and orientation during SLR activities while still requiring active student judgment and refinement. This
SLRMentor
9
suggests that conversational assistants may be integrated into teaching as ondemand supports that reinforce methodological concepts and structure practice, provided they are positioned as aids for learning and reflection rather than as replacements for instruction, supervision, or critical decision making. 5.2
Threats to Validity
Following established empirical standards in software engineering research [14], threats to validity should be acknowledged. The pilot validation of SLRMentor involved only four graduate students, which limits the strength of any generalizable claims. However, this sample represents half of the students enrolled in the target graduate course, and the validation should therefore be interpreted in light of the characteristics and scale of the educational setting. The study does not aim to provide statistical validation, but rather an initial assessment of how the tool is experienced as a learning support. As such, the results should be understood as indicative and exploratory, intended to inform the design of a larger-scale validation and subsequent tool refinement. Further evaluations with broader and more diverse student populations are required to assess transferability and to strengthen the empirical basis of the findings.
6
Conclusions and Future Work
This paper presented SLR Mentor, a conversational assistant designed to support the teaching and learning of systematic literature review planning in software engineering. The tool addresses key planning activities, including search string construction, reasoning about inclusion and exclusion criteria, and general methodological guidance, with responses grounded in established SLR guidelines through retrieval augmented generation. The results indicate that SLR Mentor is experienced primarily as a learning-oriented support that assists novice researchers in understanding and reflecting on planning decisions, rather than as a system for producing finalized review artifacts. In this way, the tool lowers initial barriers to engagement with SLR methods while maintaining the need for active judgment and methodological responsibility. Future work will focus on examining learning outcomes through larger-scale empirical studies, expanding the curated retrieval corpus, and extending support to later stages of the SLR process, including study selection, data extraction, quality assessment, and synthesis. Extending the assistant to these stages would enable learners to engage with methodological reasoning beyond planning, supporting consistent decisionmaking across the full review lifecycle.
References 1. Bolaños, F., Salatino, A., Osborne, F., Motta, E.: Artificial intelligence for literature reviews: Opportunities and challenges. Artificial Intelligence Review 57(10), 259 (2024)
10
R. Gil-Pereira et al.
2. Castelluccia, D., Visaggio, G.: Teaching evidence-based software engineering: learning by a collaborative mapping study of open source software. ACM SIGSOFT Software Engineering Notes 38(6), 1–4 (2013) 3. Chan, M.M., Amado-Salvatierra, H.R., Hernandez-Rizzardini, R., De La Roca, M.: The potential role of ai-based chatbots in engineering education. experiences from a teaching perspective. In: 2023 IEEE frontiers in education conference (FIE). pp. 1–5. IEEE (2023) 4. Cunningham-Nelson, S., Boles, W., Trouton, L., Margerison, E.: A review of chatbots in education: practical steps forward. In: 30th annual conference for the australasian association for engineering education (AAEE 2019): educators becoming agents of change: innovate, integrate, motivate. pp. 299–306. Engineers Australia (2019) 5. Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.S., Li, Q.: A survey on rag meeting llms: Towards retrieval-augmented large language models. In: Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining. pp. 6491–6501 (2024) 6. Felizardo, K.R., Lima, M.S., Deizepe, A., Conte, T.U., Steinmacher, I.: Chatgpt application in systematic literature reviews in software engineering: An evaluation of its accuracy to support the selection activity. In: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’24). pp. 25–36 (2024). https://doi.org/10.1145/3674805.3686666 7. Fernandez-y Fernandez, C., Sánchez-Soto, E., Cisnero, J.A., Juárez-Ramírez, R.: Exploring the frontier of software engineering education with chatbots. Programming and Computer Software 50(8), 796–815 (2024) 8. Jorgensen, M., Dyba, T., Kitchenham, B.: Teaching evidence-based software engineering to university students. In: 11th IEEE International Software Metrics Symposium (METRICS’05). pp. 8–pp. IEEE (2005) 9. Khan, M., Akbar, M.A., Kasurinen, J.: Integrating llms in software engineering education: Motivators, demotivators, and a roadmap towards a framework for finnish higher education institutes. In: Proceedings of the 2025 29th International Conference on Evaluation and Assessment in Software Engineering Companion. pp. 182–191 (2025) 10. Kitchenham, B., Brereton, O.P., Budgen, D., Turner, M., Bailey, J., Linkman, S.: Systematic literature reviews in software engineering – a systematic literature review. Information and Software Technology 51(1), 7–15 (2009), https://www. sciencedirect.com/science/article/pii/S0950584908001390 11. Kitchenham, B.A., Dybå, T., Jørgensen, M.: Evidence-based software engineering. In: Proceedings of the 26th International Conference on Software Engineering (ICSE 2004). pp. 273–281. IEEE (2004) 12. Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering (EASE). BCS Learning & Development (2008) 13. Pizard, S., Acerenza, F., Otegui, X., Moreno, S., Vallespir, D., Kitchenham, B.: Training students in evidence-based software engineering and systematic reviews: a systematic review and empirical study. Empirical Software Engineering 26(3), 50 (2021) 14. Ralph, P., Ali, N.b., Baltes, S., Bianculli, D., Diaz, J., Dittrich, Y., Ernst, N., Felderer, M., Feldt, R., Filieri, A., et al.: Empirical standards for software en-
SLRMentor
11
gineering research. arXiv preprint arXiv:2010.03525 (2020), https://arxiv.org/ abs/2010.03525 15. Riaz, M., Sulayman, M., Salleh, N., Mendes, E.: Experiences conducting systematic reviews from novices’ perspective. In: 14th International Conference on Evaluation and Assessment in Software Engineering (EASE). BCS Learning & Development (2010) 16. Santos, R.E., de Magalhães, C.V., da Silva, F.Q.B.: The use of systematic reviews in evidence based software engineering: A systematic mapping study. In: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. pp. 1–4 (2014) 17. Santos, R.E., da Silva, F.Q.B.: Motivation to perform systematic reviews and their impact on software engineering practice. In: Proceedings of the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. pp. 292–295. IEEE (2013) 18. Sengul, C., Neykova, R., Destefanis, G.: Software engineering education in the era of conversational ai: current trends and future directions. Frontiers in Artificial Intelligence 7, 1436350 (2024) 19. Swacha, J., Gracel, M.: Retrieval-augmented generation (rag) chatbots for education: A survey of applications. Applied Sciences 15(8), 4234 (2025) 20. de la Torre-López, J., Ramírez, A., Romero, J.R.: Artificial intelligence to automate the systematic review of scientific literature. Computing 105(10), 2171–2194 (2023) 21. Wessel, M., Gerosa, M.A., Shihab, E.: Software bots in software engineering: Benefits and challenges. In: Proceedings of the 19th International Conference on Mining Software Repositories (MSR ’22). pp. 724–725 (2022) 22. Wohlin, C., Mendes, E., Felizardo, K.R., Kalinowski, M.: Guidelines for the search strategy to update systematic literature reviews in software engineering. Information and Software Technology 127, 106366 (2020) 23. Zhang, H., Babar, M.A.: On searching relevant studies in software engineering. In: Proceedings of the 14th International Conference on Evaluation and Assessment in Software Engineering (EASE). pp. 111–120 (2010)