Students’ Practices and Skills in the LLM-Era: “You Can’t Outsource the Struggle and Still Get the Skill” Enne Rebeca Silva de Freitas Universidade Federal do Pará Belém, PA, Brazil [email protected]
Gustavo Pinto Universidade Federal do Pará Belém, PA, Brazil [email protected]
Danilo Ribeiro CESAR School Recife, PE, Brazil [email protected]
arXiv:2607.29519v1 [cs.SE] 31 Jul 2026
Abstract Generative AI tools have been rapidly learned in the daily workflow of graduate students in Software Engineering, but little is known about what AI-related skills they actually need for effective use in empirical research. Without this understanding, graduate programs cannot prepare students to conduct rigorous research in the LLM era, risking creating a generation of researchers who delegate tasks without the necessary expertise. By analyzing 1,383 posts from five research-focused subreddits, we found that students systematically outsource the cognitive effort required to develop research skills and end up with neither the expected results nor the necessary competence. Naming these missing skills is the first step toward curricula that teach graduate students to work with LLMs without being replaced by them. Keywords: AI competencies, computer science education, grey literature review.
1
Introduction
Generative Artificial Intelligence (AI) tools such as ChatGPT, GitHub Copilot, and Gemini have become embedded in the daily workflow of graduate students [12, 17, 19]. These tools now support not only writing tasks [20], but also tasks across the empirical research lifecycle from literature review and study design to coding, and data analysis [14]. However, widespread availability does not imply competent use [15]. Recent evidence suggests that students frequently accept AI-generated content with limited verification, struggle to assess the validity of its outputs, and over-rely on these tools even in their own areas of expertise [9, 17]. The result is a paradox: tools designed to accelerate research can, when used uncritically, undermine the very skills graduate education is meant to cultivate [1]. This is a problem with stakes for multiple actors. For students, over-reliance on generative AI has been associated with degradation of critical thinking [6], weakened metacognitive engagement, and reduced ability to transfer skills to new tasks [2, 6]. For universities and advisors, the absence of a shared understanding of what competent AI use looks like makes it difficult to design coherent training, mentoring, and assessment practices [23]. For the broader research community, a generation of researchers who produce outputs without internalizing the underlying skills threatens the long-term quality and vision of research works. Recent literature has begun to address these concerns, but mostly from adjacent angles [23]. A few studies have proposed AI literacy frameworks for higher education, identifying core competencies such as prompt engineering, ethical awareness, and critical evaluation of AI outputs [1, 22]. Others have measured the effects of generative AI on academic performance, reporting both productivity gains in graduate writing tasks [3] and concerns about the gap between performance and actual learning [6, 23]. 1st AI Agent Journal (2026).
What remains underexplored is the specific case of graduate students conducting empirical research. Graduate research differs from coursework in at least three important ways: 1) it is open-ended, 2) methodologically demanding, and 3) the costs of unreflective AI use—hallucinated citations, biased syntheses, unreproducible analyses—propagate directly into the scientific record. Existing AI literacy frameworks were not designed with these stakes in mind, and the competencies they propose seldom map onto the practical struggles graduate students face when integrating LLMs into, for instance, systematic reviews, reviewing coding, or empirical analyses. Minimizing this gap requires evidence grounded in what graduate students themselves perceive as the missing pieces, rather than top-down lists of competencies. This paper presents a Reddit-based grey literature review [8]. We started by collecting 1,383 researchoriented subreddits posts, and applied thematic synthesis [4] to the 54 selected posts that met our inclusion criteria. Based on this data, we addressed two research questions: RQ1: What are the LLM-based practices that graduate students employ in their research? RQ2: What are the skills that graduate students must have to take better advantage of these LLM-based practices? While RQ1 grounds the inquiry in concrete behavior, asking what graduate students actually do with LLMs in their research, in RQ2 we move to the competencies needed to perform these practices well. Our work can help graduate programs learn about what students are doing with LLMs and what they need to learn to do it responsibly.
2
Methodology
The study follows three steps: search (Section 2.1), selection (Section 2.2), and analysis (Section 2.3). 2.1
Searching
Source selection pilot We ran a pilot evaluating four alternative sources: Hacker News, Google, Twitter/X, and Reddit. Hacker News was discarded after most pilot queries returned threads focused on AI career decisions rather than research competencies. Google was used to refine the initial Boolean query strings (e.g., removing overly broad population terms and adding research-activity anchors such as research workflow); however, Google was not adopted because it is not possible to query without a paid API (direct scraping was blocked), limiting our reproducible automated collection procedure. Twitter/X was evaluated through four programmatic access strategies: 1) Google site: search, 2) internal GraphQL endpoint with guest token and browser cookies, 3) six Nitter instances1 , and 4) through the twscrape library; all failed due to platform restrictions introduced since 2023. Reddit was retained as the sole source after its public JSON API proved accessible without authentication and returned substantively relevant results in the pilot. Beyond accessibility, Reddit is well-suited for this study due to two reasons: 1) its large, research-specific communities (e.g., the r/PhD subreddit alone has more than 240k weekly users and 5k weekly posts), and 2) the platform’s pseudonymity encourages candid disclosure of struggles and skill gaps that would be unlikely to surface in formal publications or institutional surveys [18]. Searching and collecting Data were collected on April 20, 2026 via Reddit’s public JSON search endpoint2 . We targeted five subreddits: r/MachineLearning, r/academia, r/GradSchool, r/PhD, and r/compsci. These sites were chosen given their focus on graduate education and research. The search combined five subreddits with eighteen natural-language queries, totaling ninety subreddit combinations. Scripts, data and queries used are available in the replication package (DOI: https://doi.org/10.5281/zenodo.20498363). After deduplication, 1,383 unique posts were selected for screening. 2.2
Selecting
At this stage, the posts went through a rigorous selection criteria; a post is included if it satisfies all of the following items: 1) it discusses AI-related skills, competencies, or practices in the context of research activities, 2) it is relevant to graduate-level education, 3) it is relevant to Computer Science in general, 4) it was posted between January 2020 and April 2026, and 5) it is written in English or 1 2
Instances that allowed access to the content of the old Twitter, known as X [10]. /r/{subreddit}/search.json
2
Portuguese.3 Conversely, a post is excluded if it matches at least one of the following items: 1) it focuses exclusively on undergraduate or K–12 education, 2) it discusses AI tools without addressing skill gaps or research practices, 3) it is a duplicate post, 4) it contains only a link or image with no substantive text, and 5) it is no longer accessible. Two authors independently screen all candidate posts by title, body text and comments, and classify each as include, exclude, uncertain, AI tools, or Non-CC. Disagreements are resolved by discussion between the authors. A total of 1,383 Reddit posts were retrieved; 1,189 posts were excluded, 95 were classified as AI tools, 20 as Non-CC, and 25 remained uncertain, resulting in 54 included posts.
2.3
Analysing
The 54 included posts are analysed following the thematic synthesis procedure [4]. Three cycles of analysis were conducted. In the first cycle, one of the authors filtered out unrelated posts; subsequently, another author analyzed the filtered posts, and discrepancies were resolved through a discussion of the content and the comments made on the posts. Evaluating the titles of 1,383 Reddit posts, we excluded 1,189 of them, reducing the total to 194 posts. Examples of excluded posts include discussions on admissions and careers (e.g., topics were excluded when they focused primarily on general academic career decisions rather than skill development for using AI in software engineering research, such as “Software engineer trying to contribute to ML research or publish independently – advice?” and “Shifting from ‘student as first author’ to ‘mentor as last author’ – when?”). In the second cycle, we analyzed the content and comments of these posts, assigning codes that reflected their intent, with each author entering their codes independently; a single post could receive multiple codes. In the third cycle, these posts were analyzed with the aim of combining and refining codes; we filtered out a few more posts and refined our codes, resulting in 95 posts classified as AI tools, 20 as Non-CC (unrelated to AI-generated content), and 25 that remained uncertain, ultimately arriving at a final set of 54 Reddit posts. After these three cycles, the codes were grouped into categories. The resulting categories are mapped across two dimensions: LLM-based Practices correspond to RQ1 and LLM-based Skills correspond to RQ2. In the first category of practices, “False Content and lack of validation”, we grouped the following codes: fabrication of references by AI; lack of review and source checking; and text masking. In another example, in the first category of competencies, the codes develop coding practice, dependency free learning, preserve struggle as a learning method, practice writing and reviewing outputs, learn the fundamentals, and use AI for support tasks and not research generated the autonomous learning skill. Figure 1 shows the number of posts included per subreddit. By year, there were thirty-three posts in 2026, six in 2025, five in 2024, three in 2022, two in 2019, and one post in each of the remaining years.
3
After initial screening, we decided that Reddit posts that did not meet criterion four (date of publication), but met all other criteria could be included in the analysis.
3
300
Total vs Included Posts by Subreddit
306
296
262
250 Number of Posts
Total Included
293
226
200 150 100 50 0
5
PhD
Sch Grad
16
14
ool
13
ing
earn ineL
h Mac Subreddit
mia
e acad
6
com
psci
Figure 1: Total vs included posts by subreddit.
3
RQ1: The LLM-based practices
Due to space limitations, we will discuss the three most common practices below. False content and lack of validation This category highlights the use of LLMs as assistants in fabricating false references and citations in a “student’s graduation project. [...] Out of ten sources, four led nowhere – broken links, nonexistent books, actual academics writing articles they never wrote” and the masking of AI-generated content: “[...] They send walls of LLM-generated text without the ability to explain what it’s about”. We found that one of the practices related to ethical conduct is the enthusiasm for fulfilling demands for reproducible results, leading to the superficial use of AI, not as an assistant, but as a content creator, without student validation. However, students should state that “using LLMs to debug code or suggest coding approaches is acceptable, as long as you know enough to spot bugs on your own”. These practices indicate that students use LLMs without validating the output, seeking to meet deadlines without proper knowledge of what is being generated, whether code or scientific text, thus demonstrating a gap in the use of LLMs. False references are extremely serious, since they raise academic dishonesty, and consequently lead to failing grades and to the discrediting of the research. Learning to code with AI In other posts, users suggest practical strategies to facilitate learning, such as writing code with LLMs, but without relying on LLMs: “close the tab and rewrite it from scratch using only what you really understood from reading it”. However, users raise concerns in the posts that AI companies will want professionals who understand their code and implement efficient algorithms, since LLMs are merely optimized pattern recognition systems and do not perform in-depth analysis like a software engineer would. Therefore, the student cannot have “[...] false programming skills” and should be “[...] using LLMs to write the boring parts of the code and writing the main parts themselves”. We observe that using AI as an extra resource to clarify doubts, adjust documentation, or even code is a study strategy; however, one cannot outsource learning, delegating to LLMs all the work to be done, and still expect to acquire solid knowledge. Automated testing, quality assurance, and documentation During the software development process, ensuring test quality requires attention and knowledge of errors for correction. In a detailed 4
post, a user reports that “automated testing forces students to truly understand why something fails instead of guessing until it works”. Even if the student understands a given piece of code, LLMs can be used as support in creating tests, explaining the code structure, and even suggesting how to correct errors during testing. However, maintaining quality is necessary, as one Reddit user mentioned: “verify that your routines are working correctly by hand-labeling some images then running your analysis scripts”. However, there are issues related to this and the need for proper documentation. In one post, one user recommended “[...] a document to define the inputs and outputs of all relevant functions in the code base [...] create a second PDF document indicating an overview of what the feature should do”. Therefore, during testing, tasks such as understanding code cannot be delegated to AI. “[...] Students then need to learn to read error messages and meet specifications by all necessary means”, while another user points out that “most of the AI code I write comes in small, focused fragments [...] I don’t think AI is good at software engineering in that sense, of high-level architecture and design”. This user goes on to recommend the creation of “[...] minimal, invariant tests and reproductions, not entire systems”, that is, even tests created with AI support should be executed on small and more controllable tasks. RQ1 Summary: Our findings show that graduate students use LLMs across multiple stages of the research process, including content generation, programming, testing, debugging, documentation, and literature-related activities. While these tools are often employed to increase productivity and reduce effort, students frequently report practices involving limited validation of generated outputs, particularly in code and scientific writing. The posts also reveal a tension between using LLMs as productivity aids and over-relying on them for tasks that require understanding and judgment. Overall, LLMs are becoming embedded in research workflows, but their use remains strongly dependent on students’ ability to critically assess and verify the generated content.
4
RQ2: The LLM skills
Due to space limitations, we will discuss the most common skills below. Autonomous learning skill We group posts that refer to active learning skills, that is, autonomous student learning without dependence on AI as the main source: “[...] don’t use AI as a crutch”. Users highlight the importance of developing metacognitive skills and autonomous learning: “don’t outsource anything you need to learn to AI”. In the AI age, the ease of use of natural language learning has the potential to create dependence in students. However, as described in a post, “students are adults who need to learn to read error messages”, that is, for them to be able to identify errors, it is necessary to seek in-depth knowledge outside of AI. Therefore, another user says that “junior engineers could prompt their way to decent code but completely fell apart when they had to debug something novel”: using prompts helps generate code quickly, but this does not guarantee real engineering skills, especially if difficult problems arise. The use of AI accelerates task delivery, closing demands with quick resolutions within the expected service time, thus improving productivity. However, there are skill gaps in software engineering, such as in code development, where the student, upon reading the output of an LLM, cannot reproduce the code alone, without any assistance. In line with this view, another user states that “you can’t outsource the struggle and still get the skill” and recommends that students “[...] your top priority skill is learning how to learn [...]”. Another comment reinforces this idea by stating “I try not to copy and paste too much to fully understand what the code is doing”, reinforcing the idea of learning independently. Programming skills Furthermore, a few comments indicate that students should build a solid technical and conceptual foundation, going beyond a superficial mastery of the tools. This foundation allows for understanding, reproducing, and critically evaluating solutions, considering that “the gap between ‘I can read this’ and ‘I can reproduce this’ is exactly where the true skill lies”. There are references to the technical skills frequently required in the programming field, including knowledge of data structures and algorithms, data engineering tools, cloud platforms, and mathematical fundamentals such as statistics and linear algebra. Similarly, it is recommended that students write code for long enough to experience the consequences of their own decisions, because “the shorter the feedback loop, the faster you learn”. Moreover, one user emphasizes that “if you don’t fundamentally 5
understand what you’re programming at a basic level and the code itself to verify if it’s doing what the method should do, you’ll almost certainly make mistakes”. Another one complemented this perspective by stating that “you still need to develop your skills to know when the generated code is correct or not”. Research skills In addition, the development of research-related skills remains indispensable, because “if you don’t improve your skills [...] and communicating results – you won’t last long as a researcher”. A few comments suggest that junior researchers are affected by the excessive use of AI-generated code. According to one of the posts, maintaining software quality and maintainability requires adequate documentation, “explaining in the comments why each action is necessary”. This skill is also particularly relevant for computer science students, since one user argued that “if AI is intended to complement my programming skills, I need radical academic skills”. Managing LLM context skill Another frequently mentioned aspect is the need to provide adequate context to LLMs: “when I create clear and well-structured guidelines with a certain amount of context, it works very well”. However, in certain situations, human intervention remains more efficient, as illustrated by the report: “I examined the code and found the problem immediately — Claude simply wasn’t getting anywhere”. In this context, it is necessary to understand what the LLM is doing; another user reports that “[...] for things like attention masking or loss escalation [...] you only realize it if you understand what the code should be doing”, which makes clear that knowledge guides the LLM to correctly execute the initial request. Problem-solving skills In addition, we noted a few comments that suggest that complementary and interdisciplinary knowledge distinguishes professionals who merely perform tasks from those capable of formulating critical analyses and solving complex problems. As reported by one user, “if you type code, your job is subject to automation. If your job is to use technology to solve increasingly complex problems, your value will always be much greater”. RQ2 Summary: Our findings suggest that effective use of LLMs in graduate research requires a combination of traditional and AI-specific skills. Traditional competencies such as programming, critical thinking, debugging, scientific writing, research design, and problem solving remain essential because they enable students to evaluate and validate AI-generated outputs. At the same time, students increasingly need skills related to interacting with LLMs, including prompt construction, context management, output verification, and hallucination detection. Overall, the posts indicate that LLMs amplify the value of strong foundational skills rather than replacing them.
5
Discussion
Our findings show that LLMs are no longer peripheral tools in graduate research. Reports on Reddit indicate that they are used in several research-related activities, including writing, programming, debugging, testing, documentation, literature exploration, and experiment iteration. These practices suggest that LLMs are being integrated into the everyday infrastructure of research work, not only as writing assistants, but also as coding partners, search tools, and agents that support repetitive or technical tasks. At the same time, the posts reveal an important tension. LLMs can accelerate research work, but they also make it easier for students to delegate tasks that are central to their own formation as researchers. The main issue is not the use of LLMs itself, but the lack of validation, judgment, and accountability when some students accept generated outputs without understanding them. In RQ1 and RQ2, we observed that productive use depends on combining traditional skills, such as programming, debugging, research design, scientific writing, and critical thinking, with LLM-specific skills, such as prompting, context management, hallucination detection, and output verification. An implication of these findings is that graduate education should avoid defining LLM use as a binary choice between prohibition and unrestricted adoption. The data suggest a more nuanced position: students need to learn which parts of research work can be delegated, which parts can be supported, and which parts must remain under human responsibility. For example, using an LLM to generate boilerplate code, refactor a function, draft documentation, or suggest tests may be appropriate when the student can inspect and validate the result. In contrast, delegating theoretical 6
reasoning, methodological decisions, citation validation, or interpretation of results without scrutiny creates risks for both learning and research quality. A second implication concerns the changing nature of programming in empirical research. Several posts indicate that students and researchers increasingly use LLMs to write scripts, refactor code, generate tests, explain errors, and build small tools. However, these same posts emphasize that researchers still need to understand the code they run. This is particularly important in empirical software engineering and machine learning, where errors in scripts, pipelines, or experimental configurations may silently compromise results. In this context, the researcher becomes less of a passive code consumer and more of a reviewer, tester, and maintainer of AI-generated artifacts. Finally, our findings point to a broader educational challenge. If students use LLMs mainly to avoid difficulty, they can complete tasks without developing the skills that those tasks were meant to teach. However, if LLMs are used as tools for explanation, feedback, comparison, and controlled automation, they may help students engage more deeply with research work. The central challenge for graduate programs is therefore not to decide whether students should use LLMs, but to define what responsible use looks like in research training. This includes teaching students to explain generated artifacts, verify claims, preserve their own learning process, and remain responsible for every output they incorporate into their research. 5.1
Limitations
This study has some limitations. First, our findings are based exclusively on Reddit posts, which represent self-reported experiences shared by users of a specific online platform. Reddit users may not be representative of the broader population of graduate students, and discussions are naturally biased toward individuals who are willing to publicly share their concerns, frustrations, and experiences. In addition, the anonymity afforded by Reddit makes it difficult to verify users’ academic status, research experience, or actual use of LLMs. Therefore, our results should be interpreted as perceptions and reported practices rather than direct observations of student behavior. Second, the study relies on a qualitative analysis of a relatively small subset of posts selected from a larger corpus. Although we adopted a systematic search and selection process, different search queries, subreddits, or coding decisions could have led to partially different themes. Furthermore, online discussions tend to emphasize problematic, controversial, or exceptional experiences, potentially underrepresenting successful and routine uses of LLMs in graduate research. Future work could complement our findings through interviews, surveys, or observational studies involving graduate students and advisors from different institutions and research disciplines.
6
Related work
With the massive increase in research data, repositories, and artifacts in software engineering, the use of LLMs in research has proven promising, as they act in a scalable way as agile assistants, aiding in data analysis, code generation, summarization, empirical research, and other applications. LLMs are capable of increasing productivity, especially in manual and repetitive tasks, as well as in content validation in a short time. However, scientific and qualitative validation by humans is irreplaceable, since LLMs have cognitive and knowledge limitations [11, 13]. Hybrid intelligence is the specific use of basic AI tools and human intelligence to enhance cognitive capacity. For example, the use of ChatGPT enhances students’ writing performance while hindering self-regulated learning (SRL). This continuous human–AI interaction by university students triggers potential cognitive laziness, that is, a cognitive discharge due to dependence on AI assistance. Therefore, students and teachers should explore the applications and limitations, seeking mechanisms for the use of AI as educational support [22, 23]. In light of this, Fan et al. [6] reinforce the culture of AI adoption in educational institutions from implementation to exploration and questioning about the use of LLMs as methodological reinforcement in the classroom. The use of LLMs in the classroom is seen as a modern teaching methodology, in which it is necessary to include reflections and discussions about the impact of AI on society, from its ethical aspects in decision-making. It also includes approaches that encourage critical thinking regarding data privacy, biases, plagiarism, and AI hallucinations. 7
Given this, concerns also arise regarding what adaptive skills are needed in the new era of LLMs. Annapureddy et al. [1] define twelve competencies: basic AI literacy; knowledge of generative AI models; knowledge of the capacity and limitations of generative AI tools; skill to use generative AI tools; ability to detect AI-generated content; ability to assess the output of generative AI tools; skill in prompting generative AI tools (prompt engineering); ability to program and fine-tune generative models; knowledge of the contexts where generative AI is used; knowledge of the ethical implications; knowledge of the legal aspects; and ability to continuously learn. This is because AI is taking on supporting roles in hybrid work teams, producing texts, meeting minutes and report summaries, or even assisting students in grading tests and improving their scientific writing [5]. It is in this context that research in grey literature (GL) and on LLMs corroborates software engineering (SE) research, in which the analysis of empirical evidence returns aspects such as the motivations, practices, and current perceptions about the use of AI [21]. Furthermore, research in GL has credibility and serves as empirical evidence in SE, even if it has not undergone peer review [15, 16]. Reddit has been established as a viable and rich data source across multiple research disciplines. Proferes et al. [18] conducted a systematic overview of 727 studies that used Reddit as a corpus, finding it widely adopted in fields ranging from public health to computer science, and identifying best practices for ethical and methodological rigor when mining community-generated content. Complementing this, Fiesler et al. [7] performed a systematic review of ethical considerations in Reddit research, providing guidelines on consent, anonymity, and data handling that inform the ethical conduct of the present study.
7
Concluding remarks
This research investigated the use of LLMs through a GL review based on Reddit, aiming to reveal the practices adopted (RQ1) and the necessary skills (RQ2) that students need to perform research in the LLM era. A total of 1,383 posts were extracted from five research-related subreddits, of which 54 posts were included for qualitative analysis. It was identified that, although the use of LLMs increases productivity and access to quick solutions, indiscriminate use without critical thinking has led some students to cognitive outsourcing, failing to develop research skills. • In RQ1, we identified from the reports that graduate students use LLMs throughout the research process, particularly for programming, writing, testing, documentation, and literaturerelated tasks, often with varying levels of validation. • Regarding RQ2, the results indicate that students need both traditional skills, such as programming and critical thinking, and LLM-specific skills, such as prompting, context management, and output verification. 7.1
Future work
As future work, we plan to: (i) investigate whether intensive LLM usage changes how research skills are acquired and retained over time, (ii) develop evidence-based pedagogical approaches for teaching empirical research in the presence of AI assistants and agents, and (iii) establish a research agenda for defining, measuring, and assessing researcher competencies in the LLM era.
Artifact availability All data used in this work is available on Zenodo: https://doi.org/10.5281/zenodo.2049836 3.
Acknowledgments and Disclosure of Funding References [1] Ravinithesh Annapureddy, Alessandro Fornaroli, and Daniel Gatica-Perez. Generative AI literacy: Twelve defining competencies. Digital Government: Research and Practice, 6(1): 8
1–21, 2025. doi: 10.1145/3685680. [2] Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, and Rei Mariman. Generative AI can harm learning. SSRN Electronic Journal, 2024. doi: 10.2139/ssrn.4895486. The Wharton School Research Paper. [3] Allison E. Connell Pensky, Jordan H. Usdan, and Harley Chang. Generative AI’s impact on graduate student professional writing productivity and quality. International Journal of Artificial Intelligence in Education, 2025. doi: 10.1007/s40593-025-00528-z. [4] Daniela S. Cruzes and Tore Dybå. Recommended steps for thematic synthesis in software engineering. In 2011 International Symposium on Empirical Software Engineering and Measurement (ESEM ’11), pages 275–284. IEEE, 2011. doi: 10.1109/ESEM.2011.36. [5] Yogesh K. Dwivedi, Nir Kshetri, Laurie Hughes, Emma Louise Slade, Anand Jeyaraj, Arpan Kumar Kar, Abdullah M. Baabdullah, Alex Koohang, Vishnupriya Raghavan, Manju Ahuja, et al. Opinion paper: “so what if ChatGPT wrote it?” multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71:102642, 2023. ISSN 0268-4012. doi: 10.1016/j.ijinfomgt.2023.102642. [6] Yizhou Fan, Luzhen Tang, Huixiao Le, Kejie Shen, Shufang Tan, Yueying Zhao, Yuan Shen, Xinyu Li, and Dragan Gašević. Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2):489–530, 2025. doi: 10.1111/bjet.13544. [7] Casey Fiesler, Michael Zimmer, Nicholas Proferes, Sarah A. Gilbert, and Naiyan Jones. Remember the human: A systematic review of ethical considerations in reddit research. Proceedings of the ACM on Human-Computer Interaction, 8(GROUP):1–33, 2024. doi: 10.1145/3633070. [8] Vahid Garousi, Michael Felderer, and Mika V. Mäntylä. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology, 106:101–121, 2019. doi: 10.1016/j.infsof.2018.09.006. [9] Michael Gerlich. AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1):6, 2025. doi: 10.3390/soc15010006. [10] Rodrigo Ghedin. Nitter, jeito alternativo de acessar o X, é descontinuado [in portuguese]. Núcleo Jornalismo. https://nucleo.jor.br/curtas/2024/01/29/nitter-twitter-x-fim/, 2024. Accessed: 2026-05-22. [11] Victoria Gomes, Delaney Selb, Fabio Palomba, Rodrigo Spinola, and David Lo. LLM-assisted empirical software engineering: Systematic literature review and research agenda, 2026. URL https://arxiv.org/abs/2604.26192. [12] Susmita Haldar, Mary Pierce, and Luiz Fernando Capretz. Exploring the integration of generative AI tools in software testing education: A case study on ChatGPT and Copilot for preparatory testing artifacts in postgraduate learning. IEEE Access, 13:46070–46090, 2025. doi: 10.1109/ACCESS.2025.3545882. [13] Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. LLM-as-a-judge for software engineering: Literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology, 2026. doi: 10.1145/3797276. [14] Xinyi Hou, Yanjie Zhao, and Haoyu Wang. The next frontier of LLM applications: Open ecosystems and hardware synergy. CoRR, abs/2503.04596, 2025. doi: 10.48550/arXiv.2503.04 596. [15] Fernando Kamei, Igor Wiese, Gustavo Pinto, Márcio Ribeiro, and Sérgio Soares. On the use of grey literature: A survey with the brazilian software engineering research community. Brazilian Symposium on Software Engineering (SBES), 2020. doi: 10.1145/3422392.3422442. 9
[16] Fernando Kamei, Igor Wiese, Gustavo Pinto, Waldemar Ferreira, Márcio Ribeiro, Renata Souza, and Sérgio Soares. Assessing the credibility of grey literature: A study with brazilian software engineering researchers. Journal of Software Engineering Research and Development, 2022. doi: 10.5753/jserd.2019. [17] Nataliya Kosmyna, Eugene Hauptmann, Ye Tong Yuan, Jessica Situ, Xian-Hao Liao, Ashly Vivian Beresnitzky, Iris Braunstein, and Pattie Maes. Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task. arXiv preprint arXiv:2506.08872, 2025. doi: 10.48550/arXiv.2506.08872. [18] Nicholas Proferes, Naiyan Jones, Sarah Gilbert, Casey Fiesler, and Michael Zimmer. Studying Reddit: A systematic overview of disciplines, approaches, methods, and ethics. Social Media + Society, 7(2), 2021. doi: 10.1177/20563051211019004. [19] Anshul Shah, Anya Chernova, Elena Tomson, Leo Porter, William G. Griswold, and Adalbert Gerald Sosai Raj. Students’ use of GitHub Copilot for working with large code bases. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education (SIGCSE TS 2025), page 7, 2025. doi: 10.1145/3641554.3701800. [20] Oguzhan Topsakal and Tahir Cetin Akinci. Creating large language model applications utilizing LangChain: A primer on developing LLM apps fast. In International Conference on Applied Engineering and Natural Sciences (ICAENS), volume 1, pages 1050–1056, 2023. doi: 10.592 87/icaens.1127. [21] Bianca Trinkenreich, Fabio Calefato, Kelly Blincoe, Viggo Tellefsen Wivestad, Antonio Pedro Santos Alves, Júlia Condé Araújo, Marina Condé Araújo, Paolo Tell, Marcos Kalinowski, Thomas Zimmermann, and Margaret-Anne Storey. Taking a pulse on how generative AI is reshaping the software engineering research landscape, 2026. URL https: //arxiv.org/abs/2604.11184. [22] Yoshija Walter. Embracing the future of artificial intelligence in the classroom: The relevance of AI literacy, prompt engineering, and critical thinking in modern education. International Journal of Educational Technology in Higher Education, 21(1):15, 2024. doi: 10.1186/s41239 -024-00448-3. [23] Lixiang Yan, Samuel Greiff, Ziwen Teuber, and Dragan Gašević. Promises and challenges of generative artificial intelligence for human learning. Nature Human Behaviour, 8(10): 1839–1850, 2024. doi: 10.1038/s41562-024-02004-5.
10