arXiv CSopen access

TagDebt: A Bot to Support Technical Debt Management

TagDebt: A Bot to Support Technical Debt Management João Paulo Biazottoa, b*, Daniel Feitosaa , Paris Avgerioua , Elisa Yumi Nakagawab

arXiv:2605.29869v1 [cs.SE] 28 May 2026

a

University of Groningen, The Netherlands. b University of São Paulo, Brazil.

*Corresponding author(s). E-mail(s): [email protected]; Contributing authors: [email protected]; [email protected]; [email protected]; Abstract Context: Technical debt (TD) is a widely studied metaphor that helps to explain how sub-optimal decisions, which usually have short-term benefits, can harm software maintainability over time. Although incurring TD is not intrinsically bad, tracking and managing TD are crucial to avoid its negative effects. Hence, researchers and practitioners have proposed and developed diverse approaches and tools for managing TD. However, we are still lacking specialized tools for technical debt management (TDM), specifically ones that can be easily integrated into existing development workflows. Objective: We present and evaluate TagDebt, a bot that can be integrated within GitHub repositories and automatically assign labels to issues (i.e., SATD or non-SATD). TagDebt helps in the identification of TD (i.e., by looking for self-admitted technical debt (SATD)), leading to more efficient TDM. Methods: We carried out a Design Science Research study to design and implement TagDebt. For its evaluation, we executed a Technology Acceptance Model (TAM) study through interviews with 16 practitioners, to check the bot’s usefulness, ease of use, and contextual factors that might impact the bot’s usage (such as team size and practitioners’ roles). Results: Overall, practitioners found that TagDebt is useful, especially for organizing issues and reducing manual work. Furthermore, they pointed out that the bot is overall easy to use, and its documentation is clear. The analysis also revealed that contextual factors, such as team and codebase size, impact the decision to adopt TagDebt. Finally, several improvements were suggested, such as including features to check and update the source code. Conclusion: TagDebt is a proof-of-concept for the development and usage of more specialized

1

tools for TDM. It helps to make TD visible without disrupting existing workflows, which could lead to increased adoption of TDM tools and, consequently, help practitioners avoid the risks of unmanaged TD. Keywords: technical debt; technical debt management; tool; bot; machine learning model

1 Introduction Technical Debt (TD) was first coined by Cunningham (Cunningham, 1992) and refers to the future cost of suboptimal decisions made during software development, such as poorly written code (code debt) or inadequate documentation (documentation debt). These decisions may be intentional, made under time-to-market pressure despite known drawbacks, or unintentional, resulting from mistakes or lack of expertise (Li et al., 2015). Although TD may not cause immediate harm (Rios et al., 2018), its accumulation over time can severely impact maintainability, increase costs, or even lead to project abandonment (Guo and Seaman, 2011; Avgeriou et al., 2023, 2025). To mitigate these risks, technical debt management (TDM) practices have been proposed, covering eight key activities, which involve identification, representation/documentation, communication, monitoring, measurement, prioritization, repayment, and prevention (Li et al., 2015). In particular, TD identification refers to mapping existing TD items that should be later documented, prioritized, and paid back. In addition, TD monitoring helps to keep TD items visible. Despite their importance, carrying out TDM activities imposes some challenges (Avgeriou et al., 2021): they are time-consuming (Besker et al., 2018), may increase long-term costs (Alves et al., 2016), and often lack adequate tool support (Biazotto et al., 2025a). Existing tools, plugins, scripts, and bots (Biazotto et al., 2023) are mostly general-purpose solutions (e.g., SonarQube1 ) adapted for TDM rather than being built specifically for it (Jeronimo Junior and Travassos, 2022; Avgeriou et al., 2023; Biazotto et al., 2025a). In this context, the Manifesto on Reframing Technical Debt (Avgeriou et al., 2025) explicitly calls for specialized tools to make TDM more efficient. One manifesto recommendation, i.e., “to develop workflow-based TDM tools”, emphasizes the need for solutions that integrate seamlessly into existing development processes. In addition, in a previous study (Biazotto et al., 2025b), we identified the same concern reported by practitioners, who highlighted that TDM tools must be consistent with their workflows to reduce the friction in tool adoption. However, current tools for TDM (as reported by Biazotto et al. (2023); Silva et al. (2022); Jeronimo Junior and Travassos (2022)) usually require updates and changes to existing workflows (e.g., adding the tool as a new step in the CI / CD pipeline). The overhead of configuring and adopting such tools, combined with the lack of specific features for TDM, helps explain why the adoption of such tools remains 1

https://www.sonarsource.com

2

relatively low (Besker et al., 2019; Biazotto et al., 2025b), compromising the uptake of TDM; this characterizes the problem we intend to tackle in this study. Motivated by those gaps, we defined the following research problems: “(i) Is it possible to develop a specialized TDM tool that is easy to integrate within existing development workflows?”; and “(ii) Considering that we can develop such a tool, would practitioners intend to adopt it?”. In this context, the main objective of this study is to design, implement, and evaluate a TDM tool that would fit existing workflows and be in line with current practitioner needs. To do so, we carried out design science research (DSR) following the framework proposed by ?. As a result, in this paper, we introduce TagDebt2 , a bot that provides practitioners with a solution to identify self-admitted technical debt (SATD) items in GitHub issues. To develop and evaluate TagDebt, we followed DSR phases as follows: first, we explored the social context, i.e., TDM in practice, to understand challenges, goals, and needs for TDM tools. Then, we explored the knowledge context (i.e., research on TDM) to identify existing solutions and designs for the TDM tool. We then designed TagDebt, addressing the identified challenges. Finally, we evaluated TagDebt in the investigation phase of DSR using the established Technology Acceptance Model (TAM), and discussed practitioners’ intention to use TagDebt. Beyond proposing another TD detection bot, this paper positions TagDebt as a workflow-integrated TDM tool and investigates how such automation is perceived by practitioners. Our focus is aligned with broader software engineering concerns about the adoption of automation and tools. For instance, we considered prior work showing that perceived usefulness and ease of use are central to technology adoption (Davis et al., 1989) and that automation must be aligned with socio-technical contexts such as team practices and project characteristics (Elazhary et al., 2021). In this light, TagDebt is designed to (i) embed TD identification into existing GitHub-based workflows with low configuration overhead and few changes to developers’ current practices, and (ii) serve as an object of study on how practitioners perceive automated SATD labeling. Overall, this study provides the following contributions: • TagDebt bot: We introduce TagDebt, a publicly available and fully open-source bot that integrates with GitHub repositories to automatically identify and label issues that report SATD items. TagDebt operationalizes natural language processing (NLP) solutions for SATD identification (in our current implementation, we use the model by Li et al. (2022) and a Large Language Model (LLM)-based approach, using GPT-5-mini). Such solutions (e.g., ML models or LLMs) analyze issue descriptions and add a label to that issue as SATD or non-SATD. Importantly, TagDebt was designed so that the SATD detection component can be easily replaced when desired, enabling the adoption of future models with improved accuracy or models tailored to specific project contexts. However, we note that regardless of the employed detection function, TagDebt will still use natural language as input, and therefore it can only detect SATD, and not TD in other sources like source code. TagDebt is intended to support TD identification and alerting practitioners about the identified TD items. With TagDebt, 2

https://github.com/marketplace/tagdebt-bot

3

we embed TDM directly into the development workflow, reducing configuration overhead. Finally, to the best of our knowledge, TagDebt is the first bot that is able to identify TD in GitHub issues without any type of special tag (e.g., #TODO, #FIXME). • Empirical evidence on the artifact’s usefulness, ease of use, and adoption contexts: Through an evaluation with practitioners, we collected empirical evidence about TagDebt. The results highlight the contextual factors that make it more useful, and also highlight contexts that would benefit the most from adopting the bot. Besides, we list improvements that can be implemented in the bot in future work. These contributions help to scope the bot’s value to practitioners and improve its transferability to industry. • An adapted Technology Acceptance Model (TAM) questionnaire for assessing TDM tools: We applied an adapted version of the TAM method, which contains questions specific to evaluating TDM tools. This methodological contribution can help other researchers and practitioners to replicate our approach to design and validate TDM tools. Such contributions have several implications for practitioners. By integrating into GitHub workflows with minimal configuration effort, TagDebt enables practitioners to detect SATD items earlier (i.e., at the moment issues are created), reducing the risks of debt accumulation. Its configurability allows teams to adapt detection rules to their specific coding standards, documentation practices, and project priorities, thereby improving the precision and relevance of TD identification. Finally, TagDebt minimizes disruption in existing workflows and facilitates adoption in both open-source and industrial contexts. Regarding implications for researchers, TagDebt offers a reproducible open source platform to study SATD detection in issue tracking. The design of the TAM study presented in this work can be reused to evaluate other TDM tools, supporting comparative studies in the future. In addition, the artifact and its evaluation contribute to the growing body of empirical evidence on the role of automation in TDM, helping shape future research agendas on TD tooling and adoption factors. The remainder of this paper is organized as follows. Section 2 presents the background on TD, TDM, and bots in software engineering. Section 3 shows the research method and describes the phases of the DSR study. Section 4 describes TagDebt in detail and Section 5 presents the design of the bot’s evaluation. Section 6 reports the results of the TagDebt evaluation, and such results are discussed in Section 7. Section 8 presents threats to validity and mitigation strategies, and Section 9 concludes the paper with final remarks and future work.

2 Related Work In this section, we elaborate on the literature related to our study, which encompasses Steps 1 and 2 of our research method (i.e., Understand Social Context and Understand Knowledge Context). Section 2.1 provides a brief overview of the background of this study, including key aspects of TD, its management, and the use of tools and automation to increase TDM efficiency. In addition, since our study reports on a bot 4

for TDM, in Section 2.2, we discuss how bots have been explored in TDM and compare them with TagDebt.

2.1 TD and its Management The TD present in a system can be introduced at various moments of the software development process since debt is not strictly related to the source code. Consequently, there are different TD types based on their source, e.g., source code, architectural decisions, tests, and infrastructure (Li et al., 2015; Alves et al., 2016; Jeronimo Junior and Travassos, 2022; Bavota and Russo, 2016). Li et al. (2015) defined nine types of TD: • Requirements TD: It relates to requirements elicitation and can refer to, for instance, a lack of requirements or a misunderstanding of some of them. • Architectural TD: bad decisions related to architectural design that could compromise ‘internal’ quality attributes of the software, such as evolvability or maintainability; • Design TD: poor decisions made during the design phase, e.g., the division of responsibility among different classes; • Code TD: violations of code quality aspects, for instance, duplicate code, spaghetti code, and duplicated variables; • Test TD: non-optimal decisions taken in elaboration or execution of tests, e.g., lack of tests; • Build TD: bad decisions that can harm the software building process, e.g., bad dependency management; • Documentation TD: poor documentation in terms of correctness, completeness, and up-to-date aspects; • Infrastructure TD: non-optimal decisions related to the selection of technologies for software development, e.g., using old technologies; and • Versioning TD: problems in source code versioning, such as a lack of multi-version support. To keep TD under control, various activities have been proposed to help practitioners manage it (McGregor et al., 2012; Guo and Seaman, 2011; dos Santos et al., 2013; Alves et al., 2016). Li et al. (2015) summarized eight main activities present in the literature. During identification, TD items are detected using several techniques, such as manual inspection or static code analysis. The identified elements can be documented during the representation/documentation activities, and stakeholders are informed about the TD elements during the communication activity. The TD items can then be monitored during the monitoring activity, which ensures that unsolved TD items are under control. The measurement activity is used to quantify the amount of TD in a system and, in turn, enables the prioritization activity, i.e., ranking TD items that must be solved first. TD items can then be fixed during the repayment activity, which also deals with the problems caused by the accumulation of TD. Finally, it is also possible to avoid undesired TD through the prevention activity. In addition to the TD types and TD activities described above, it is pertinent to expand on the concept of self-admitted technical debt (SATD). SATD pertains 5

to TD elements that developers themselves formally recognize as such (Maldonado and Shihab, 2015). For instance, this occurs when developers annotate source code with comments indicating discrepancies or areas in need of changes. SATD refers to a form of documenting TD, which is orthogonal to the underlying TD problem (e.g., a code smell, architectural drift, etc.). In the context of our paper, “type” refers to the problem (e.g., code debt, design debt, testing debt, etc.). An SATD item will report a problem, which can be associated with a “type” of debt (e.g., code debt). In that sense, SATD is not a type of TD on its own. This is supported by several studies that showed that the nature of problems reported in SATD has also been found in other study contexts, e.g., using static analysis tools, or by interviewing developers. Such a parallel can be observed by comparing the categorization put forward by Li et al. (2015), in their systematic mapping study, and the categorization put forward by Bavota and Russo (2016), synthesized from looking at SATD items. Both studies report a nearidentical top-level categorization. The nature of problems does not change, but how we identify them (through text in comments/issues or using a static source code analysis tool) does. Although focusing on specific types of TD can bring benefits, a type-agnostic tool can offer broader support for TDM. Reflecting on the challenges of TDM, we considered natural language processing, i.e., SATD detection, as a viable alternative to detect TD items and improve management practices. In this way, as long as a TD item is reported as SATD (i.e., reported in natural language), the bot can identify it and raise awareness of its presence. To achieve this, TagDebt automates the identification, which is typically frequent in TDM processes (Jeronimo Junior and Travassos, 2022; Biazotto et al., 2023). By systematically labeling potential TD items in issues, the tool not only reduces the manual effort usually required for this activity but also provides the necessary information for subsequent management actions, such as prioritization and repayment. It is worth noting that there is a relationship between our previous work (Biazotto et al., 2025b) and this current study. In our previous work, we conducted a qualitative survey with 103 practitioners (89 valid responses) from industry and OSS. From the answers, we derived a model of practitioners’ concerns and rationales for TDM tool adoption, showing that adoption decisions are deeply related to keeping control over tool execution and outputs. Additionally, we elicited 46 requirements for TDM tools and clustered them into two categories: “Information to be provided” and “Tool usage”. These requirements were intended to guide both the improvement of existing tools and the development of new ones aligned with practitioners’ needs. Overall, the study argues that efficient TDM automation depends as much on interaction design and configurability as on detection capability, reinforcing the need for highly customizable, low-disruption, human-in-the-loop tooling. This current study is a follow-up to the previous one, inspired and guided by its findings. In this current paper, we designed and implemented a tool grounded in this prior knowledge and evaluated it. Therefore, the main contribution of this paper is a novel tool that demonstrates the feasibility of developing a TDM tool aligned with practitioners’ concerns. As a result, we provide new knowledge both by showing the

6

design decisions behind implementing a TDM tool and by presenting practitioners’ opinions about it, which can support the development and improvement of other tools.

2.2 Bots in SE and TDM The concept of bots can be traced back to 1950 (Turing, 1950), and a core aspect related to the development and usage of bots is the effective interaction between humans and machines (DALE, 2016; Vinciarelli et al., 2015; Zue and Glass, 2000). Today, advances in artificial intelligence (AI) and natural language processing (NLP) have led to the widespread development of bots across various domains and to new forms of human-machine collaboration to build modern work environments (Lee et al., 2017). When it comes to SE, bots help with a variety of tasks, including facilitating communication and helping decision-making processes (Storey and Zagalsky, 2016). Bots have become popular on messaging platforms, showing the potential to impact both the social and technical aspects of software development (Lin et al., 2016). In collaborative development ecosystems, bots automate actions that are typically performed by humans (Lebeuf et al., 2018), such as repairing failures in GitHub projects (Urli et al., 2018) and coordinating collaborative modeling efforts (Pérez-Soler et al., 2017). Similarly, some bots can help automate the deployment and assessment of SE analysis methods (Beschastnikh et al., 2017). When it comes to TDM, there is less support than for other SE tasks. However, there are some bots that can automate tasks related to TD. Ochoa et al. (2022) proposed BreakBot3 , a bot designed to notify developers if any breaking changes4 happen in libraries currently in use. BreakBot does this by performing a static analysis on the library’s source code and the source code of the software projects that depend on this library. While BreakBot is concerned with managing build TD (one of the nine types of TD outlined by Li et al. (2015)), TagDebt has a much broader scope, relying on NLP to identify SATD items reported in issues. This aspect of TagDebt provides greater flexibility to practitioners. Wyrich and Bogner (2019) introduced Refactoring-Bot5 , which analyzes the source code of projects to identify code smells and automatically refactors them. Code smells, a well-known indicator of TD, include suboptimal code, which in turn compromises the design of the software. Thus, the bot supports TDM by addressing code TD and design TD. Refactoring-Bot presents the refactored changes in the form of pull requests, giving developers the chance to review the changes before integrating them into their software. Mohayeji et al. (2022a) reported on a study examining the impact of TODO Bot6 on the projects where it is adopted. TODO Bot creates issues about source code comments that it identifies as containing the “TODO” keyword, which indicates postponed activities and things that are not quite right yet and need further work. This 3

https://github.com/alien-tools/breakbot A breaking change is a change to supported functionality between released versions of a library that would require a customer to do work in order to upgrade to the newer version. It is advised that library owners (or “language leads”) document for customers what constitutes approved and supported usage of their libraries. 5 https://github.com/Refactoring-Bot/Refactoring-Bot 6 https://todo.jasonet.co/ 4

7

is precisely a form of TD, since these comments highlight “non-optimal or incomplete solutions” (e.g., “TODO: I believe the following code is obsolete”). The findings of this study indicate that the adoption of TODO Bot into software projects encouraged practitioners to introduce TODO comments to increase the visibility of postponed decisions and TD. We note that there is an overlap between the goals of TODO Bot and TagDebt (our bot) in the sense that both can identify TD using natural language. However, while TODO Bot uses source code comments to detect TD, TagDebt is designed to find TD in issues, more specifically in their content (i.e., title, description, or both). Besides, TODO Bot requires developers to use the tag “TODO” in each code comment, while TagDebt does not require any specific tag. Phaithoon et al. (2021) introduced the FixMe bot, which operates by identifying comments in the source code that reference issues that need to be resolved. The bot monitors these referenced issues (including the tag “FIXME”) and notifies practitioners when the issues are resolved, allowing practitioners to fix/improve the modules depending on the issues. The comments identified by FixMe represent an aspect of TD denoted as “Hold TD” to indicate that the code TD is temporarily put “on hold” until the related issue is resolved. Table 1 provides a comparison between existing bots and TagDebt. Specifically in terms of limitations, BreakBot (Ochoa et al., 2022) is effective for preventing breaking-change problems; however, it targets only build TD and depends on being able to statically analyze both the library and its clients, leaving other TD types and artifacts uncovered. Refactoring-Bot (Wyrich and Bogner, 2019) operationalizes code-smell removal through automated refactorings, but it is limited to the smells and transformations encoded in the bot and may not generalize to projects in general. As for TODO Bot (Mohayeji et al., 2022a), it helps increase the visibility of self-admitted TD in code comments, but it relies on explicit “TODO” markers, meaning that it can miss implicit TD and may create noisy or low-priority issues. FixMe (Phaithoon et al., 2021) similarly depends on explicit tags/references (e.g., FIXME and linked issues) and, while it monitors issue resolution, it does not reason about the TD content reported in issues themselves. TagDebt tackles these limitations by shifting the detection to issues, which are artifacts often used by practitioners to report TD concerns (Kashiwa et al., 2022). Since TagDebt relies on NLP to classify SATD-related issues, it can capture multiple TD types (as long as they are documented as SATD) and may support early triage via automatic labeling. Compared to the other bots, TagDebt may reduce adoption friction and does not require specific commenting or tagging conventions. Bots have evolved significantly since their conceptual origin, especially with recent advances in AI, ML, and NLP, which have enabled their widespread adoption across various domains, including SE. In the context of TDM, existing bots such as FixMe work on source code and rely on explicit developer annotations (e.g., #TODO/#FIXME) to identify and track TD, primarily supporting the monitoring and repayment of code-level TD. TagDebt, in contrast, is designed to operate on issue trackers and to capture SATD described in natural language (e.g., refactoring requests). Therefore, it complements existing bots by extending TDM automation to earlier stages of development (i.e., issue management). To the best of our knowledge,

8

Table 1: Comparison of TDM-related bots Bot

Goal

Artifact

Technique

BreakBot

Detect breaking changes and their impact. Find smells and refactor automatically.

Library PRs and library/client code. Project source code.

Static analysis of API changes/usages. Smell detection and refactoring rules.

TODO Bot

Turn TODO comments into issues.

Code comments with “TODO”.

Keyword tion.

FixMe Bot

Track on-hold SATD linked to issues.

Comments (e.g., FIXME) and referenced issues.

SATD detection and issue monitoring.

Multiple Alerts when TD issues close. types (issuereported).

TagDebt (this work)

Label SATD items reported in issues.

Issue title/description.

SATD detection in issues.

Multiple Auto-labels TD issues. types (as long as they are documented as SATD).

RefactoringBot

detec-

TD type(s) Build TD.

Output

Limitation

PR notifications

Narrow scope; needs client and library code. May not fit project constraints.

Code Opens PRs TD, with refacDesign tored code. TD. Multiple Creates TD issues. types (issuereported).

Requires explicit “TODO” tag; misses implicit TD and may generate noise. Depends on explicit tags/references; does not analyze issue content itself. Subject to NLP false positives/negatives.

there are no other TDM bots that classify SATD directly from issue text using an NLP solution, and we explicitly position TagDebt as addressing this gap.

3 Research Method To guide the design and evaluation of the tool, we adapted ?’s framework (i.e., DSR) to the specific context of TDM. As Figure 1 shows, this framework helped us establish a structured process that connects the social context (i.e., the needs and demands reported by practitioners) and the knowledge context (i.e., research about TDM). This adaptation ensured that the framework was not applied abstractly, but was operationalized to address the concrete challenges of this study. In summary, we started in the “Understand the social context” phase to identify the challenges and concerns reported by practitioners about TDM. This exploration raised awareness about the problems in TDM. This led to the “Understand the knowledge context” phase to look for existing solutions and designs for the problems. This phase showed how existing tools are used for TDM and their limitations. Next, in the “Design” phase, TagDebt was designed and implemented using the best practices found in the previous phase. Finally, in the “Investigation” phase, TagDebt was evaluated regarding its usefulness, ease of use, contextual factors for its adoption, and potential improvements. Figure 1 summarizes the phases we followed in this study. The boxes represent 9

the phases we performed, while the arrows show the information exchanged between the phases/contexts.

Social Context Technical Debt Management in Practice Phase 1 Understand the social context Review literature to understand goals and needs for TDM in practice Goals and needs for TDM

New TDM tool

Design Science Phase 3 Design

Tool to be investigated

Phase 4 Investigation

Design and implement a specialized TDM tool

Knowledge and improvements to the tool

Evaluate practitioners' perceptions about the TDM tool

Existing designs of TDM Tools

New design of TDM tool

Phase 2 Understand the knowledge context

Existing knowledge about TDM tools

Review literature to identify exisiting designs of TDM tools Knowledge Context Technical Debt Management Research

New knowledge about TDM tools

Fig. 1: Research Method

We adopted DSR because our objective is to develop and evaluate an artifact (TagDebt) that enhances support for TDM. Given that TDM involves complex socio-technical challenges, ranging from accurate debt identification to effective prioritization and remediation, there is a clear need for solutions that are both practically viable and theoretically informed. DSR provides a structured framework for iterative design, implementation, and validation of such an artifact, ensuring that it addresses real-world constraints while contributing to the scientific understanding of TDM practices. By focusing on the development of the artifact in terms of both empirical evidence and stakeholder input, our goal is to produce a solution that not only supports current software development workflows but also advances the state of knowledge in the field. The four phases of our research method are detailed as follows: 10

3.1 Understand the Social and Knowledge Contexts To explore the social context, we used recent literature on TDM. Specifically, we relied on the results of two recent studies we conducted. First, to deepen the understanding of the concerns practitioners have about TDM tooling, we reused the results of a survey with practitioners (Biazotto et al., 2025b), which yielded 89 valid responses and revealed nine main concerns about TDM tools. The second study was an analysis of practitioner discussions on Stack Exchange (Biazotto et al., 2025a) that offered complementary insights into practitioner perceptions. We also reviewed the Manifesto on Reframing Technical Debt (Avgeriou et al., 2025), which further articulated practitioners’ demands, and we examined related work that focuses on tools for TDM. Accordingly, we note that the understanding of the social context did not involve interacting directly with practitioners or conducting additional data collection or analysis. In this phase, we focused only on reviewing existing literature. To explore the knowledge context, we used the results of a systematic mapping study (SMS) (Biazotto et al., 2023) that identified 122 tools that support TDM. This study allowed us to map the current landscape of TDM tools, along with their main characteristics and challenges. Among those, we further investigated tools that were aligned with the main goal of this study (i.e., a tool easily integrated into the software development workflow). As a result, we reviewed the existing bots that can support TDM, including TODO (Mohayeji et al., 2022a) and FixMe (Phaithoon et al., 2021). These sources provided us with the necessary knowledge to design TagDebt. We presented, in Section 2, the information we gathered after exploring both social and knowledge contexts.

3.2 Design To design a tool for TDM, we used the requirements described by the practitioners in our previous study (Biazotto et al., 2025b). We filtered the requirements to build a complete tool use case, as we explain in Section 4.1. To implement the bot, we followed the guidelines provided by Wessel et al. (2023), which support the development of bots for GitHub. That article highlights recurrent problems identified in existing bots, such as their “noisy” behavior, where bots leave numerous comments in issues or PRs (pull requests) with excessive information, thereby overwhelming developers and disrupting their workflow. It also addresses the overly humanized comments by bots as well as the complexity and poor documentation for configuring them. That article also suggests seven approaches to avoid common pitfalls, which we used to implement the features of TagDebt. We approached the implementation of the requirements in an iterative manner, where the implementation of one requirement followed after the completion and manual testing of the previous one. Testing each requirement involved creating test issues to interact with a deployed instance of the bot and to ensure that it works as expected. Section 4 describes the design and implementation of TagDebt. 11

3.3 Investigation TagDebt aims to provide a specialized solution for TDM. This implies that TagDebt needs to meet the expectations of practitioners. Hence, we assessed to what extent and why practitioners intend to use TagDebt and are willing to adopt it in their daily work. Individual practitioners’ intention to use TagDebt can be determined by three factors: usefulness of the bot, ease of use of the bot, and contextual factors. According to Davis et al. (1989), the usefulness and ease of use play a fundamental role in predicting the degree to which an individual would use a new technology. In addition, the intention to use also depends heavily on contextual factors beyond the influence of a concrete technology, such as the individual’s background, environment setup, and facilitating conditions (Venkatesh et al., 2003). The development of TagDebt was mainly guided by a model of concerns presented in our previous study (Biazotto et al., 2025b). A core concern is that practitioners want to keep control over tools’ execution and outputs, and this control is mainly related to the interaction between tools and practitioners. To explore how well TagDebt can support this concern, we then decided to focus on evaluating the interaction between TagDebt and practitioners. Similar to previous evaluations (Manteuffel et al., 2016; Kemell et al., 2019), we deemed that a TAM-based evaluation is well-aligned with our main goals (i.e., check the interaction between TagDebt and practitioners and how easily TagDebt is integrated into workflows). The details on how we planned and carried out the evaluation of the TagDebt bot will be further discussed in Section 5, and Section 6 reports the results of such evaluation.

4 TagDebt Bot This section introduces the TagDebt bot, which refers to the Design phase of DSR. First, it is important to explain why we suggested this tool to tackle the lack of specialized TDM tools. As we highlighted in Sections 1 and 2, there are some initiatives to identify TD in issues and pull requests. However, existing ML models are bound to local execution and do not have a well-defined interface to interact with practitioners. Therefore, TagDebt intends to be this interface. In fact, we take advantage of the bot’s benefits (e.g., easy integration with development workflows) and combine it with very specialized tools for managing TD (e.g., existing ML models or LLMs for the detection of TD). This goal aligns with the Reframing Technical Debt Manifesto (Avgeriou et al., 2025), which highlights the need for more specialized tools for TDM. We also identified the same concern in Biazotto et al. (2025b), which shows the importance of TDM tools that remain consistent with existing workflows. Finally, Avgeriou et al. (2023) stresses that many TD tools remain difficult to adopt because they are not embedded in the workflows practitioners already use, which increases friction and reduces sustained usage. In TagDebt, this concern is addressed by the operationalization of TD support inside the issue-tracking routine in GitHub (rather than disrupting existing workflows). The TagDebt outputs are produced and consumed in the same place where teams already triage, discuss, and organize work (i.e., via native issue artifacts such as labels), ensuring that the identification of TD is delivered as a native workflow action rather than an external report. 12

4.1 Requirements Elicitation To elicit the requirements for TagDebt, we mainly relied on our previous work (Biazotto et al., 2025b). In that study, we conducted a survey that received 103 responses. The survey was structured in five practical scenarios that illustrate the usage of tools to support TDM. From this study, we collected two main resources: • A practical scenario in which a bot labels issues in issue tracking systems: This scenario was evaluated by the survey respondents as very useful. Since we have the initial evidence of the bot’s usefulness, we deemed that implementing this tool could add value for TDM. • List of requirements for TDM tools: The study provides a list of 46 requirements to be implemented in TDM tools. In this paper, we filtered these requirements to select the ones that would compose a complete use case for a GitHub-based bot focused on issue labeling. For instance, in our previous study, we identified that notifying practitioners based on issue labels is a requirement for TDM tools. This requirement complements the automated labeling provided by the bot because it helps developers pay attention to specific issues. Of the 46 requirements presented in the previous study, we selected five for the first version of the bot and imported ipsis literis from the previous study (i.e., no additional data analysis was carried out on the raw data of the previous study). Taking into account the artifacts mentioned previously, five requirements were elicited to develop a full-featured use case for the bot. The requirements are summarized as follows: • Requirement 1: The bot will label issues that contain SATD. • Requirement 2: The bot will send notifications by email. • Requirement 3: The bot will allow customization of which practitioners should be notified. • Requirement 4: The bot will enable customization of the labels that trigger notifications. • Requirement 5: The bot will enable customization of notifications considering the time since an issue was opened (that is, lingering issues). For Requirement 1, the bot should be able to label issues. This is the core requirement upon which the following four requirements are developed and expanded. It is mainly inspired by the practical scenario presented by Biazotto et al. (2025b). Furthermore, a core recommendation from Avgeriou et al. (2023) is to strengthen automation for TD identification, so that debt can be found early and with minimal overhead. Therefore, TagDebt implements this recommendation by making automated issue labeling the primary focus of Requirement 1. This decision ensures that identification results are not merely diagnostic signals, but become visible to practitioners, who may use the labels for filtering and triaging TD items. Considering this requirement, the TagDebt bot mainly supports the identification of SATD, since it will process issue titles and descriptions to find SATD items reported in such issues. In Requirement 2, the bot should send a notification to the practitioners by email; this would help to increase the visibility of issues that contain SATD items. To refine this feature, Requirement 3 defines that the bot should allow practitioners

13

to configure who should receive emails about issues. In most software projects, there are usually multiple developers working on different parts of the software. Because of this, the email notifications sent by the bot might only concern a certain group of developers, while others might not be interested in receiving them. Next, Requirement 4 states that the bot should include an option to configure the labels that trigger email notifications. As mentioned earlier in this section, the bot has a SATD detection component (i.e., either the ML model by Li et al. (2022) or GPT-5-mini) that generates and assigns labels to issues based on the title, description, or both. While the current version of the bot provides two types of labels (i.e., SATD or non-SATD), implementing this requirement makes the bot ready to accommodate other SATD detectors that can provide multiple types of labels. By allowing practitioners to select the specific labels that should trigger email notifications, the bot can enable them to prioritize and focus on the issues they find most important. To fulfill Requirement 5, the bot should provide an option to configure email notifications considering the time since an issue was opened. Issues that are open and have not been resolved or have not progressed in a long time are called lingering or stale issues. If these lingering issues accumulate, they might diminish the visibility of urgent or higher priority issues. In addition, if these problems persist over time, they can become outdated and confuse developers who join software projects at later stages (Wessel et al., 2023). Therefore, by developing and implementing this feature in the bot, practitioners can receive periodic updates about lingering issues, enabling them to decide on the appropriate course of action.

4.2 Architecture of TagDebt Figure 2 illustrates the overview of TagDebt’s architecture using a Systems Context Diagram from the C4 model7 , highlighting the relationships between the components of the bot and the GitHub repository it interacts with and operates on.

4.2.1 Bot’s Plug-in To use TagDebt on a GitHub repository, developers need to install its plug-in. The installation must be carried out by individuals authorized to modify the repository settings, typically the repository maintainers. Once the bot’s plug-in is installed, any other project contributor with equal or lower permissions can use and interact with TagDebt. The bot’s plug-in is a GitHub App8 . When new GitHub issues are created or comments are posted on these issues, the GitHub App triggers webhook notifications (web requests) and forwards them to the bot’s application (its backend). These requests provide all the necessary information in their payload for the bot to perform certain actions. Besides sending webhook notifications, the bot’s plug-in is responsible for establishing the connection between GitHub and the backend application of our bot, acting 7 8

https://c4model.com/diagrams/system-context https://developer.github.com/apps/about-apps/

14

Practitioner [Person] A person that uses TagDebt for managing technical debt in Github Issues. Manages Issues (e.g., open, comment)

Github Issues [Software System] The Github application to manage issues in repositories.

Installs on Github repositories

Is installed on

Interacts with

Bot's Plugin [Software System] A Github App that can be installed in different Github repositories.

Makes API calls

Send webhooks

Bot's Application [Software System] Application that implements the connection between Bot's Plugin and the Machine Learning Model that identifies Technical Debt Requests labels

Returns Labels

SATD Detection Module [Software System] Module that implements the interface to access the SATD detection solution (e.g., ML model)

Fig. 2: Overview of TagDebt’s architecture

as an intermediary for any type of communication that happens between the two. The bot’s GitHub App is depicted in Figure 3.

4.2.2 Bot’s Configuration File Once the bot’s plug-in is successfully installed on the repository, the bot must be configured. Configuring TagDebt is an essential step, allowing developers to specify how they want the bot to operate in different scenarios. This task can be performed by any project contributor with write access to the repository. During the design of the configuration file, we tried to reduce the complexity as much as possible. This concern is aligned with Avgeriou et al. (2023), which states that minimizing configuration overhead is crucial to increase the adoption of TDM tools. Specifically, TagDebt is configured through a simple configuration file placed in the repository. Besides, TagDebt provides a default configuration stored in its backend for off-the-shelf use 15

Fig. 3: Visualization of TagDebt’s GitHub App

when teams prefer minimal setup. This design keeps setup lightweight and consistent with common development practices (versioned configuration in the repo), reducing the need for specialized tooling expertise. To configure the bot, a config.json file must be included on the main branch of the repository, within a Bot folder. However, there is also a default configuration stored in the bot’s backend, in case developers want an off-the-shelf solution. We opted for a JSON file format to configure TagDebt because its syntax is straightforward, and JSON files are widely used to configure different kinds of tools and applications (Harrand et al., 2021). For example, Visual Studio Code (VS Code), one of the most popular code editors developed by Microsoft, uses JSON files named settings.json to allow users to customize various settings associated with the editor9 . Thus, it is fair to assume that developers are already familiar with this type of procedure. The config.json file contains multiple configuration options for the bot, each represented by a field paired with its corresponding value. Whenever TagDebt performs an operation, such as processing a command to label an issue using its detection function, it retrieves the latest config.json file from the repository. This way, we ensure that the file is always updated. Figure 4 presents the main configuration options within config.json. Due to space limitations, some fields are omitted in the figure; however, all options and their descriptions can be found in the bot’s Wiki page10 .

4.2.3 Bot’s Application The bot application, or backend, is structured into three different modules, each with a different purpose, to enforce the separation of concerns and enhance modularity. To represent the Bot’s application, we used a Component Diagram from the C4 model11 , as depicted in Figure 5. Each module is described below. 9

https://code.visualstudio.com/docs/getstarted/settings https://github.com/biazottoj/tagdebt-bot/wiki/Documentation https://c4model.com/diagrams/component

10 11

16

Caption object array numeric field text field boolean field

Fig. 4: TagDebt’s default configuration file

App The app.py module coordinates the overall functionality of the bot. It handles the logic for processing incoming GitHub webhooks generated by the bot’s GitHub App. To ensure security, this module validates these requests using an HMAC (Hash-based Message Authentication Code) to confirm they originate from GitHub. Additionally, it retrieves the latest configuration file (config.json) stored in the repository, if available, by requesting it from the GitHub API via the bot’s GitHub App. Upon receiving a webhook, the module parses the payload to determine the type of event it needs to handle: an issue creation event or an issue comment event, and routes it to the appropriate handler function. • For issue comment events, it examines the content of the comments, and if it identifies a command instructing the bot to label the issue, the module communicates with the SATD Detection Module (i.e., uses either the ML model by Li et al. (2022) or the LLM-based plugin) to classify whether the issue contains TD, and subsequently assigns the predicted label to the issue. • Alternatively, if it identifies a command instructing the bot to label an issue with a specific label, it assigns the provided label directly to the issue, without invoking the detection function. • For issue creation events, the module can post welcome comments, automatically generate labels using the detection function, if configured to do so, and assign them to issues. 17

Bot's Plugin [Software System] A Github App that can be installed in different Github repositories. Send webhooks

Lingering Issues Processor [Component: Python File] A Python file that implements the logic to monitor issues that are inactive for a certain time.

Makes API calls

Provides information

App [Component: Python File]

Provides information

A Python file that implements the core functions of the bot (e.g., process webhooks).

Provides information Email Sender [Component: Python File] A Python file that implements the logic to build and send email to practitioners.

Requests labels

Bot's Application

Returns labels

[Container]

SATD Detection Module [Software System] Module that implements the interface to access the SATD detection solution (e.g., ML model)

Fig. 5: Overview of TagDebt’s Application Component

Email Sender The component is responsible for sending an email to practitioners. It checks whether the conditions necessary to trigger email notifications are met, based on the config.json file, and the information provided by either the main module or the lingering issues processing module. If these conditions are met, it sends the emails accordingly. Such conditions include the labels generated by the detection function and the time since issues were opened or were last active.

Lingering Issues Processor This component periodically checks for lingering (or stale) issues in the repositories where the bot’s plug-in is installed to notify practitioners about them. This process is scheduled by the main module to run at specific intervals, which can be specified by the practitioner within the source code of the bot’s application before starting it (if not specified, it defaults to running every day). 18

4.2.4 TagDebt’s Frontend TagDebt is designed to operate autonomously, requiring no human intervention aside from setting it up according to the documentation12 . Therefore, TagDebt does not require a dedicated frontend. Instead, GitHub itself serves as the “frontend” for interacting with the bot, and developers send commands to TagDebt through GitHub issue comments. It is worth noting that some bots, such as the FixMe GitHub bot (Phaithoon et al., 2021), employ a frontend to allow developers to configure it through a GUI. While this might represent a more visually appealing way of configuring the bot, it essentially holds the same value as doing it directly in the repository by modifying the config.json 13 file using GitHub’s integrated text editor, as is the case with TagDebt.

4.3 TagDebt’s Execution Flow To help understand TagDebt’s execution flow and how most of its components interact, we present an example usage scenario, which we will explain using Figure 6. To represent this usage scenario, we used a Dynamic Diagram from the C4 Model14 . To simplify the understanding of this scenario, we omit the modules related to email sending and the lingering issues checker from the Bot’s Application container. The numbered arrows in the figure indicate the sequence of steps performed in this scenario.

1: Commands the Bot through a issue comment

A person that uses TagDebt to manage technical debt in Github Issues.

Module that implements the interface to access the SATD detection solution (e.g., ML model)

The Github application to manage issues in repositories. 2: Executes the Github App

Practitioner [Person]

SATD Detection Module [Software System]

Github Issues [Software System]

5: Requests label

8: Assign a label to an issue

Bot's Plugin [Software System]

3: Send Webhooks 4: Make API call to retrieve configuration file

A Github App that can be installed in different Github repositories.

7: Make API call to provide a label to an issue

6: Returns label

App [Component: Python File] A Python file that implements the core functions of the bot (e.g., process webhooks). Bot's Application [Container]

Fig. 6: TagDebt’s execution flow Consider a developer who posts a comment on a GitHub issue commanding TagDebt to label that issue (Step 1) (i.e., a /tdbot label comment). Once the comment is posted, GitHub Issues executes the bot’s plug-in (Step 2) and sends a webhook notification about this comment to the main App module component (Step 3). The component first validates this webhook event and subsequently requests and retrieves 12

https://github.com/biazottoj/tagdebt-bot/wiki/Quick-Start The complete documentation for the TagDebt bot can be found at https://github.com/biazottoj/ tagdebt-bot/wiki/Documentation 14 https://c4model.com/diagrams/dynamic 13

19

the latest configuration file from the repository by making a request to GitHub’s API via the bot’s plug-in (Step 4). Next, the App component handles the issue comment event by parsing the content of the comment and identifying the issue labeling command. It then determines which part of the issue (i.e., title, description, or both) should be analyzed to predict whether the issue contains SATD. The part of the issue that should be analyzed is based on the settings specified in the configuration file. Once determined, a request is sent to the SATD Detection Module (step 5). In step 6, the SATD Detection Module responds with the predicted label (“TD” or “non-TD”). Following this, the app.py module requests GitHub’s API, via the bot’s plug-in, to assign the predicted label to the corresponding issue (step 7). Finally, in Step 8, the Bot’s Plugin assigns the provided label to the issue.

4.4 Requirements Implementation In this section, we present how we implemented each requirement we described in Section 4.1. First, implementing Requirement 1 demands deploying or using an NLP solution to identify SATD in issues. We considered that NLP solutions could be deployed and replaced, considering that AI for the identification of TD is constantly evolving (Li et al., 2023; Avgeriou et al., 2025). Therefore, we decided to implement a factory design pattern (Gamma et al., 1994), allowing the system to dynamically integrate various NLP solutions without modifying the core application logic. This decision is also aligned with a suggestion from Avgeriou et al. (2023), which proposed AI-based capabilities as a key enabler for next-generation TDM tooling. In TagDebt, SATD identification is fully based on AI detection functions. Moreover, in TagDebt, the detection function is encapsulated behind a consistent interface (Figure 7), allowing practitioners to employ different AI alternatives while preserving the same workflow behavior (i.e., issue labeling). The current version of TagDebt (v1.1) implements two detection alternatives, i.e., two plugins. The first plugin wraps the Text CNN model proposed by Li et al. (2022), which was trained to identify SATD in issue tracking systems. In that study, the authors built a dataset of 4,200 issues (23,180 issue sections, including 3,277 SATD sections) and evaluated the optimized Text CNN via stratified 10-fold cross-validation, reporting an average precision of 0.685, recall of 0.689, and F1-score of 0.686. We use Li et al. (2022) as a reasonable default baseline because: (i) it was formally trained to detect SATD in issues, which aligns with the bot’s operating context (issue trackers); (ii) it supports reproducibility, as Li et al. provide a thorough replication package, including examples and training data, which increases the reliability and verifiability of the baseline; and (iii) it offers ease of integration, since its API is simple and reduces engineering overhead to integrate it into a GitHub bot; and (iv) it was a state-of-theart model at the time we first envisioned the study and implemented the bot (which predated the recent LLM-based advancements in TD detection). At the same time, TagDebt’s detection engine is modular and easily replaceable exactly for this reason: different contexts may benefit from different models, and swapping the engine allows practitioners and researchers to adapt TagDebt without being constrained to a single classifier. 20

satd_llm_instructor

satd_li_2022

Fig. 7: TagDebt’s detection module and two examples of compatible NLP solutions (i.e., satd li 2022 and satd llm instructor) The second plugin wraps an LLM-based approach to prompt OpenAI, Anthropic or Google Gemini models. Through this plugin, we also evaluated OpenAI’s GPT5-mini model using the issue dataset from Li et al. (2022). To this end, we used all SATD entries from Li et al. (2022) (n=1089) and an equally-sized random sample of non-SATD issue descriptions (i.e., N=2178). We needed to sample the non-SATD randomly because there were many more non-SATD samples in the dataset. Specifically, the GPT-5-mini model achieved a precision of 0.759, recall of 0.722, and F1-score of 0.740. The plugin setup is as simple as choosing the model in the configuration file (via a string) and adding the API key to the environment variables. If the developer wishes, the prompt text, API call configurations, and model can be easily adjusted. We chose GPT-5-mini for the evaluation because it is more affordable than the larger counterparts, while providing good performance for SATD classification, e.g., as recently reported by Maarleveld et al. (2026). Finally, it is important to note that the LLMbased plugin evaluation needs to be interpreted as proof-of-concept regarding potential alternatives that could be used as the detection function of TagDebt. The core contribution of this paper (according to the evaluation presented in Section 3.3) concerns the mechanisms provided by the bot to integrate SATD detectors into GitHub, as well as its perceived usefulness and ease of use. Following the C4 model, in Figure 7, we present a Code Diagram15 , using the UML Class Diagram notation. The module factory.py manages the creation of instances of the detection function (e.g., LLM plugin). The factory has a registry where the plugins are registered 15

https://c4model.com/diagrams/code

21

with their initialization functions16 . The initialization functions are then used to add new detection solutions to the registry, associating them with their respective constructor functions. The detection solutions are instantiated dynamically based on the “type” field, which defines which detection plugin will be used for the classification, based on the provided configuration file. The bot’s backend interacts with the detection plugins via a REST API, sending issue data and receiving classification results (i.e., a tag, either “TD” or “non-TD”). To implement Requirement 2 (i.e., email notifications), we used Python’s built-in smtplib and the SMTP protocol. This approach allows direct email sending through any compatible server, including Gmail and Outlook. The functionality was encapsulated in a separate module, emailSender.py, which defines a function that is called by the main application. Practitioners can configure an SMTP server with credentials securely stored in a hidden file. Each email session establishes and terminates a connection to prevent idle timeouts and unnecessary resource usage. For email content, the MIMEText class from email.mime.text is used to create plain text messages. Secure transmission is ensured through TLS encryption, which is widely supported and preferred over SSL due to known vulnerabilities. To configure notifications, two fields can be manipulated in the bot file config.json: a) a boolean flag, i.e., send-emails, that defines if notifications should be sent; and b) the field when-to-send, which specifies when emails should be triggered (e.g., when a certain tag is added to an issue). As for Requirement 3 (i.e., who the bot will notify), the bot’s configuration file contains a field named recipients within the email-info section. This field contains a list of email addresses of practitioners who wish to receive email notifications from the bot. The decision to allow practitioners to customize their email notification preferences using the existing bot configuration file was based on the aim of incorporating all settings in one place, instead of maintaining a separate file for email addresses, in an effort to reduce the complexity of bots’ configurability highlighted by Wessel et al. (2023). Practitioners have the flexibility to opt in or out of receiving email notifications from the bot at any time by simply modifying the configuration file in the repository to add or remove their email addresses. For Requirement 4 (i.e., which labels trigger notifications), we use two fields and two sections in the bot’s configuration file (config.json): • the label field within the email-body-template section; and • the label field within the email-subject-template section. To provide further customization, we introduced placeholders (denoted “/placeholder”) that can be used in the email body templates. The bot email sender parses the templates and replaces the placeholders with data from the labeled issue. For instance, the “/label” placeholder is replaced with the label generated by the detection function and assigned by the bot to the issue, and /issue link is replaced with the hyperlink to the issue where the bot assigned the label. Finally, to implement Requirement 5 (i.e., lingering issues), the bot includes customizable functionality to identify and alert users about issues that remain open or unresolved beyond a defined time threshold. Using the BackgroundScheduler 16 We assumed a plugin would have a method/endpoint to be called and trigger a classification, similar to the models implemented by Li et al. (2022) and Liu et al. (2018).

22

class17 from the Advanced Python Scheduler library18 , the bot runs periodic checks in the background without interfering with other operations. The logic for detecting lingering issues is implemented in the lingeringIssuesProcessor.py module, while scheduling is handled in app.py, with customizable intervals defined by the lingering check frequency parameter within the config.json file. Two key configuration options (i.e., lingering-issue-threshold and lingering-mode) allow practitioners to specify how many days define an issue as lingering and whether this is based on creation or last modification date. Additionally, the bot supports customizable email templates for both subject and body content, with a dedicated prepare lingering email helper function added to the emailSender.py module to format and send these notifications.

5 Design of TagDebt Bot Evaluation In this section, we detail the design of TagDebt’s evaluation. Specifically, we report its main goal, research questions, and the methods we employed for data collection and analysis.

5.1 Objective and Research Questions In this study, our primary goal is to design and develop a tool that is aligned with recent demands for TDM tools, as reported in the literature (e.g., the Manifesto on Reframing Technical Debt (Avgeriou et al., 2025)). Such recommendations involve embedding SATD identification into existing development workflows (in our case, we decided to integrate TagDebt with GitHub issues) with low configuration overhead and without requiring developers to change their current practices. In other words, the central research problem we address is the lack of workflow-based, adoptable TDM tools, rather than the lack of high-performing classifiers per se. Besides, most of the design decisions in TagDebt development are focused on practitioners (e.g., do not require changes to developers’ practices). Hence, practitioners’ perceptions of usefulness, ease of use, and contextual factors are the most suitable evaluation criteria to determine whether the artifact actually responds to the identified problems. We structured the evaluation objective according to the Goal-Question-Metric template (van Solingen et al., 2002) as follows: “Analyze software practitioners’ opinions for the purpose of evaluating TagDebt with respect to its usefulness, ease of use, and contextual factors from the point of view of practitioners with varied roles in the context of both industrial and open-source software (OSS) development.” Based on the goal, the TagDebt bot’s evaluation comprised four RQs as follows: RQ1 - How do practitioners perceive the usefulness of TagDebt? Considering the definition of perceived usefulness proposed in (Venkatesh et al., 2003), we investigate to what extent practitioners believe that TagDebt supports TDM and enhances their performance while managing TD. 17 18

https://apscheduler.readthedocs.io/en/3.x/modules/schedulers/background.html https://apscheduler.readthedocs.io

23

RQ2 - How do practitioners perceive the ease of use of TagDebt? Based on the definition of perceived ease of use proposed in (Venkatesh et al., 2003), we want to know to what extent practitioners believe that using TagDebt will not require much effort due to its ease of use. RQ3 - What are the contextual factors and how do they influence the intention to use TagDebt? In practice, the usage of a tool often depends on external factors, for example, the nature and size of a software project, the preferred way of working of individuals, or corporate culture and guidelines. This RQ intends to elicit and understand the context that leads to a successful application of the bot in real-world projects. Furthermore, this RQ allows us to evaluate to what extent our results are applicable and transferable to the software industry. RQ4 - What features could be implemented to improve TagDebt? This RQ aims to identify concrete suggestions for new features or enhancements that could increase the value of TagDebt. These suggestions may stem from gaps observed by practitioners, unmet needs during usage, or opportunities to better support existing workflows. By understanding how TagDebt can be improved, we can guide its future development, increase user satisfaction, and increase its adoption.

5.2 Data Collection Based on our goal and RQs, we considered multiple acceptance models for evaluating TagDebt, including classical TAM (Davis et al., 1989) and extended models, e.g., UTAUT (Venkatesh et al., 2012). After analyzing their constructs, we deemed classical TAM sufficient because its core constructs overlap with UTAUT’s core perspectives (i.e., Perceived Usefulness and Perceived Ease of Use align with Performance Expectancy and Effort Expectancy), and these constructs are the ones most directly related to our focus on early acceptance and adoption barriers of a GitHub-integrated bot. TAM is widely adopted to understand the acceptance of users of information systems. In SE, TAM has been instrumental in evaluating the adoption of software tools, platforms, and processes by developers. TAM can evaluate two primary factors: usefulness and ease of use. Usefulness reflects the extent to which a user believes that a system enhances job performance, while ease of use refers to the perceived effort required to use the system. According to Davis et al. (1989), these two factors influence an individual’s intention to use a given technology. Since this is an exploratory study, we chose interviews instead of a Likert-scale questionnaire (which is more common when applying TAM). This is because a purely quantitative Likert-scale survey would provide limited explanatory depth, which goes against our goal to elicit richer qualitative insights, including contextual factors, concerns, and improvement opportunities that explain why practitioners perceive TagDebt as useful or easy to use (or not). To apply the TAM-based interviews, we developed a questionnaire based on previous studies (e.g., (Babar et al., 2007; Manteuffel et al., 2016; Ferreira et al., 2024)), adapting the constructs to the context of TDM. Our main objective was to 24

evaluate the extent to which TagDebt is perceived as useful in identifying TD items, reported as SATD, during the software development process. We opted for openended questions because they allow us to have in-depth discussions and ask follow-up questions, which help to better understand practitioners’ opinions. In Table 2, we list the constructs that we used to define each question in our interview guide, as well as their corresponding constructs in UTAUT. For instance, we used the construct “usefulness” to define U1 - Do you think that the bot would help to identify and monitor TD items more quickly and easily?, and then investigated whether TagDebt would help to improve practitioners’ productivity.

Table 2: Mapping between our interview questions and TAM/UTAUT constructs. ID U1

TAM Perceived Usefulness (PU)

UTAUT Performance Expectancy (PE)

U2

Perceived Usefulness (PU)

Performance Expectancy (PE)

U3

Perceived Usefulness (PU)

Performance Expectancy (PE)

U4

Perceived Usefulness (PU)

Performance Expectancy (PE)

E1

Perceived Ease of Use (PEOU)

Effort Expectancy (EE)

E2

Perceived Ease of Use (PEOU)

Effort Expectancy (EE)

E3

Perceived Ease of Use (PEOU)

Effort Expectancy (EE)

E4

Perceived Ease of Use (PEOU)

Effort Expectancy (EE)

Open External variables

Facilitating Conditions

Rationale Assesses whether embedding TD labeling in GitHub is perceived as reducing the time and effort required to identify TD, which is central to our goal of evaluating perceived usefulness as a proxy for acceptance. Captures perceived productivity/coverage gains, supporting our goal of understanding whether practitioners perceive practical value in adopting TagDebt for triage. Targets perceived effectiveness from the practitioners’ perspective (i.e., whether outputs seem to support identifying the right items), which informs feasibility and perceived usefulness in workflow contexts. Provides an overall usefulness judgement to triangulate U1–U3 and summarize perceived value, aligning with our goal of assessing acceptance in an exploratory evaluation. Assesses learnability and onboarding friction, which are critical barriers to adoption for GitHub-based automation and directly relate to our focus on ease of installation and use. Configuration overhead is a key adoption barrier for workflow-integrated tools; this question evaluates perceived setup effort and surfaces context constraints affecting feasibility. Captures memorability and sustained operability (low cognitive overhead), which influences whether a tool remains viable beyond a first trial, consistent with our acceptance-focused goal. Documentation quality affects perceived effort and adoption readiness; assessing it supports our goal of identifying concrete improvement points for the bot. Elicits missing capabilities, constraints, and improvement opportunities that shape adoption beyond PU/PEOU, consistent with our goal of collecting practitioner concerns and requirements for enhancing TagDebt.

Interview question Do you think that the bot would help to identify and monitor TD items more quickly and easily?

Do you think using the bot would help you to identify and monitor a higher number of TD items? Do you think using the bot would help you to correctly identify and monitor the right TD items?

Do you think the bot is useful for identifying and monitoring TD items? Was it easy to learn how to operate the bot?

Was it easy to configure the bot?

Do you think it would be easy to remember how to use the bot?

Do you find the documentation easy to use?

Is there any other feature you are missing?

To evaluate the questionnaire, we performed a pilot study. This study aimed to ensure the understandability, adequacy, and feasibility of our interview instruments prior to conducting full-scale interviews. A practitioner from our personal network was invited to join the pilot study and assigned to use the bot for a few days before the interview. From the pilot study, we observed that the interview would last around 25

30–40 minutes; we deemed such a duration acceptable to gather relevant data, without tiring the interviewees. In addition, this pilot study allowed us to refine the interview instruments. For example, we initially considered conducting structured interviews, i.e., asking each question individually and in a specified order. Nevertheless, we noted that a semi-structured interview would enable us to collect more insights. We updated the instruments accordingly and used the questions as a checklist (i.e., topics/constructs we should discuss during the interview). The interview transcript from the pilot study was not coded, but it is also available in our replication package. The population of our study comprises practitioners who are involved in realworld projects, since they are the audience that may potentially use our bot. In terms of recruitment, we adopted a convenience sampling strategy by approaching practitioners from our own professional network and complementing it with a snowballing approach, i.e., asking interviewees to indicate other practitioners who could also contribute to the evaluation. Convenience sampling (Linåker et al., 2015) is a non-probabilistic sampling, in which it is not possible to observe randomness in the selected units from the population. Therefore, there is a trade-off when using convenience sampling: while participants can be recruited quickly, and at low cost and effort, this sampling strategy can limit generalizability (Baltes and Ralph, 2020). To mitigate this threat, before recruiting interviewees, we defined a sampling model, according to Table 3, to ensure diversity in the backgrounds of the respondents. Although we tried to interview as many practitioners as possible, our main goal was to cover a representative range of profiles that could provide varied perspectives on TagDebt. In particular, as a key inclusion criterion, interviewees needed to have industrial software development experience (either current or past); therefore, we did not interview individuals with no industrial experience. Besides, we deemed that two characteristics could directly impact respondents’ perceptions about the bot: • The practitioner’s role influences their demands when managing TD (Rios et al., 2020), and it might impact their perceptions about TagDebt’s usefulness. For example, practitioners might focus on information that leads to actionable TDM tasks such as refactoring code, while managers might focus on strategic information that would help with planning and decision-making. To select the roles, we relied on (Rios et al., 2020) (reported on page 12, Table 4, in their manuscript), since their work reports a thorough design of a global survey on TD. The selected roles are mainly based on the TD types and cover all the software development life cycle (SDLC). As shown in Table 3, four roles were considered: Software/System Architect, Test analyst/Q&A Analyst, Developer/Software Engineer, Project Manager/Team Lead. • The practitioner’s experience may also influence their perception of tools (Rios et al., 2020); therefore, we had to balance our sample to consider a wide range of experience levels. To define this aspect, we based the categories on a previous survey we reported in (Biazotto et al., 2025b) and defined 3 intervals of experience: 1 to 5 years, 6-10 years, and more than 10 years. We did not select participants with less than one year of experience because novice practitioners tend to perform simpler tasks that do not involve TDM.

26

Table 3: Sampling Model Themes Role Software/System Architect Test Analyst/Q&A Analyst Developer/Software Engineer Project Manager/Team Lead Experience 1 - 5 years 6 - 10 years more than 10 years

Respondents r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14 r15 r16

✓ ✓ ✓

✓

✓

✓

✓ ✓

✓

✓

✓ ✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓ ✓

✓

✓

✓ ✓

✓

✓

✓

To run the interviews, we asked the practitioners to install and use TagDebt to open (and inspect) issues, without enforcing a specific usage scenario. In particular, we allowed respondents to choose the context that made the most sense to them, such as a real project they work on, a toy repository, or simulated issues created only for the assessment. While this flexibility may reduce comparability across participants, it enabled practitioners to assess the bot in a context that is meaningful and realistic for their own workflow. Moreover, since this is an exploratory study focused on early feedback, we believe that a strictly controlled scenario could have limited the qualitative insights we aimed to collect. Finally, the model provided by Li et al. (2022) was used as the default one for the evaluation. However, we note that practitioners did not evaluate the detection function itself, but the bot’s usefulness and ease of use. This means that the detection function does not impact the evaluation setup. Given this evaluation setup, we also did not systematically collect detailed information about participants’ current working environments (e.g., organizational maturity, existing TD processes, team distribution, or technology stack). Instead, we deemed it more appropriate to characterize participants in terms of their industrial experience and roles and to rely on their accumulated experience when reflecting on usefulness and ease of use, rather than attempting to map perceptions to a specific organizational setting that may not have been the one used during the evaluation. Nonetheless, we elicited and reported contextual factors that practitioners expect to influence adoption (e.g., team size and project/codebase characteristics), providing a proxy for the contexts in which TagDebt would be more likely to be used. During the interview, each participant was characterized regarding their role and experience and asked to provide a general impression of using TagDebt. Then, specific questions about the usefulness (RQ1) (e.g., “Do you think that the bot would help to identify and monitor TD items more quickly and easily?”) and ease of use (RQ2) (e.g., “Was it easy to learn how to operate the bot?”) of the bot were asked. Regarding contextual factors (RQ3 ), they were extracted from follow-up questions to the practitioners. For instance, a respondent made the following statement during the interview: “r3: I saw that you probably designed it for people who work in large teams. Maybe they have a bottleneck when they have to assign labels.” Hence, we followed up 27

to understand if the team size is a factor that would influence the bot’s usage, which was further confirmed by that respondent while coding the transcript; so, we assigned the code “contextual factor - team size” to that incident. At the end of the interview, practitioners had the opportunity to suggest improvements for the bot (RQ4 ). Each interview was recorded with the authorization of the participants, who were informed that their participation was voluntary and they could withdraw their data at any time.

5.3 Data Analysis To answer our four RQs, we used thematic synthesis to recognize, interpret, and present patterns (themes) within the data. Since the data we collected are mainly qualitative, we deemed this qualitative research synthesis method appropriate for this study. We followed the five steps defined in (Cruzes and Dyba, 2011), as depicted in Figure 8 and described below: Step 3

Step 4

Translate codes into themes

Create a model of higher-order themes

Step 3.1 5 Define themes for usefulness

Step 4.1 Define categories 5 for usefulness

8 themes for usefulness

3 categories for usefulness

Step 3.2 5 Define themes for ease of use

Step 4.2 Define categories 5 for ease of use

Step 1 Obtain data

Step 2 Apply open5 coding on interview transcripts

6 themes for ease of use

3 categories for ease of use

16 interview transcripts

106 codes (260 indicators)

Step 3.3 5 Define themes for contextual factors

Step 4.3 Define categories 5 for contextual factors

5 themes for contextual factors

1 categories for contextual factors

Step 3.4 5 Define themes for bot improvements

Step 4.4 Define categories 5 for bot improvements

30 themes for bot improvements

4 categories for bot improvements

Step 5 Assess the trustworthiness of the synthesis Discussion of credibility, confirmability, dependability, and transferability

Fig. 8: Data analysis method for TagDebt evaluation

• Step 1 - Obtain data: After conducting the interviews with the 16 practitioners, we performed the transcription of each interview to make a systematic data extraction possible. 28

• Step 2 - Apply open coding on interview transcripts: We applied the open coding technique to break down practitioners’ answers into smaller pieces of information (referred to as indicators). Those were represented by a code (e.g., a word, phrase, label, etc.), which was used to navigate through different indicators and compare them using constant comparison. For example, one practitioner stated that: “r1: And then eventually you can assign based on the label to different people. And based on that, you can prioritize and organize your priorities.” Hence, we assigned the code usefulness - the bot could support prioritization to that indicator, which helped us compare that indicator with others. To code the data, we used an integrated approach (Cruzes and Dyba, 2011). Initially, two authors coded a sample of 16 answers (which were randomly selected), generating as many codes as possible from the data. After that, they discussed the generated codes until reaching an agreement. Next, the first author coded all the transcripts, using both the previously generated set of codes and proposing new codes, as necessary to better represent the indicators. After that, the four authors (the two authors who were involved in the open coding, plus the last two authors) discussed the codes and reached an agreement on the indicators and codes to represent them. Considering that we had three aspects to consider (i.e., usefulness, ease of use, and contextual factors), we assigned an aspect to the codes (e.g., “usefulness - team size”) to help us cluster the information about those aspects. The application of open coding resulted in a set of 106 codes and 260 indicators. Table 4 presents a few examples of these codes and indicators. Table 4: Example of coding process answer OK, first of all, classification and categorization, but that’s general for any kind of labels or tags. The thing is you can easily group them and see the status of your project, the status of your code. Yes, it was. It was very easy, and I could understand from the first reading what you wanted with each option there.

author 1 simplifies aggregation

author 2 helps in classifying issues

final code usefulness helps in classifying issues.

good documentation

documentation is simple

usefulness - documentation is simple

• Step 3 - Translate codes into themes: A theme emerges through the process of coding, categorization, and reflective analysis, and represents an abstract construct that imbues recurring experiences with meaning and coherence. In this step, we mainly used selective and theoretical coding, which enabled us to translate codes into themes. The emphasis of this approach is constant comparison, which helps to mitigate potential biases and enhance data exploration. For example, after performing open coding, we obtained codes such as “usefulness - bot can support planning TD repayment” and “usefulness - the bot can support with issue 29

prioritization.” During selective coding, these codes were grouped, and the theme “Prioritizing TD items” was defined. In theoretical coding, we then grouped themes into broader categories. For instance, the themes “Prioritize TD items” and “Increase TD awareness” were grouped under the category “TD Visibility”. We applied this same procedure to all RQs: for each RQ, we independently translated its codes into themes and then into categories (i.e., Steps 3.1 to 3.4 in Figure 8) so that the themes emerged individually and specifically for each RQ. It is important to note that we did not adhere strictly to all steps of Grounded Theory (Strauss and Corbin, 1990), and therefore, the resulting themes cannot be claimed as a fully developed theory. Interactive data collection and analysis were not employed; thus, theoretical saturation cannot be guaranteed, a point further discussed in Section 8. • Step 4 - Create a model of higher-order themes: The themes that emerged in the previous step were explored and interpreted to create a model consisting of higher-order themes. In this study, we propose a set of categories of themes related to the three aspects of adopting TagDebt (usefulness, ease of use, and contextual factors). For instance, themes such as “Reduces repetitive or manual work” and “Helps prevent human errors” were grouped under the category “Developer Support”. We defined the categories individually for each one of the RQs (i.e., Steps 4.1 to 4.4 in Figure 8). Section 6 presents and discusses these categories. • Step 5 - Assess the trustworthiness of the synthesis: Assessing the trustworthiness (or reliability) of interpretations derived from thematic synthesis involves constructing arguments supporting the most plausible interpretations. Research findings do not have a singular correct interpretation but rather the most probable interpretation from a specific standpoint. Therefore, enhancing trustworthiness entails presenting findings in a manner that encourages readers to consider alternative interpretations. To evaluate the trustworthiness of the results, we examined the concepts of credibility, confirmability, dependability, and transferability. Section 8 details this evaluation.

6 Results of TagDebt Evaluation This section presents the results for the Investigation phase of DSR. Overall, the results are based on the themes related to usefulness, ease of use, contextual factors, and improvements. Figure 9 summarizes the themes identified for the four RQs, which are further discussed in Sections 6.1 to 6.4.

6.1 Usefulness of TagDebt To answer RQ1 , we investigated how practitioners perceive the usefulness of the TagDebt bot. Overall, participants identified multiple ways in which TagDebt adds value to TDM. The topics related to usefulness were classified into three categories: (i) TD Visibility; (ii) Developer Support; and (iii) Limitations. Table 5 shows the themes within each category. 30

Installing and Configuring Classifying TD Items

TD Visibility

Simple to install, configure, and operate Configuration file is clear

Increasing TD awareness Prioritizing TD items Reducing repetitive or manual work Helping prevent human errors

Ease of Use Developer Support

Learning and Understanding

Usefulness

Documentation is simple and complete Documentation has some limitations Remembering how to use the bot

Commands as comments are not ideal

Low feature overhead

Operating

Standardizing issue report Classification Precision Descriptive labels and classification

TagDebt Limitations

Improvements Codebase Size Team Size Practitioners’ Roles Privacy and Security Policies Type of Software

Configuration and Setup Integration and Analysis Feedback for users Label management

Contextual Factors

Fig. 9: Summary of the themes identified for usefulness (RQ1 ), ease of use (RQ2 ), contextual factors (RQ3 ), and improvements (RQ4 )

Three themes were identified in the category TD Visibility. Classifying TD items (supported by 13/16 respondents; 25 indicators) relates to the main feature of the bot: “r1: [It helps in] classification, categorization. . . that’s general for any kind of labels or tags”. In addition to classification, the bot helps to keep TD visible over time, since labeled TD items can be aggregated, helping developers filter the issues: “r5: in just one click [using GitHub’s features] you can have all the lists [of issues per label] otherwise you would have to go issue-by-issue to understand the nature of each [TD] item”. Finally, participants also reported that the bot could improve the visibility of TD, helping to pinpoint maintenance hotspots: “r10: [I can] see where there are more pending tasks or which modules require more attention”. TagDebt also helps practitioners to avoid the possible drawbacks of TD accumulation: “r8: You end up not considering [TD], and it becomes a snowball. You don’t understand the impact. . . I think having a tool to alert you or help you manage all this is really helpful.” The lack of visibility often leads to TD being forgotten, and TagDebt helps developers to stay aware of TD items that will be managed at a more convenient time: “r9: After the technical debt is identified, it may get lost. . . the team may just be focused on delivering, delivering, delivering. . . and after a couple of months, we won’t even remember it anymore.” TagDebt also helps in Increasing TD awareness (supported by 9/16 respondents; 14 indicators), since TD items become visible to developers (through labeled issues). Such awareness helps practitioners to make more informed decisions during the development process: “r6: [It could] help with making us more aware to maybe 31

Table 5: Themes related to usefulness, the respondents supporting each theme, and the total of indicators per theme Themes TD Visibility Classifying TD Items Increasing TD awareness Prioritizing TD items Developer Support Reducing repetitive or manual work Helping prevent human errors Standardizing issue reports Low feature overhead Limitations Classification Precision Descriptive labels and classification

Respondents r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14 r15 r16 Ind.

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓ ✓ ✓

✓ ✓

✓ ✓

✓

✓

✓ ✓ ✓

✓

✓

✓

✓ ✓

✓

✓ ✓ ✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓ ✓

✓

✓ ✓

✓

✓ ✓

✓ ✓

✓

✓

✓ ✓

✓

✓ ✓ ✓

✓ ✓

✓ ✓

✓

✓

✓ ✓

reconsider the issue or look into it and have a quick discussion about [whether] this can introduce technical debt.” Some participants also mentioned that the awareness promoted by TagDebt can help reduce the accumulation of TD over time: “r6: I would say it would not help in the shorter term, but maybe for the longer term because it helps you introduce less technical debt.” Furthermore, the awareness of TD items helps teams better plan maintenance actions earlier in the development process: “r6: [The bot would be useful] even before the design, helping developers avoid or reduce TD early on.” For supporting Prioritizing TD Items (supported by 9/16 respondents; 12 indicators), TagDebt helps group the TD items and helps practitioners to identify critical issues that need to be addressed first: “r2: once [the bot] understands what technical debt is, maybe we can identify what is critical, what is minor.” Finally, by easily analyzing the issues labeled by TagDebt, TD items can be assigned to the right stakeholder, who will decide how to proceed depending on their priorities: “r1: you can assign [issues] to different people, and based on that, you can prioritize and organize... Your priorities.” Regarding Developer Support, TagDebt can automate daily tasks and reduce the cognitive load of developers. Since TagDebt automatically labels issues, it helps in Reducing repetitive or manual work (supported by 12/16 respondents; 25 indicators), such as manually identifying TD (which is time-consuming): “r9: I believe it is a very useful tool thinking about this technical debt context, because it ends up being a very manual process and for that reason very flawed”. Once TagDebt reduces the time to carry out simpler tasks (that is, assign labels), developers can focus on tasks that add more value to companies: “r8: all this manual work often takes time and doesn’t require specific knowledge. . . a tool that can remove these manual steps 32

✓ ✓

25 14 12

✓

25

✓ ✓

7 7 4

✓

14 8

and leave decision-making to the developer would optimize the work a lot.” Therefore, TagDebt helps practitioners increase their productivity and reduce the time involved in TDM: “r12: If we could, for example, pass the labels and the description of what each label is, like which problems they relate to, I think it would save a lot of time. . . it would help in mapping these things. . . I see it solving an agility problem in the team’s daily routine”. TagDebt is also useful in Helping prevent human errors (supported by 6/16 respondents; 7 indicators), since it can be used as a double-check mechanism for manually-assigned labels: “r5: The bot could be used as a double check. . . the final answer will be the developer’s one.” The bot can trigger developers to reflect on their own labels and improve developers’ precision in identifying TD: “r7: If you classify the message with label A but the bot labels with label B, it will force you to think critically. . . oh, why did the bot classify with label B? Maybe I have to look again and avoid the error.” Participants, nonetheless, note that TagDebt would be more useful as a supporting tool, rather than replacing human judgment. Therefore, it can add more value in situations where developers may overlook tagging an issue: “r5: In some cases, the developer might not tag the issue at all because he is not paying attention, he forgets. . . in those cases, the bot could help. It is helpful.” An interviewee also described that the content of issues might be ambiguous, so TagDebt could perform a first check: “r10: It happens a lot [...] you have a bunch of open debt. . . sometimes the text is ambiguous. I think if there were a tool that could facilitate this or do it automatically, it would definitely make life easier for everyone.” Standardizing issue reports (supported by 5/16 respondents; 7 indicators) is another benefit that practitioners identified when using TagDebt: “r1: If you are a software house, you are a vendor, you need to have a standardized template, and you cannot permit your developers to open an issue with just text.” Practitioners imply that the bot can help teams organically improve issue descriptions, and this can help them to define standard templates for issue reports: “r10: I see it [the bot] as a template, like for opening PRs. In all companies I have worked for, there is always a PR template - it is how we agree to describe what is being done in that pull request. It’s a great way to standardize things, you know?” One of TagDebt’s characteristics that was also highlighted is the Low feature overhead (supported by 4/16 respondents; 4 indicators), since it avoids unnecessary complexity or excessive features: “r7: I think it is great in their specific use case. Labeling the issue...” Such specific functionality is a strength, since it avoids excessive information load that a huge set of metrics and diagnoses could cause: “r8: It’s not really for diagnostic suggestions. It’s something very specific. I think that is OK”. There were also several critical reactions to the TagDebt bot. Such criticism was mostly related to Classification Precision (supported by 10/16 respondents; 14 indicators) and Descriptive labels and classification. The first theme is that the bot’s precision is a concern, and it may reduce its usefulness in certain contexts: “r1: I would not rely on the auto label because otherwise, I would probably spend most of the time changing labels because I meant something.” This concern with precision reduces the trust in the bot, and is taken into account when using TagDebt: “r2: could be a good starting point, but it first needs to ’gain the team’s trust”’. We clustered such

33

themes into the category Limitations, since we deemed it key that practitioners and researchers also learn from the drawbacks present in our bot. More descriptive labels and classification (supported by 6/16 respondents; 8 indicators) mainly calls for more explainability about how the bot classifies an issue. First, the bot could provide a more descriptive label, explaining why the issue has a TD item: “r5: They just say something like ‘td’ or ‘non-TD’, but they don’t explain what [this means]”. The lack of such explanations may lead to the labels reducing the value added by the bot: “r6: If there were explanations, then that would give much more direction. Because now... It’s up to the developer to figure out where the technical debt is introduced or why it got that label.” Finally, understanding the reasoning behind the classification was mentioned as relevant: “r7: See why it made that classification? That would be great.” This shows that more informative tags could help clarify the bot’s decisions and improve its usefulness. Answer to RQ1 : Overall, practitioners perceived TagDebt as a useful tool to support TDM: most respondents (13/16) highlighted that it increases the visibility of TD, while a considerable number (9/16) reported that it helps increase awareness and supports prioritizing TD items. However, themes such as classification precision (mentioned by 10/16 respondents) and the lack of more descriptive labels and explanations (mentioned by 6/16 respondents) are pointed out as limitations that could hinder trust in the bot.

6.2 Ease of Use of TagDebt To answer RQ2 , we investigated how practitioners perceive the ease of use of the TagDebt bot. Overall, participants discussed several aspects related to learning, configuring, and operating the bot in their workflows. The topics related to ease of use were classified into three categories: (i) Installing and Configuring; (ii) Learning and Understanding; and (iii) Operating. Table 6 shows the themes within each category. TagDebt was considered Simple to install, configure, and operate (supported by 13/16 respondents; 27 indicators): “r5: It was quite simple. I had no kind of issue during the installation process.” Installing it was considered straightforward, which was expected due to the integration of the bot with GitHub: “r9: I managed to install it in the repository, create the file, and classify the issues in less than 10 minutes”. Similarly, practitioners considered that the Configuration file is clear (supported by 8/16 respondents; 10 indicators), which indicates that JSON configuration files are familiar to most developers, and may help to reduce the overhead for configuring the bot: “r1: I think the JSON file is fine”. However, a participant mentioned that the number of configuration options might be overwhelming: “r9: It’s confusing also in the configuration part of the JSON file. . . I think since there are so many configurations, it ends up being a bit confusing.” In this scenario, practitioners recognize that the default config.json provided is handy and helps to reduce the time and effort to configure the bot: “r9: You kind of just go with the default configuration file, you know? Just to start using it quickly and not waste too much time.” In a second category, Learning and Understanding, we identified that the Documentation is simple and complete (supported by 15/16 respondents; 34

Table 6: Themes related to ease of use, the respondents supporting each theme, and the total of indicators per theme Themes Installing and Configuring Simple to install, configure, and operate Configuration file is clear Learning and Understanding Documentation is simple and complete Documentation has some limitations Remembering how to use the bot

Respondents r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14 r15 r16 Ind.

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓ ✓

✓

✓

✓

Operating Commands as comments are not ideal

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

27

✓

✓

✓

10

✓

✓

✓

28 7

✓

✓

✓

✓

✓

28 indicators), indicating that practitioners found TagDebt’s documentation clear, straightforward, and well-structured. The quick-start guide and configuration guides were noted for making the setup easier: “r9: Yes, I found it [the quick-start guide] really easy to find my way around the sections. I think it is well structured.” There was an example of a configuration file, which also contributed positively to the ease of use of the documentation: “r9: I really liked that there’s a separate repository with the configuration and the classification the bot gave to each issue. . . it really helped to understand.” However, the clarity of the documentation appears to be tied to the technical experience of the user: “r1: You have to have a programming background... developers will understand very easily. . . no one else. Neither would a technical PM probably understand the whole thing.” This suggests that while the documentation is effective for technical staff, managers might consider it less clear. Some suggestions for improvement emerged, such as including more visual references in the description of the marketplace: “r2: Maybe a screenshot directly in the documentation [on GitHub Marketplace] could help, because we have to open another link [to access the quick-start guide]”. Although the documentation was considered clear, we identified some flaws reported by practitioners, as per the theme Documentation has some limitations (supported by 4/16 respondents; 7 indicators). A recurring issue was a lack of clarity about how the bot works and what users should expect during installation and usage: “r3: It wasn’t clear at all which one was supposed to do, because one hand mentions you can install from marketplace and on the other it mentions that if something is not working, you have to install it properly... It is a bit involved, to be honest.” There was also uncertainty about configuration details: “r3: Even though I installed it from the marketplace, I was really confused about configuration. I was expected to 35

14

4

find something to tell what labels to use. But the configuration did not tell me what the defaults are... so I assumed it had some defaults, but it didn’t contribute to the positive experience.” Others reported taking longer than expected to understand how the bot classifies the issues: “r10: I liked it a lot, but it took me a bit to understand how it would work. . . I thought it would do some automatic mapping or scanning in the code. . . then I understood it was based on GitHub issues.” Another theme that emerged from the data analysis was Remembering how to use the bot (supported by 13/16 respondents; 14 indicators). For this theme, we identified mixed feelings among the participants. Most of them found the bot commands relatively simple to remember, and they would not face problems in using the bot: “r1: all the parameters in common are quite easy to remember” and “r8: there are relatively few commands and there’s a help command. I think it’s very straightforward.” On the other hand, some participants highlighted that the bot has many configuration options, making it difficult to remember how to use it after a period of inactivity: “r7: If I don’t use it like every day, I will forget the command, I think” and “r9: I think I might have difficulty, not because it’s hard, but because we have a lot on our minds.” Nonetheless, the participants also mentioned that the help command is a very useful mechanism to help them remember how to use the bot: “r2: Once a week, I would have to use the help because I forgot how to auto-categorize or auto-label... but after using the help, it’s fine.” and “r10: I remember that there’s a help command that shows everything, so I’d manage.” In the category Operating, we found the theme Commands as comments are not ideal (supported by 4/16 respondents; 4 indicators). One participant mentioned that keeping commands as comments increases the size of issues and may hinder practitioners’ understanding: “r5: If I write a comment like label, I mean, it’s not interesting for someone else to know it. . . Why should I have a persistent comment that is just a comment?” Additionally, there was a suggestion that the bot’s interaction model (through comments) should be more explicitly documented: “r8: Maybe the documentation should state that the interaction happens through comments.” Another participant mentioned that it was not clear that the bot would not put a comment with the classification: “r10: One thing I didn’t understand well was when I put the command to add a label, it just said it classified or not, but I didn’t see the classification. I was like, Where is the classification?” Answer to RQ2 : Overall, practitioners found TagDebt easy to install, configure, and operate. Most respondents (13/16) indicated that the bot is simple to install, configure, and use, and half of the respondents (8/16) mentioned that the configuration file is clear and familiar. In contrast, some respondents indicated concerns about understanding the configuration details (4/16) and using comments as input for the bot (4/16), highlighting that there is still room for polishing and improving the usability of the TagDebt bot.

36

6.3 Contextual Factors for Using TagDebt To answer RQ3 , we investigated the contextual factors that influence practitioners’ intention to use the TagDebt bot. Contextual factors refer to aspects that go beyond the tool’s internal features (e.g., team characteristics or project environment) but still shape its adoption and perceived value. The topics related to contextual factors were grouped into five themes, which we summarize in Table 7.

Table 7: Contextual factors that might influence the adoption of TagDebt, the respondents supporting each factor, and the total of indicators per factor Themes Team Size Code Base Size Practitioners’ Roles Privacy and Security Policies Type of software

Respondents r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14 r15 r16 Ind. ✓ ✓ ✓ ✓ ✓ ✓ 12 ✓ ✓ ✓ ✓ ✓ ✓ 8 ✓ ✓ ✓ ✓ ✓ 7 ✓ ✓ 3 ✓ ✓ 3

First, participants suggested that Team Size (supported by 6/16 respondents; 12 indicators) impacts the decision to adopt TagDebt. Several interviewees indicated that the benefits of using the bot become more evident when the number of team members increases: “r3: [I think the bot was] designed for people who work in large teams”. This happens especially because there might be bottlenecks in labeling when multiple developers are opening issues in the same repository. However, the bot would also not be ideal if there are many contributors/team members. According to the practitioners, the bot would be mostly useful for teams between 10 and 25 developers/contributors: “r3: Oh well, I’m not sure these are the correct terms to talk about teams. I mean, a single person can manage only this many people, so are 10 people.” and “r1: But also with some limitations at some points, only the bot will not be enough. If you have a team of 25 people, commit.” On the other hand, in small teams, the bot seems to be irrelevant. One participant highlighted that they would not even consider using the bot if only one or two people are working in a given project: “r3: I found it quite useless for my project [...] no one else collaborates, so there is no need to communicate.” While this suggests a limitation in perceived applicability for smaller teams, it also reinforces the bot’s potential value in larger, more collaborative settings. Practitioners also mention that in smaller projects, the manual effort required to label issues is minimal, and therefore the Codebase Size (supported by 6/16 respondents; 8 indicators) is a factor to consider when adopting TagDebt: “r3: the amount of tasks they’d have to classify has to be quite high because it doesn’t take much effort to classify a task and this doesn’t have to be done so often.” However, in contexts with a high volume of issues, participants recognized the potential for errors and oversights, such as developers forgetting to assign labels, which is the context where TagDebt seems suitable: “r5: with the increase of numbers, the importance of the bot increases. With huge numbers, it makes sense to automate the classification”. 37

Practitioners’ Roles (supported by 5/16 respondents; 7 indicators) is a theme indicating that different types of stakeholders might not be interested in adopting the bot. For instance, one participant described how the adoption of the tool is typically a request that comes from developers, who are more interested in using the bot and are often responsible for introducing the new tools: “r1: The bot is very useful. This is how it works, and they will tell this to their PM team, leads product manager, whoever, OK. And the developers need to clarify the intention of the bot.” In addition, different roles within the team perceive the bot’s functionality differently. Some developers do not consider labeling essential: “r3: for some people, it doesn’t matter what it [the bot] says”. However, participants mentioned that managers could benefit from using the bot if they are responsible for managing issues and backlogs: “r10: Generally, it is more relevant for the agility person, who handles things like the Scrum board, the Kanban, etc., within the team. It’s usually this person who looks at what is being done and what is being addressed. What often happens in a more technical context is that, in a company that has a chapter or something more organized among developers of the same area, this comes into play.”. In the Privacy and Security Policies (supported by 2/16 respondents; 3 indicators) theme, we summarize that participants have concerns about using the bot in organizations where code privacy and data control are critical: “r1: most of the time you don’t want to give access to any third-party solution to your private repository”. This concern is mainly related to critical domain applications, such as finance and health insurance: “r1: for banking, insurance, and all the finance, they will not push any code to GitHub.com”. In this scenario, practitioners mentioned that organizations would prefer to have their own deployed instance of the bot, which is a possible solution: “r1: I imagine the organizations that I met don’t like GitHub but [prefer] Enterprise, GitLab, and all those kinds of solutions”. Finally, the Type of Software (supported by 2/16 respondents; 3 indicators) (i.e., proprietary or OSS) also may impact the decision to adopt TagDebt. The practitioners mentioned that the bot would benefit OSS projects more: “r7: Uh, I think the issue is more like an OSS project use case. Because there are going to be other people who will do it. Request or demand the issue.” Hence, while the bot can be helpful in both settings (as already highlighted), internal policies and workflows could impact the adoption of the bot in private organizations. This shows that developers working in OSS projects would be the primary audience for TagDebt. Practitioners indicated that the intention to use TagDebt depends on several contextual factors, such as the codebase size and team size: while small projects or teams find little value in automation, larger and more collaborative settings see greater benefits in reducing errors and managing workload. The roles of practitioners also influence the adoption, as developers often introduce the tool, but managers or roles responsible for backlogs and boards may gain more from its outputs. Concerns about privacy and security policies can limit adoption in domains like finance and health, where self-hosted instances are preferred over third-party services. Finally, the type of software matters, with open source projects perceived as a more natural fit due to their collaborative nature, while private organizations may face restrictions from internal policies.

38

Answer to RQ3 : Overall, practitioners indicated that the intention to use TagDebt depends on several contextual factors, such as Team size (6/16 respondents) and codebase size (6/16 respondents). Specifically, small projects or teams may see little value in automation, while larger and more collaborative settings benefit more from using the bot. Besides, Privacy and security policies were explicitly mentioned by 2/16 practitioners (3 indicators), indicating that in sensitive domains (e.g., finance, insurance), self-hosted instances of TagDebt might be preferred.

6.4 Improvements for TagDebt To answer RQ4 , we identified the improvements to the bot suggested by the respondents. Overall, the suggestions cover both technical and usability aspects, such as a more intuitive configuration process, integration with other existing tools, and better communication features. Table 8 summarizes the list of improvements we identified.

Table 8: List of improvements for TagDebt, the respondents supporting each theme, and the total of indicators per theme Themes Integration and Analysis Feedback for users Label management Configuration and Setup

Respondents r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11 r12 r13 r14 r15 r16 Ind. ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 28 ✓ ✓ ✓ ✓ ✓ ✓ ✓ 9 ✓ ✓ ✓ ✓ ✓ 9 ✓ ✓ ✓ ✓ ✓ 7

Participants suggested Improvements in integration and analysis (supported by 10/16 respondents; 28 indicators) and suggested that the bot could use multiple data sources, such as the full issue discussion, source code, design documents, meetings, and other technical artifacts, to enhance the explainability of issue classification. There was also a suggestion to integrate the bot with GitHub Copilot, which is also aligned with leveraging more data sources and would enable the bot to consider source code in the issue classification. Finally, the bot could further help issue monitoring if the issue complexity were also considered. Nonetheless, it was not possible to identify how to measure the complexity. Regarding the interaction with practitioners, Improvements in communication and feedback (supported by 7/16 respondents; 9 indicators) were also mentioned. Users expect the bot to provide more actionable and contextual feedback, especially during the development workflow. For instance, providing comments related to the issue directly in the code further helps practitioners to contextualize the TD items. Participants also mentioned the bot could suggest how to fix a problem, providing more information for practitioners’ decision-making. Finally, participants also mentioned that the bot could also be useful if used in pull requests, where discussions about TD also happen. These features would make the bot more useful in critical stages of the software life cycle. 39

Improvements in Label management (supported by 5/16 respondents; 9 indicators) were also suggested by practitioners. They suggested that the bot should reuse existing repository labels, ensuring standardization and better alignment with current workflows. The need to reconcile multiple labels applied manually or automatically was also discussed, with options for customization. Another important request was the ability to classify the issue by the module or part of the system it relates to, giving more contextual information to the assigned label. Finally, users reported that the bot could also help identify similar or duplicated issues, especially in projects with a long history of issues. This improvement facilitates the visualization of issues, helping in their management. Practitioners suggested that we carry out Improvements related to configuration and setup (supported by 5/16 respondents; 7 indicators), such as making the bot’s configuration process more practical and centralized, for instance, by having a global configuration file for multiple repositories and/or providing support for maintaining multiple configuration files, which will be used in multiple repositories. Furthermore, participants suggested that we provide an external page with a graphical interface for generating and customizing settings. This would reduce the effort required to understand the configuration options. Finally, it was suggested that commands issued as comments could be automatically deleted after execution to keep the discussion cleaner. Answer to RQ4 : Overall, Integration and analysis were the most frequently mentioned improvements (10/16 respondents). Specifically, practitioners called for richer integration and analysis capabilities, such as using multiple data sources (issue discussions, code, documents), considering issue complexity, and integrating with tools like GitHub Copilot. Regarding Configuration and setup, more centralized configuration (e.g., reusable or global configuration files) was required. Finally, 7/16 respondents suggested that the bot provide more contextual and actionable support, including comments linked to code, suggestions for fixes, and support in pull requests.

7 Discussion This study was motivated by two research problems: “(i) Is it possible to develop a specialized TDM tool that is easy to integrate within existing software development workflows?”; and “(ii) Considering that we can develop such a tool, would practitioners intend to adopt it?”. After developing and evaluating TagDebt, we learned that bots are a good alternative for seamlessly including TDM in existing workflows and increasing the perceived value of TDM. Our results also reveal that, beyond technical integration, factors such as trust, explainability, and ease of configuration play an equally important role in determining the actual usefulness of a TDM tool. Hence, while integrating it into workflows is possible, the tool design must account for sociotechnical dimensions, ensuring that automation complements developers’ reasoning processes. 40

Regarding the adoption of TDM tools, the context plays a crucial role, particularly in terms of team size, organizational alignment, and regulatory constraints. Larger teams seem to benefit significantly more from TDM tools like TagDebt, as they alleviate coordination and labeling bottlenecks, whereas smaller teams or individual developers may see limited value. Moreover, adoption is facilitated when tools demonstrate transparency in their operations and provide clear, simple documentation. In the following sections, we elaborate on the evaluation results to illustrate these findings in detail.

7.1 Interpretation of the Results of RQ1 According to practitioners, the usefulness of TagDebt has, to some extent, a sociotechnical aspect. As for most TDM tools, TagDebt can achieve broader goals, such as improving team coordination and improving communication among the team. In addition, TagDebt complements developers’ own reasoning processes, instead of replacing and fully automating TD identification. TagDebt’s ability to increase visibility and awareness of TD supports better decision-making and may lead the teams to better define priorities and reduce friction in communication. Another aspect related to its usefulness is trust in automation. Although participants acknowledged productivity improvements, transparency and explainability are mentioned as key to validating bot decisions. This information is aligned with existing literature (Biazotto et al., 2025a,b; Avgeriou et al., 2025), which shows the need to keep humans in the loop when managing TD. TagDebt’s evaluation shows that the value of TDM tooling is not only related to the tool’s functionality, but also to the interaction between developers and the tool. To maximize adoption, bots like TagDebt must offer more than correct classifications: they must foster trust, seamlessly integrate into existing practices and workflows, and contribute to shared understanding within teams. Overall, our findings shed light on how developers perceive workflow-integrated automated assistants in their daily work. Practitioners usually perceive TagDebt as a way to reduce manual effort (e.g., avoiding repetitive labeling) and increase visibility and awareness of TD, but they consistently framed the bot as a supporting tool rather than a fully autonomous tool. Concerns about classification precision and the demand for more descriptive labels and explanations indicate that developers rely on the bot as a starting point for analysis or “second opinion,” but they still expect to keep control over final decisions. This suggests that, for TD-related automation, transparent feedback (e.g., why a label was assigned, which parts of the issue triggered the classification) is key for tool adoption.

7.2 Interpretation of the Results of RQ2 The findings for RQ2 indicate that TagDebt is easy to use, especially due to its smooth integration with GitHub, and having a centralized configuration file for each repository. These attributes suggest that other TDM tools could focus on effectively aligning with common development workflows and developers’ preferences, such 41

as reusing familiar configuration practices (e.g., JSON/YAML). Therefore, TagDebt might have a low barrier to adoption. While the overall user experience seems to be good, a few issues were identified and therefore represent opportunities to refine the bot and its documentation. First, while the number of options in the configuration file increases the developer’s control over the bot (which is aligned with the results we found in (Biazotto et al., 2025b)), it may become slightly convoluted for first-time users. This issue may also be the reason some practitioners acknowledge that they would not remember the configuration options if they do not use the bot on a daily basis. Hence, when developing tools for TDM, refining the configuration files and specifying a minimum set of options are good practices to reduce the complexity in configuring the tools. Similarly, while the documentation was considered clear, some participants had issues understanding the purpose and classification mechanisms of the bot (e.g., it was not clear that the source code would not be analyzed). Then, new TDM tools should have simple and straightforward documentation to avoid such issues and improve their adoption. The reported ease of installation, configuration, and use (e.g., most respondents considered the documentation simple and the configuration file clear) suggests that practitioners are open to adopting lightweight, workflow-integrated automation for TD, especially when it leverages familiar tooling (GitHub) and requires minimal onboarding. At the same time, our data indicate that this readiness is, in part, due to TagDebt’s simplicity (i.e., it supports only issue labeling and notifications), with limited configuration overhead and a clear interaction model. Although more sophisticated AI-based TDM tools may offer richer analyses (e.g., cross-artifact reasoning over issues, code, and architecture), they will likely need to preserve this low-overhead integration while also providing more substantial support for understanding and controlling automated decisions. Hence, our results suggest that practitioners are ready for more AI-driven TDM support, as long as it remains transparent, lightweight, and well-aligned with existing workflows.

7.3 Interpretation of the Results of RQ3 In RQ3, we identified that the intention to adopt and use TagDebt is influenced by several contextual factors. One of the most cited factors is team size (which reinforces the socio-technical aspect, as mentioned in Section 7.1). Practitioners consistently stated that larger teams would benefit more from adopting TagDebt. When these teams start to work on repositories and open issues, it is common for labeling bottlenecks to occur. In addition, coordinating and standardizing labeling on those larger teams becomes challenging, and therefore, it would make sense to adopt the bot. In contrast, for smaller teams or individual developers, the bot seems to be unnecessary. This is because the number of issues and labels is too low, and developers would spend more time reviewing the bot’s suggestions than adding the labels themselves. Therefore, when developing tools for TDM, scoping the environment for tool adoption is crucial to extract more value from them. Similarly to the previous factor, the developer’s role in this context also impacts the adoption of TagDebt. In general, developers are the main audience for the bot, 42

although managers can benefit from using the bot if they are responsible for managing and prioritizing the issues. This factor reinforces the importance of organizational alignment in tool adoption: while developers may benefit immediately, the tool’s success also depends on management expectations and the perceived value in the long term. Privacy and infrastructure constraints emerged as another key factor. Relying on GitHub infrastructure may raise concerns for companies in sectors where regulatory or security constraints are stricter (e.g., finance and insurance). Hence, our decision to make the bot open source, enabling local deployment and support, may help work around privacy constraints. These findings indicate that the decision to adopt TagDebt, and possibly other TDM tools, is highly contextual. Therefore, when proposing tools for TDM, it is crucial to incorporate adaptability and configurability as core design principles to better support adoption across multiple environments. The contextual factors we identified (e.g., team size, codebase size, roles, privacy and security policies, and software type) highlight that adoption decisions for workflow-based TDM tools are inherently socio-technical. Many practitioners perceived TagDebt as particularly valuable in medium-to-large, collaborative teams and projects with a substantial number of issues, where manual labeling becomes errorprone and time-consuming. Conversely, in very small teams or small codebases, automation was often seen as unnecessary overhead. Additionally, strict privacy and security policies in certain domains (e.g., finance, health) constrain the use of thirdparty services, motivating self-hosted or on-premise deployments. Together, these findings emphasize that tool adoption cannot be evaluated in isolation from team structure, governance, and organizational constraints.

7.4 Interpretation of the Results of RQ4 Some improvements to TagDebt involve streamlining the configuration and setup process, which reinforces the idea that TDM tools must introduce minimal friction to be adopted. Alternatives to reduce friction include centralized configuration management and making the configuration process easier, for instance, by using a graphical interface. Hence, when developing tools for TDM, vendors should take into account that the tools’ operational complexity is a potential barrier for adoption. Beyond ease of use, practitioners clearly desire more context-aware and integrated analytical capabilities. The ability to draw insights from multiple sources (ranging from issue discussions to source code) and from design documentation was seen as essential for improving the precision and relevance of TD classification. This reflects that effective TDM requires understanding technical artifacts holistically rather than relying on isolated data points. In addition, users indicated that the bot could engage more proactively within development workflows, such as suggesting refactoring actions on pull requests. This preference implies that TDM tools should provide information that can be easily checked by practitioners, and lead to practical actions. 43

Labeling and organizational features also garnered attention, with calls for better reuse of existing labels and semantic enrichment through module-based classification. These improvements would foster alignment with existing team conventions and improve the interpretability of debt items, ultimately supporting prioritization and strategic planning. The themes that emerged from our study suggest several design lessons for other workflow-integrated tools. First, embedding functionality directly into existing artifacts (e.g., GitHub issues) and reusing project concepts (e.g., repository labels) can reduce the cognitive and process overhead for tool adoption. Second, providing sensible defaults (e.g., a ready-to-use configuration file) while allowing gradual customization helps teams get started quickly and invest further effort only if the tool proves valuable. Third, interaction mechanisms should be designed to balance simplicity with cleanliness of the workflow. For instance, participants appreciated comment-based commands but also requested automatic removal of “command” comments to avoid cluttering issue discussions. Finally, our results underline the importance of explainability and richer feedback (e.g., justifications for labels, pointers to relevant parts of the issue, integration with code review or pull requests). These aspects can inform the design of future workflow-based TDM tools and, more broadly, other bots that aim to support developers without disrupting established practices. Overall, the suggested improvements reflect a desire for TagDebt to evolve from a straightforward labeling tool into a more integrated, intelligent assistant that balances usability with deeper contextual understanding. This evolution could make the bot more relevant across diverse team sizes, project complexities, and development contexts, ultimately fostering more effective TDM practices.

7.5 Implications for Practitioners The findings from our study on the TagDebt bot offer several valuable insights for practitioners and tool vendors. These findings are associated mainly with the adoption or improvement of automated tools for TDM. Below, we summarize each of those implications: 1. Enhance Visibility and Prioritization of TD Items: Practitioners can significantly benefit from automating the classification of SATD items by using TagDebt. The bot helps teams maintain the visibility of debt. This is particularly critical in larger teams or projects where manual labeling becomes impractical. Incorporating such automation can reduce the cognitive load on developers and minimize errors caused by manual labeling or oversight. 2. Team and Environment Characteristics when adopting TDM tools: We help practitioners understand how the environment and the team impact the perceived usefulness of automated TDM tools, which is highly context-dependent. For instance, larger teams benefit from tools that help coordinate actions. In contrast, smaller teams may find limited value, suggesting that adoption decisions should be carefully considered. Furthermore, practitioners should be aware of privacy and security policies, especially in sensitive or regulated environments, ensuring that tool integrations comply with organizational constraints. 44

3. Balancing Ease of Use with Configuration Flexibility: While TagDebt is considered easy to install and operate, practitioners recognize that it has an initial learning curve related to configuration and command usage. Hence, vendors should be aware that a higher number of configuration options may lead to complexity in setting up the tools. Therefore, effective onboarding, supported by clear documentation and ready-to-go setups, is essential to facilitate adoption and continued use of TDM tools. 4. Integrating Tools Seamlessly into Developer Workflows: To maximize tools’ value, vendors should provide functionality that works directly within the development life cycle, such as comments in pull requests or code suggestions. This tight integration improves the relevance of TD information and may increase the awareness of TD. 5. Continuous Improvement through User Feedback: Our study highlights the importance of engaging with users to identify improvements to existing tools. Our study also advocates for more explainability and transparency in tools to improve trust in them. Vendors should detail how data is analyzed to help practitioners understand how to use the information. 6. Low-Overhead and Explainable Tools: We deem that such results suggest similar principles apply to workflow-integrated tools in general. For developers to perceive a tool as genuinely useful, it should require minimal configuration and setup effort, integrating into existing workflows with little friction rather than demanding extensive tuning or process changes. At the same time, tools that classify, recommend, or prioritize items (e.g., refactoring actions) need to provide explainable outputs, making clear why a decision was made, which signals were used, and with what level of confidence. This would help practitioners quickly validate, contest, or refine those decisions. 7. Trade-offs in tool usage: These findings can also be transferred to tool usage more broadly. Our results indicate that adopting tools usually leads to a tradeoff between configuration effort and learning curve on one side, and expected productivity gains on the other. Teams are more willing to invest in setting up and learning a tool when system size or process complexity is high enough that manual work becomes a clear bottleneck. In contrast, in small teams or simpler projects, the perceived overhead of configuring, integrating, and mastering a new tool can hinder its benefits, leading practitioners to prefer manual solutions. Therefore, when designing and evaluating development tools in general, it is essential to consider where this balance lies in different contexts and to minimize upfront overhead so that productivity benefits become visible early in the adoption process. In summary, practitioners who aim to implement or optimize TDM tools such as TagDebt should prioritize automation that enhances visibility and prioritization, carefully consider their team and project context, emphasize usability and seamless workflow integration, and actively engage in continuous improvement to maximize the benefits of such technologies.

45

7.6 Implications for Researchers The findings from our investigation into the use and perception of TagDebt provide several directions and considerations for researchers working on TDM tools and SE automation. 1. Exploring Contextual Factors in Tool Adoption: Our results highlight the critical role of contextual factors (such as team size, developer roles, project type, and organizational privacy constraints) in shaping the adoption and perceived usefulness of TDM tools. Researchers should investigate these contextual dimensions more deeply, potentially developing frameworks or models that predict when and how such tools are most beneficial. This understanding can guide the design of adaptable tools that fit diverse development environments. 2. Advancing Usability Research for Developer Tools: While ease of use was generally acknowledged, challenges remain around configuration complexity, remembering commands, and interaction modalities (e.g., commands via comments). Future research should focus on improving the usability of automated tools through better interfaces, contextual help, and more intuitive interaction models. Studies involving longitudinal user experience and cognitive load can provide valuable insights into sustained tool adoption. 3. Enhancing Automated Analysis and Feedback Mechanisms: Participants expressed a desire for richer analysis that incorporates multiple data sources beyond issue titles, such as code, documentation, and discussion threads, to improve classification accuracy and contextual relevance. Researchers can explore novel techniques in natural language processing, machine learning, and software repository mining to develop more sophisticated models to identify and prioritize TD. In addition, embedding actionable feedback directly into developers’ workflows remains an open challenge that deserves further exploration. 4. Facilitating Continuous Improvement and User-Centered Design: The diverse suggestions for feature enhancements emphasize the importance of iterative, usercentered design processes in the development of TDM tools. Researchers should work closely with practitioners to identify evolving needs and assess the impact of new features. Mixed-method studies that combine quantitative usage metrics with qualitative feedback can help bridge the gap between tool capabilities and practical utility. Overall, this study underscores several promising avenues for advancing research on automated TDM tools. By addressing contextual influences, improving usability, expanding analytical depth, and fostering collaboration with practitioners, researchers can contribute to more effective and widely adopted solutions in SE.

8 Threats to Validity As in any empirical study, there are some threats to the validity of our results and contributions. We deemed that each phase of our research method (DSR) may pose specific threats. To ensure that we cover all potential threats and provide a better understanding of how those threats spread throughout the research method, we organize this section considering the DSR phases (i.e., Understand the social context, 46

Understand the knowledge context, Design, and Investigation) and also discuss the associated mitigation actions. In the Understand the social context and Understand the knowledge context phases, our objective was to understand the problems and demands with respect to TDM and the potential solutions to such problems. Both phases suffer from the same threat: since we did not apply a systematic approach for reviewing the literature (e.g., SMS or systematic literature review), it is possible that some problems and/or solutions related to TDM were not identified. However, we mitigated those threats by focusing on recent literature, which we considered to contain the state of the art on TDM. For instance, the Manifesto on Reframing Technical Debt, a core reference in our study, is the result of a seminar and focus groups in which the most prominent researchers in TD were involved, and therefore, it summarizes the current understanding of TDM needs. In addition, our two previous studies (Biazotto et al., 2025a,b) highlight the needs of practitioners from two different sources (i.e., StackExchange posts and a survey with practitioners), providing strong evidence on the need for more specialized TDM tools. Our SMS (Biazotto et al. (2023)) summarizes information on more than 120 TDM tools and provides a broad understanding of the tooling for TDM. Focusing on recent evidence, we are confident that we have a good overview of current TDM problems and solutions. Regarding the Design phase, we addressed a solution to TDM problems. As mentioned in Section 2, a core challenge faced by developers is the friction that TDM tools usually impose on existing development workflows. Our suggestion of proposing a bot, rather than other types of tools, is also a potential threat. We mitigate this threat by basing our decision on previous work that showed that bots are widely accepted by developers and, therefore, bots could be a good alternative to support TDM (Phaithoon et al., 2021; Biazotto et al., 2023). Another threat within the Design phase is the adoption of an NLP solution. Although current research on NLP solutions for TD detection showed their potential, practitioners still do not fully trust such solutions, which could affect their perceptions about the bot. However, we deemed that any type of tool would face similar challenges, as presented in (Biazotto et al., 2025a). To mitigate this threat, we focused on a very specific task (i.e., assigning labels), which can be easily reviewed by practitioners. In addition, we simplified model replacement, which might help practitioners keep up-to-date with advances in AI for TDM. Another threat to validity is that the requirements implemented in TagDebt might not be ideal or complete for all contexts in which practitioners manage TD. We reused a previously published set of practitioner-derived requirements from our prior work (Biazotto et al., 2025b). While this mitigates the risk of ad-hoc requirement definition by grounding the tool design on empirical evidence, it also implies that our selection of a subset of requirements may not capture all expectations that different teams may have for TDM tooling. We mitigate this threat in two ways: first, we selected requirements that together compose a coherent end-to-end use case (i.e., issue labeling complemented by configurable notifications), rather than implementing isolated features; second, we evaluated the resulting artifact with practitioners to elicit improvement points and additional needs. Nonetheless, future work should extend TagDebt by implementing additional requirements from the original list and by

47

refining the requirements through further empirical studies in diverse organizational settings. There are two other threats related to the Design phase: the selection of the platform to run the bot and the selection of the NLP-based solutions. Regarding the selection of the platform, other platforms such as Jira19 could have been chosen, and this selection could affect the perception of the bot by developers. However, GitHub is one of the largest software repositories, and most of the works related to TD identification are carried out using GitHub (Phaithoon et al., 2021; Yetistiren et al., 2022; Mohayeji et al., 2022b). Furthermore, GitHub has several bots and tools that developers use extensively. Therefore, we mitigate this threat by focusing on a well-known platform. As for the selection of the detection function, Li et al.’s model (Li et al., 2022) is a model specific for identifying SATD in issues. Although its accuracy is not too high (around 70%), we deemed it fair for an initial version of our bot. To mitigate this threat, we developed a plugin-based system that, as already stated before, enables practitioners to easily replace the model, thereby increasing the potential for practitioners to adopt the bot. The plugin’s flexibility is made explicit, since we also provide an LLM-based alternative, which implements GPT-5-mini. This LLM was selected because it is more affordable than the higher-hyperparameter counterparts, while providing good performance for TD classification, e.g., as recently reported by Maarleveld et al. (2026). There is also a threat related to the focus on issues rather than other software artifacts (e.g., source code or pull requests). This decision might also impact the usefulness of the bot. To mitigate this threat, we based our bot on a scenario presented in our previous study (Biazotto et al., 2025b), which was reviewed by practitioners who considered a bot in issue tracking systems as useful for TDM. The Investigation phase is key for DSR, as it provides empirical evidence on the developed artifact. In our context, we understand the perceptions of practitioners about the bot. To carry out our evaluation, we used a TAM-based method, and data analysis was carried out using thematic analysis, which poses its own set of threats. First, the study findings depend to some extent on how participants were selected and how they interpreted the questions in the interview. To mitigate this, we relied on a semi-structured interview protocol and encouraged open-ended discussions, allowing participants to express their critical views freely. In addition, each participant was interviewed only once. There is a threat related to the limited depth of insight from single-session interviews. Although no follow-up interviews were conducted, we addressed this limitation by thoroughly transcribing and coding the interviews to capture as much nuance as possible. Besides, participants were expected to use the bot for a few days before the interview. This allowed them to review the bot at their own pace and interact with it as much as they wanted. Still regarding the Investigation phase, there is a threat related to the way we adapted TAM constructs into interview questions. Classical TAM instruments typically use Likert-scale items framed in a positive way (e.g., statements about whether a system is helpful or easy to use). In our qualitative setup, we kept this positive framing and adapted those items into closed questions (e.g., “Do you think the bot 19

https://www.atlassian.com/software/jira

48

would help to identify and monitor TD items more quickly and easily?” or “Was it easy to learn how to operate the bot?”). While these questions were always followed by open-ended and neutral probes (e.g., “In which situations would it not help?”), the initial wording may still have encouraged participants to focus first on benefits. We mitigated this risk by systematically asking for drawbacks, conditions under which the bot would not be appropriate, and suggestions for improvement, and by incorporating both positive and negative indicators into our thematic analysis (e.g., concerns about precision, team size, configuration complexity, and documentation limitations). We deem that such mitigation actions were sufficient to reduce, at least to some extent, the potential bias caused by the set of questions. We decided not to explicitly characterize participants’ working environments, and our results may be more difficult to interpret with respect to specific organizational or project conditions, and this may affect the transferability of the findings. In particular, perceptions of usefulness and ease of use can be shaped by factors such as organizational maturity, existing TD management practices, socio-technical constraints, and the project ecosystem; thus, the lack of structured contextual profiling may limit external validity. To mitigate this limitation, we asked participants to justify their answers and analyzed the contextual factors they reported as influencing adoption (e.g., team size, project characteristics, and codebase complexity). Future work should incorporate a more controlled in-project deployment and collect richer organizational and project-level descriptors to better explain acceptance across contexts. To code the data in our study, we adopted an integrated approach (Cruzes and Dyba, 2011). Initially, two authors independently coded a randomly selected sample of 16 responses, generating as many codes as possible based on the data. They then discussed the generated codes until they reached a consensus. We followed a similar process for defining the themes and building the model. The first author proposed themes based on indicators from the data, which were subsequently discussed and reviewed by the other three authors. This approach allowed us to have more flexibility in code assignment and also to deepen our understanding of the data while discussing the codes and topics. In addition, we were concerned with defining themes that captured all aspects of the data. However, because we did not have categories to compare in the first round of coding (involving two authors), and because only one author coded the full interview set, we are not able to calculate an inter-rater agreement metric (e.g., Cohen’s Kappa). Nonetheless, since all four authors thoroughly discussed the codes and themes, which evolved organically throughout the data analysis, we are confident that the lack of inter-rater agreement was sufficiently mitigated. Another threat to the generalizability of our findings is that we relied on convenience sampling, inviting participants from our professional network. In this study, we relied on semi-structured interviews to enable in-depth discussions about TagDebt and its interactions with practitioners; however, this research method limited the number of participants we could recruit, which may reduce the diversity of perspectives and affect generalizability. To mitigate this threat, we followed the sampling and recruitment strategy described in Section 3.3, recruiting participants with different roles (e.g., software engineers, architects, Q&A analysts, and team leaders), with varying

49

levels of experience (from 2 to 15 years), and with varied backgrounds and experience in TDM, ensuring that all interviewees met our inclusion criteria (i.e., experience with software development in industry). These decisions helped us capture a broad range of insights, even though the sample size and sampling strategy still pose limitations. Nonetheless, we acknowledge that broader longitudinal studies or large-scale replications are needed to further strengthen external validity. Furthermore, the bot was developed for GitHub Issues, which may limit the applicability of the findings to teams using other platforms (e.g., Jira, GitLab). Regarding our methods for data analysis, although we used coding, which is a technique present in Grounded Theory (Strauss and Corbin, 1990) to analyze the practitioners’ answers and define the themes (as presented in Section 5), not all steps of Grounded Theory were followed. Specifically, data collection and analysis steps were not concurrent (i.e., we collected all the data before analyzing it), which prevented us from claiming the theoretical saturation of our themes. Hence, replications of our study can help capture potential themes not identified in the current version of the study and contribute to advancing the understanding of TagDebt toward theoretical saturation. To support such replications and reproductions, we created a replication package20 with all the necessary data and scripts to run the analyses. Finally, it is also relevant to discuss Reflexivity. The authors have extensive research experience in TDM and were directly involved in the conception and development of TagDebt. While this expertise is valuable for framing the study and interpreting practitioners’ feedback, it might also introduce a risk of confirmation bias (e.g., focusing on positive experiences or overlooking critical remarks). To mitigate this threat, we adopted several strategies. First, the data collection followed a semi-structured protocol, with open-ended follow-up questions that explicitly invited participants to report both benefits and drawbacks. Second, we used a bottom-up thematic analysis and grounded our codes and themes in verbatim quotations, preserving practitioners’ wording as much as possible. Third, all authors participated in the coding and theme discussions, reducing bias. While some residual influence of our positions cannot be fully excluded, these procedures were designed to make our interpretive process more transparent and to limit the impact of our prior expectations on the reported findings.

9 Conclusion and Future Work A core lesson we learned while developing and evaluating TagDebt is that TDM tools must be easy to configure, easily integrated with existing workflows, and keep the human in the loop. According to our evaluation, TagDebt was shown to help improve the visibility of TD, as it increases the team’s awareness about the TD items, leading to more informed decisions. The automation provided by TagDebt reduces repetitive manual tasks, enabling developers to focus on tasks that add more value to companies. In addition, TDM tools must focus on helping developers avoid errors and standardize issue reporting. 20

https://doi.org/10.5281/zenodo.16934566 or https://github.com/biazottoj/rp-tag-debt-bot

50

Contextual factors play an important role in the intention of practitioners to use TagDebt, as well as other TDM tools. Larger teams and more complex projects benefit more from automation, whereas smaller teams often perceive less value. In addition, the roles of the practitioners, the type of project, and organizational privacy policies influence adoption decisions. These insights emphasize the importance of tailoring TDM tools to diverse development environments. The contributions of this study open up several research directions for future work. First, expanding the bot’s analytical capabilities by integrating additional project artifacts (e.g., code, documentation) could improve classification precision and contextual relevance. Second, designing more intuitive interaction models, beyond comment-based commands, may enhance usability and encourage sustained use. Third, investigating the scalability and impact of such automation in large and complex teams can provide practical guidelines for organizational adoption. As future work, we also plan to conduct a controlled comparative evaluation of TagDebt against alternative approaches for TD labeling and surfacing in GitHub, such as tag-based bots that detect explicit annotations (e.g., #TODO/#FIXME) and simpler labeling mechanisms supported by repository conventions. This study will require a curated set of projects and issues, controlled tasks, and consistent ground truth so that we can assess differences in user preference, perceived effort, workflow disruption, and correctness of produced labels. Such a comparative evaluation will complement our current practitioner-centered investigation by quantifying the benefits and trade-offs of TagDebt relative to existing baselines under comparable conditions. In future work, we also plan to strengthen the empirical grounding of TagDebt by triangulating its evaluation with complementary tools and approaches that support other stages of TDM. For instance, we intend to evaluate TagDebt alongside static-analysis-based TD identification tools, or prioritization and repayment support approaches. Conducting such comparisons requires a different study design, including multiple artifacts, controlled tasks, and consistent project contexts to ensure fairness and interpretability, and therefore, it was outside the scope of our current exploratory, human-centric evaluation. By framing TagDebt as a step toward workflow-integrated TDM support, we aim to use these future comparative studies not only to quantify trade-offs across granularity levels (issue-level versus code-level) but also to guide concrete improvements to TagDebt, such as richer context-aware explanations and more actionable feedback. Another potential research direction is a second evaluation round using UTAUT constructs with a more controlled and possibly longitudinal design. Such an evaluation could explicitly address constructs such as social influence and facilitating conditions when the goal shifts from exploratory feasibility to organizational-scale adoption. In addition, in such longitudinal studies, it is also possible to assess the real-world impact of automated TDM tools on software quality, team productivity, and decision-making to validate and refine their design. Such a study could employ TagDebt in one or more projects for an extended period, observing its usage and measuring whether and how it influences practitioners’ decision-making. Besides, we can check whether TagDebt affects other TDM activities, such as prioritization and repayment. We can also investigate if the usage of TagDebt can reduce the accumulation of TD over

51

time. Finally, the development of standardized taxonomies and customizable labeling frameworks could improve communication between heterogeneous teams and projects.

10 Declarations 10.1 Funding This study was financed in part by the Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES) - Finance Code 001, São Paulo Research Foundation (FAPESP) 2023/0488-5, and National Council for Scientific and Technological Development (CNPq) 313245/2021-5.

10.2 Ethical approval Our study follows the core principles outlined in the ACM SIGSOFT empirical standard “Ethics (Studies with Human Participants)” (ACM SIGSOFT, 2024a). No risks of harm to participants were foreseen. The activity for evaluating our proposed tool required an estimated commitment of 30 minutes, comparable in scope and duration to programming assessments commonly applied in recruitment processes. References: ACM SIGSOFT (2024a) Ethics (Studies with Human Participants). https: //github.com/acmsigsoft/EmpiricalStandards/blob/master/docs/supplements/ EthicsHumanParticipants.md (Commit d131d720)

10.3 Informed consent The participants were informed about the research project and received all necessary details to ensure their understanding before giving their consent. They were informed that participation is entirely voluntary and that they may withdraw from the study at any time without providing a reason, as long as this occurs before the first scientific output is submitted for publication. It was explained that participation involves participating in an interview, during which data, such as role and experience, will be collected. Participants were informed about how data would be processed and that recordings would be deleted as soon as the transcripts were generated. In addition, participants were asked for specific consents: • Agreement for the anonymized transcript of the interview to be deposited in an open repository (such as Zenodo) so it can be reused for future research and learning purposes. • Agreement to be contacted for future studies.

10.4 Author Contributions All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by João Paulo Biazotto, and reviewed by the other three authors. The first draft of the manuscript was written by João 52

Paulo Biazotto and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

10.5 Data Availability Statement The generated datasets and scripts for data analysis are available in a Zenodo repository at https://doi.org/10.5281/zenodo.16934566. Alternatively, the same replication package is also available on Github repository: https://github.com/biazottoj/rp-tagdebt-bot

10.6 Conflict of Interest The authors declared that they have no conflict of interest.

10.7 Clinical Trial Number Clinical trial number: not applicable.

References Alves, N.S.R., Mendes, T.S., Mendonça, M.G., Spı́nola, R.O., Shull, F., Seaman, C.: Identification and management of technical debt: A systematic mapping study. Information and Software Technology, 100–121 (2016) https://doi.org/10.1016/j. infsof.2015.10.008 Avgeriou, P., Ozkaya, I., Chatzigeorgiou, A., Ciolkowski, M., Ernst, N.A., Koontz, R.J., Poort, E., Shull, F.: Technical debt management: The road ahead for successful software delivery. In: IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), pp. 15–30 (2023). https: //doi.org/10.1109/ICSE-FoSE59343.2023.00007 Avgeriou, P., Ozkaya, I., Koziolek, H., Codabux, Z., Ernst, N.: Manifesto from Dagstuhl Perspectives Workshop 24452 – Reframing Technical Debt. arXiv (2025). https://doi.org/10.48550/ARXIV.2505.13009 Avgeriou, P., Taibi, D., Ampatzoglou, A., Fontana, F.A., Besker, T., Chatzigeorgiou, A., Lenarduzzi, V., Martini, A., Moschou, A., Pigazzini, I., Saarimaki, N., Sas, D.D., Toledo, S.S., Tsintzira, A.A.: An overview and comparison of technical debt measurement tools. IEEE Software 38(3), 61–71 (2021) https://doi.org/10.1109/ ms.2020.3024958 Biazotto, J.a.P., Feitosa, D., Avgeriou, P., Nakagawa, E.Y.: Technical debt management automation: State of the art and future perspectives. Information and Software Technology, 107375 (2023) https://doi.org/10.1016/j.infsof.2023.107375 Biazotto, J.a., Feitosa, D., Avgeriou, P., Nakagawa, E.: Automating technical debt management: Insights from practitioner discussions in stack exchange. In: 2025 53

IEEE/ACM International Conference on Technical Debt (TechDebt) (2025). https: //doi.org/10.48550/arXiv.2502.03153 Biazotto, J.a.P., Feitosa, D., Avgeriou, P., Nakagawa, E.Y.: Understanding practitioners’ reasoning and requirements for efficient tool support in technical debt management. Empirical Software Engineering 30(5) (2025) https://doi.org/10. 1007/s10664-025-10691-5 Beschastnikh, I., Lungu, M.F., Zhuang, Y.: Accelerating software engineering research adoption with analysis bots. In: 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track (ICSENIER), pp. 35–38. IEEE, ??? (2017). https://doi.org/10.1109/icse-nier.2017.17 Besker, T., Martini, A., Bosch, J.: Technical debt cripples software developer productivity: A longitudinal study on developers’ daily software development work. In: International Conference on Technical Debt (TechDebt), pp. 105–114 (2018). https://doi.org/10.1145/3194164.3194178 Besker, T., Martini, A., Bosch, J.: Software developer productivity loss due to technical debt—a replication and extension study examining developers’ development work. Journal of Systems and Software 156, 41–61 (2019) https://doi.org/10.1016/ j.jss.2019.06.004 Bavota, G., Russo, B.: A large-scale empirical study on self-admitted technical debt. In: Proceedings of the 13th International Conference on Mining Software Repositories, pp. 315–326. ACM, Austin, TX, USA (2016). https://doi.org/10.1145/2901739. 2901742 Baltes, S., Ralph, P.: Sampling in Software Engineering Research: A Critical Review and Guidelines. arXiv (2020). https://doi.org/10.48550/ARXIV.2002.07764 . https: //arxiv.org/abs/2002.07764 Babar, M.A., Winkler, D., Biffl, S.: Evaluating the usefulness and ease of use of a groupware tool for the software architecture evaluation process. In: First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), pp. 430–439. IEEE, ??? (2007). https://doi.org/10.1109/esem.2007.48 . http: //dx.doi.org/10.1109/esem.2007.48 Cruzes, D.S., Dyba, T.: Recommended steps for thematic synthesis in software engineering. In: International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 275–284 (2011). https://doi.org/10.1109/ESEM.2011. 36 Cunningham, W.: The wycash portfolio management system. In: Addendum to the Proceedings on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA), pp. 29–30 (1992). https://doi.org/10.1145/157709.157715

54

DALE, R.: The return of the chatbots. Natural Language Engineering 22(5), 811–817 (2016) https://doi.org/10.1017/s1351324916000243 Davis, F.D., Bagozzi, R.P., Warshaw, P.R.: User acceptance of computer technology: A comparison of two theoretical models. Management Science 35(8), 982–1003 (1989) https://doi.org/10.1287/mnsc.35.8.982 Santos, P.S.M., Varella, A., Dantas, C.R., Borges, D.B.: Visualizing and managing technical debt in agile development: An experience report. In: Proceedings of the 14th International Conference on Agile Software Development and Extreme Programming (XP’13), Viena, Austria, pp. 121–134 (2013). https://doi.org/10.1007/ 978-3-642-38314-4 9 Elazhary, O., Storey, M.-A., Ernst, N.A., Paradis, E.: Adept: A socio-technical theory of continuous integration. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), pp. 26–30. IEEE, ??? (2021). https://doi.org/10.1109/icse-nier52604.2021.00014 . http://dx. doi.org/10.1109/ICSE-NIER52604.2021.00014 Ferreira, F.H.C., Nakagawa, E.Y., Bertolino, A., Lonetti, F., Neves, V.d.O., Santos, R.P.: A framework for the design of fault-tolerant systems-of-systems. Journal of Systems and Software 211, 112010 (2024) https://doi.org/10.1016/j.jss.2024. 112010 Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading, MA (1994) Guo, Y., Seaman, C.: A portfolio approach to technical debt management. In: Proceedings of the 2nd Workshop on Managing Technical Debt (MTD ’11), pp. 31–34. Association for Computing Machinery, Waikiki, HI, USA (2011). https://doi.org/ 10.1145/1985362.1985370 Harrand, N., Durieux, T., Broman, D., Baudry, B.: The behavioral diversity of java json libraries. In: 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), pp. 412–422. IEEE, ??? (2021). https://doi.org/10.1109/ issre52982.2021.00050 Jeronimo Junior, H., Travassos, G.H.: Consolidating a common perspective on technical debt and its management through a tertiary study. Information and Software Technology 149, 106964 (2022) https://doi.org/10.1016/j.infsof.2022.106964 Kemell, K.-K., Evensen, A., Wang, X., Risku, J., Nguyen-Duc, A., Abrahamsson, P.: A tool-based approach for essentializing software engineering practices. In: 2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 51–55. IEEE, ??? (2019). https://doi.org/10.1109/seaa.2019.00016 . http://dx.doi.org/10.1109/seaa.2019.00016

55

Kashiwa, Y., Nishikawa, R., Kamei, Y., Kondo, M., Shihab, E., Sato, R., Ubayashi, N.: An empirical study on self-admitted technical debt in modern code review. Information and Software Technology 146, 106855 (2022) https://doi.org/10.1016/ j.infsof.2022.106855 Li, Z., Avgeriou, P., Liang, P.: A systematic mapping study on technical debt and its management. Journal of Systems and Software 101, 193–220 (2015) https://doi. org/10.1016/j.jss.2014.12.027 Lee, M., Frank, L., Beute, F., Kort, Y., Ijsselsteijn, W.: Bots mind the social-technical gap (2017) https://doi.org/10.18420/ECSCW2017-14 Liu, Z., Huang, Q., Xia, X., Shihab, E., Lo, D., Li, S.: SATD detector. In: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings. ACM, Gothenburg, Sweden (2018) Li, Y., Soliman, M., Avgeriou, P.: Identifying self-admitted technical debt in issue tracking systems using machine learning. Empirical Software Engineering 27(6) (2022) https://doi.org/10.1007/s10664-022-10128-3 Li, Y., Soliman, M., Avgeriou, P., Van Ittersum, M.: Debtviz: A tool for identifying, measuring, visualizing, and monitoring self-admitted technical debt. In: 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 558–562 (2023). https://doi.org/10.1109/ICSME58846.2023.00072 Linåker, J., Sulaman, S., Host, M., Mello, R.: Guidelines for Conducting Surveys in Software Engineering. Lund University, Lund, Sweden (2015) Lebeuf, C., Storey, M.-A., Zagalsky, A.: Software bots. IEEE Software 35(1), 18–23 (2018) https://doi.org/10.1109/ms.2017.4541027 Lin, B., Zagalsky, A., Storey, M.-A., Serebrenik, A.: Why developers are slacking off: Understanding how software teams use slack. In: Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion. CSCW ’16, pp. 333–336. ACM, ??? (2016). https://doi.org/10.1145/ 2818052.2869117 Maarleveld, J., Destefanis, G., Feitosa, D.: Evolving kubernetes: A technical debt perspective. In: Proceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26). ACM, Rio de Janeiro, Brazil (2026). https://doi.org/10. 1145/3793302.3793335 Mohayeji, H., Ebert, F., Arts, E., Constantinou, E., Serebrenik, A.: On the adoption of a todo bot on github: A preliminary study. In: 2022 IEEE/ACM 4th International Workshop on Bots in Software Engineering (BotSE), pp. 23–27 (2022). https://doi. org/10.1145/3528228.3528408

56

Mohayeji, H., Ebert, F., Arts, E., Constantinou, E., Serebrenik, A.: On the adoption of a todo bot on github: A preliminary study. In: 2022 IEEE/ACM 4th International Workshop on Bots in Software Engineering (BotSE), pp. 23–27 (2022). https://doi. org/10.1145/3528228.3528408 McGregor, J.D., Monteith, J.Y., Zhang, J.: Technical debt aggregation in ecosystems. In: Proceedings of the 3rd International Workshop on Managing Technical Debt (MTD’12), Zurich, Switzerland, pp. 27–30 (2012). https://doi.org/10.1109/MTD. 2012.6225996 Maldonado, E.d.S., Shihab, E.: Detecting and quantifying different types of selfadmitted technical debt. In: 2015 IEEE 7th International Workshop on Managing Technical Debt (MTD), pp. 9–15 (2015). https://doi.org/10.1109/MTD.2015. 7332619 Manteuffel, C., Tofan, D., Avgeriou, P., Koziolek, H., Goldschmidt, T.: Decision architect – a decision documentation tool for industry. Journal of Systems and Software 112, 181–198 (2016) https://doi.org/10.1016/j.jss.2015.10.034 Ochoa, L., Degueule, T., Falleri, J.-R.: Breakbot: analyzing the impact of breaking changes to assist library evolution. In: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results. ICSE ’22. ACM, ??? (2022). https://doi.org/10.1145/3510455.3512783 Pérez-Soler, S., Guerra, E., Lara, J., Jurado, F.: The rise of the (modelling) bots: towards assisted modelling via social networks. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. ASE ’17, pp. 723–728. IEEE Press, ??? (2017) Phaithoon, S., Wongnil, S., Pussawong, P., Choetkiertikul, M., Ragkhitwetsagul, C., Sunetnanta, T., Maipradit, R., Hata, H., Matsumoto, K.: FixMe: A GitHub bot for detecting and monitoring on-hold self-admitted technical debt. In: Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), Online Event, pp. 1257–1261 (2021). https://doi.org/10.1109/ase51524. 2021.9678680 Rios, N., Spı́nola, R.O., Mendonça, M., Seaman, C.: The most common causes and effects of technical debt: First results from a global family of industrial surveys. In: 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1–10 (2018). https://doi.org/10.1145/3239235.3268917 Rios, N., Spı́nola, R.O., Mendonça, M., Seaman, C.: The practitioners’ point of view on the concept of technical debt and its causes and consequences: a design for a global family of industrial surveys and its first results from Brazil. Empirical Software Engineering 25(5), 3216–3287 (2020) https://doi.org/10.1007/s10664-020-09832-9 Strauss, A., Corbin, J.: Basics of Qualitative Research: Grounded Theory Procedures 57

and Techniques. Sage Publications, Thousand Oaks, US (1990) Silva, J.D.S., Neto, J.G., Kulesza, U., Freitas, G., Rebouças, R., Coelho, R.: Exploring technical debt tools: A systematic mapping study. In: International Conference on Enterprise Information Systems (ICEIS), pp. 280–303 (2022). https://doi.org/10. 1007/978-3-031-08965-7 14 Storey, M.-A., Zagalsky, A.: Disrupting developer productivity one bot at a time. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. FSE’16, pp. 928–931. ACM, ??? (2016). https: //doi.org/10.1145/2950290.2983989 Turing, A.M.: Computing machinery and intelligence, pp. 11–35. MIT Press, Cambridge, MA, USA (1950) Urli, S., Yu, Z., Seinturier, L., Monperrus, M.: How to design a program repair bot?: insights from the repairnator project. In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. ICSE ’18, pp. 95–104. ACM, ??? (2018). https://doi.org/10.1145/3183519.3183540 Vinciarelli, A., Esposito, A., André, E., Bonin, F., Chetouani, M., Cohn, J.F., Cristani, M., Fuhrmann, F., Gilmartin, E., Hammal, Z., Heylen, D., Kaiser, R., Koutsombogera, M., Potamianos, A., Renals, S., Riccardi, G., Salah, A.A.: Open challenges in modelling, analysis and synthesis of human behaviour in human–human and human–machine interactions. Cognitive Computation 7(4), 397–413 (2015) https: //doi.org/10.1007/s12559-015-9326-z Venkatesh, Morris, Davis, Davis: User acceptance of information technology: Toward a unified view. MIS Quarterly 27(3), 425 (2003) https://doi.org/10.2307/30036540 Solingen, R., Basili, V., Caldiera, G., Rombach, H.D.: Goal Question Metric (GQM) Approach. John Wiley & Sons, Inc. (2002). https://doi.org/10.1002/0471028959. sof142 Venkatesh, Thong, Xu: Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Quarterly 36(1), 157 (2012) https://doi.org/10.2307/41410412 Wyrich, M., Bogner, J.: Towards an autonomous bot for automatic source code refactoring. In: 2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE), pp. 24–28 (2019). https://doi.org/10.1109/BotSE.2019.00015 Wessel, M., Zaidman, A., Gerosa, M.A., Steinmacher, I.: Guidelines for developing bots for github. IEEE Software 40(3), 72–79 (2023) https://doi.org/10.1109/MS. 2022.3224813 Yetistiren, B., Ozsoy, I., Tuzun, E.: Assessing the quality of github copilot’s code

58

generation. In: Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering. PROMISE 2022, pp. 62– 71. Association for Computing Machinery, New York, NY, USA (2022). https: //doi.org/10.1145/3558489.3559072 Zue, V.W., Glass, J.R.: Conversational interfaces: advances and challenges. Proceedings of the IEEE 88(8), 1166–1180 (2000) https://doi.org/10.1109/5.880078

59

Related documents

• The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability#978496
• Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction#978497
• The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs#978498
• DeltaSelect: Affordable A/B Testing for Coding Agents#978499
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500