Toward an Understanding of Developer Behaviour while Using Bug Localization Tools Pablo Diaz Pedreira
[email protected] The Open University Milton Keynes, Buckinghamshire, UK
Tamara Lopez
[email protected] The Open University Milton Keynes, Buckinghamshire, UK
arXiv:2605.04828v1 [cs.SE] 6 May 2026
Abstract Bug fixing is a complex and time-consuming task in software development. Bug localization research tends to focus on the accuracy of automated tools that suggest source code files for developers to look at. However, little is known about how developers use these tools in practice. This paper reports on an ongoing qualitative user study. Eleven participants worked through four realistic bug localization tasks in a controlled environment and were given varying levels of support information offered by a specialized tool. Participants were asked to think aloud in a semi-structured interview session. The preliminary findings provide insight into three aspects of practice: how developers interact with tools, the role social and contextual information plays, and problem solving. The study demonstrates that bug localization is complex and suggests that the adoption of effective tools depends on more than their accuracy.
CCS Concepts • Software and its engineering → Software maintenance tools; • Human-centered computing → Empirical studies in HCI .
Keywords software maintenance, qualitative study, software errors ACM Reference Format: Pablo Diaz Pedreira, Tamara Lopez, and Michel Wermelinger. 2026. Toward an Understanding of Developer Behaviour while Using Bug Localization Tools. In Proceedings of The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026). ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
When an error is found in a software system, it is typically recorded in the form of a bug report (BR), whose main body contains the description of the problem [6, 9, 23]. Later, the BR is triaged and a developer starts to fix the error. Among the first tasks, the developer must determine where in the source code the changes need to be done. This is called bug localization (BL) and has been identified as the most time-consuming part of the bug-fixing process [7]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. EASE 2026, Glasgow, United Kingdom © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
Michel Wermelinger
[email protected] The Open University Milton Keynes, Buckinghamshire, UK
Automating BL has been extensively studied using information retrieval [4, 16, 22] and machine learning [12, 18, 21]. Both approaches take one BR and the source code as input, and return a ranked list of recommended files to apply the necessary fixes. Research has focused on improving the accuracy of BL tools, primarily using new computational techniques and additional inputs. Accuracy is evaluated through closed bug reports from open source projects where the files fixed for the reported bugs are known [19]. By centring the research on accuracy, developers who use the tool’s recommendations to support their BL work have been overlooked. This work aims to address that gap. The objective is to collect evidence on how a BL tool is used by developers in a scenario similar to a real use case, and to assess how different information given by the tool affect the developer’s behaviour, asking: • How do software developers use BL tool information to find the reported error? To answer this question, a study was performed in which participants undertook four BL tasks, and were asked to think aloud. Additional questions were asked to help participants explain their problem solving process. A qualitative analysis of the participants’ dialogue and actions is underway. So far, a preliminary analysis of 11 sessions has identified three patterns related to the usage of tools, the social environment, and problem solving. Behaviours in each pattern reveal an underlying complexity of locating bugs beyond the simple prediction offered by BL tools, regardless of their accuracy. The examples include iterative refinements in key source code searches, requests of additional information not present in the BR, and reading the BR more than once to change the bug search approach. Furthermore, we saw that the output of a BL tool may direct, reinforce, or make the developer reconsider key decisions, but it also may mislead and distract.
2
Background
The BL literature focuses on new automated tools and their accuracy, e.g. [16, 21, 22], even to predict whether a ranked list is likely to be effective or not [10]. While these approaches aim to evaluate new BL techniques or new sources of information, how the tools can help in the broader context of bug fixing and how and whether they are actually used by developers is not clear. Two surveys among software developers have assessed the perceived value of BL tools in real projects. One is focused on industry [11], while the other includes open source developers [8]. These studies found that over 90% of developers gave “Essential” and “Worthwhile” ratings when asked for the importance of BL tools [8], and that 68% are willing to use them [11]. This positive response contrasts with the expectations of the tool’s accuracy. Only 12% [8] and 20% [11] of participants would be satisfied if given a
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Pablo Diaz Pedreira, Tamara Lopez, and Michel Wermelinger
correct Top-1 recommendation only 20% of the time. For the use of a tool to be satisfactory, 3 out of 4 participants expect 75% of Top-1 results to be accurate in one study [8] while 4 out of 5 participants of the other study expect 80% accuracy [11]. This level of accuracy is still out of reach for BL tools [12, 18, 21]. The main reason for the lack of adoption of BL tools among professional developers has been identified as the lack of confidence in the tools [11], with developers preferring to set breakpoints or print statements to understand the program behaviour. Overall, the results in prior work show a perceived need for help in locating bugs, but that tools have to be reliable.
3
Method
The aim of this study is to observe how developers use a BL tool in similar conditions to an actual use case. Following Braz et al. [3], the study uses task-directed sessions in which the participant tries to locate the buggy code related with example BRs. Participants think aloud while performing the tasks [17], with semi-structured questions asked to gain further insight into their thoughts. After that, the participant’s dialogue and actions were analysed.
3.1
Study design
The study aims to create a setting similar to a real use case of a BL tool, in order to gather truthful reasoning and actions. All participants were given the same BRs for the same project (Tomcat), to compare their BL approaches. We selected Tomcat for 4 reasons. First, it is used extensively in BL research, making it easier to translate the observations to other works. Second, it is complex enough for the use of a BL tool to be helpful while not being too overwhelming for the participants, given the time constraints. Third, it is written in Java, a widely used programming language, that made it easier to recruit participants. Fourth, some participants might already be familiar with this popular project, allowing us to contrast their BL approach with those who don’t know the codebase. For the tasks, we chose 4 closed BRs of varying difficulty for the tool, measured by its accuracy: a single fixed file, ranked first; two fixed files, one ranked first, the other outside the Top-10; a single fixed file, in the Top-10 but not first; no fixed files in the Top-10. References to the source code are one of the main strategies used by BL tools [4, 16, 22] and by developers [5]. To minimize the bias of the tool and the risk that participants would use a single strategy, we chose BRs that refer to source code in different ways: a stack trace, a code example, a file name, and no direct reference. In Tomcat, the mean number of fixed files per BR is 2.4 and the median is 1 [19, 20]. We thus selected BRs with 1 to 3 fixed files. Table 1 summarizes the BR selection.
Table 1: Selected BRs BR id
Fixed files (Rank)
Reference type
54087 54095 54124 54144
HttpServlet (1) DefaultServlet (1), TestDefaultServlet (21) AsyncContextImpl (4) Util (31), Out (498), TestOut (1511)
stack trace none code example file name
The BL tool selected is DreamLoc [13], which uses machine learning. We modified DreamLoc to take additional information from a project’s GitHub repository and trained it on Tomcat for this study. The added information are seven numbers about files and BRs: recent changes, days since the last bug-related change, total changes, number of authors, lines of code, percentage of lines of code changed, and ratio of number of authors to file size. The model was trained and tested with disjointed sets of closed BRs. The 4 BRs used in the study were selected from the test set, according to the criteria given before, and therefore the model had not seen them. For each BR, we produced three kinds of information: the recommendation list (RL) of the top-10 files; their confidence scores (CS), as computed by DreamLoc; and their summaries (FS), generated by Claude Sonnet 4.0. Each was increasingly introduced from task to task. To reduce confounding factors about how participants locate bugs, we divided participants evenly into 4 groups, each tackling the BRs in a different order and thus with different associated information (see Table 3). We chose IntelliJ as the IDE for participants to use because it is commonly used in professional Java projects, and provides two key features for this study: advanced search options and a tree representation of the project structure and files (Figure 1). To address a potential lack of familiarity with IntelliJ, participants could complete a small introductory practice task. All but 2 participants skipped the introductory task, marked by an asterisk in Table 2.
Figure 1: IntelliJ IDE showing BR 54095 for Task 4. Tool information (ranked list, confidence scores, file summaries) in orange. Project explorer in red. BR in blue and green.
3.2
Participants
Sessions were held with 11 participants with different levels of software engineering expertise. They were not expected to have used BL tools or IntelliJ before. Contrary to our hopes, no participant was familiar with Tomcat. Therefore, our setting is more akin to an onboarding scenario where a developer works on a project for the first time. Table 2 shows the demographic information for participants: the professional software and Java experience are given in years; the bug fixing experience is as participants reported it; the problem-solving confidence level is summarized from their answers; JS stands for Javascript.
Toward an Understanding of Developer Behaviour while Using Bug Localization Tools
Table 2: Participants’ demographics. Part
Prof
Java
Bug fixing
Confident
Other languages
A1 A2 A3* B1* B2 B3 C1 C2 C3 D1 D2
19 15 4 10 2.5 14 10 4 28 2 30
9 <1 No 2–3 3–4 3 No <1 No No 25
7/10 Daily Only own code All the time 1.5 years 7 years 20% 50% 66% Only own code Daily
Very Fairly Very Very Not Very Very Fairly Fairly Fairly Very
C, C++ Python, Perl Python, C, R C#, JS, Python JS C# C#, C++, Python JS, Python JS, SQL Python, Cobol
3.3
Data collection
The sessions were an informal meeting between the first author and the participant. The tasks were performed exclusively with the researcher’s computer, ensuring a common development environment. Eight sessions were in person; 3 via videoconference, with remote control over the researcher’s computer. The protocol was otherwise identical. During the session, the participant’s voice and screen were recorded and observation notes were taken. Table 3: BR distribution for each task and each participant. Group (size)
task 1 RL
task 2 RL + CS
task 3 RL + FS
task 4 RL + CS + FS
A (3) B (3) C (3) D (2)
54087 54144 54124 54095
54095 54087 54144 54124
54124 54095 54087 54144
54144 54124 54095 54087
Each participant was given 4 BL tasks. For each task, the participant was given a BR, the extra information (Table 3), and the project’s code. The participant was expected to read the BR and try to locate the buggy part of the code. Each task was timed for 15 minutes to keep the session’s pace and to reduce the risk of not going through the 4 tasks, but participants were free to end the task early or take a bit more time. Importantly, the study is non-performative: participants were not expected to actually locate nor fix the bugs. When participants were silent, they were prompted with an open-ended question about their current action, e.g. “Why did you choose to check this file?” After finishing each task, the participants were asked to briefly evaluate both the BR and the provided information (recommendation list, confidence score, file summaries). The demographic questions (Table 2) were only asked after a participant completed all tasks, to reduce researcher bias during the session. This study was approved by the ethics board of our university.
3.4
Analysis
Once the data was collected, the audio was transcribed to an anonymised script, and the video to an action log of the participants’
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
actions. The action log does not reflect the actions’ intent, which is covered by the observation notes. An inductive approach was chosen to provide an expansive view on BL tool usage. Following [2], the analysis process began with familiarization: examining the transcript, the action log, and the observation notes. Here, three patterns were identified, directed by the observation notes. The patterns correspondingly suggest three different aspects of the problem to be considered: developer interaction with BL tools, social and contextual information, and problem solving. The three patterns and these three aspects became the focus of the next steps in the analysis presented in this work. In a second cycle of familiarization, 8 episodes were extracted for detailed analysis. Each episode represents 3 to 10 minutes of a task. A third iteration over the episodes, consisted in two levels of coding [2]: attribute codes to organize the data and topic codes to sort the data into searchable categories relevant to the research question. The following is an example: [Attribute Codes: Transcript (data type); December 2025 (time period); Edinburgh coffee shop (location)] [Topic Code: BL Tool usage] 45:50 - 46:00 I’m looking for a little one (reading the BL tool file summaries), because it’s like a lot of these files size, some of this are huge. Two thousand line files. The codebook, additional details, and examples of the process are included in the supplementary materials.
4
Findings
The following subsections describe eight episodes that illustrate how BL tools are used and how they influence developer decision making. Participant quotes have been normalized when necessary to improve readability, but the full dataset preserved all interjections and repetitions to give indications into reasoning of the participants and to connect thoughts with actions. The ellipsis (...) is used to indicate a pause and when between square brackets ([...]) indicates parts of the dialogue omitted.
4.1
Deciding where to start
To locate a bug, developers tend to take a code reference from the BR and search the codebase for it [5]. In this study, we observed that searches sometimes are a sequence of iterative refinements, with participants using the tool’s results to move through the source code with purpose, rather than in an exploratory manner. In episodes 1 to 3, participants locate BR 54087, which has a stack trace. The only fixed file (HttpServlet.java) is the first recommended file and appears in the middle of the stack trace, whereas the first file in the stack trace (Request.java) appears in rank 4. Episode 1 is taken from task 2 of participant B3: locate BR 54087 given the recommended files and their confidence scores. B3 comments while reading the BR: “The stack trace is really useful!”. B3 follows by checking the BL tool recommendations: “And if I were to look into the BL tool, we’ve got the HttpServlet, seems quite relevant. So it reflects what the stack trace requires.” The participant opens the HttpServlet file, ranked first by the BL tool, and searches within the file first for “last”, and later for “getLastModified”, both taken from the BR stack trace. The BL tool aids the decision to select a file among the stack trace to start the search. Additionally, it helps reduce the search space, instead of a global search.
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
Using the BL tool does not always guarantee that suggestions will be followed. Episode 2 is taken from task 3 of C2: locate BR 54087 using the recommended files and their summaries. C2 checks the ranked list and the summaries but discards the suggestions. Their reasoning is: “Right. Interesting this talking [in the BR] about passing dates and none of the recommended files seem to have anything to do with that. I see an abstract HTTP processor [hovers the mouse over file AbstractHttp11Processor, the 9th recommended file]. But a low level abstract might not be what we necessarily need to be looking at there. So instead I’m going to see if I can open up this section [file Request, the first in the stack trace].” Next, C2 navigates to file Request and searches for function “getDateHeader”. The participant does try to connect the stack trace with the tool recommendations, but discards the suggestions, deeming a low level abstraction file to not be related. This judgement could be because C2 was not familiar with the project nor Java. However, the stack trace by itself may have the same directing capability as the ranked list. Episode 3 is taken from task 4 of participant D2: locate BR 54087 given all information. D2 jumps to the stack trace, skipping other parts of the BR: “I got a stack trace, I got an error.” Next, D2 navigates to the file Request using the project explorer. Like in episode 2, the participant searches within Request for “getDateHeader”, the function at the top of the stack trace. When prompted about why they chose this approach, D2 responded: “Well, I went straight to the top of this stack. Found this. Found the tests [referring to value checks in the source code].” The episode shows that the same reduction of the search space takes place (directed local file search instead of global searching). In these episodes, the tool was used in different ways to decide where to start searching from a stack trace. First, it supported the correct selection of the file from the stack trace. Second, the selection made in the stack trace was ultimately discarded, possibly due to a lack of project and Java knowledge. Third, it was ignored and the person examined the stack trace in order. In episodes 2 and 3, participants did not have the time to reach the actual buggy file after their initial choice, underscoring the importance of an early search decision. What separates episode 1 from 2 and 3 is connecting the BR stack trace (where the first file is not the buggy file) and the tool suggestions (where the first file is the buggy file) before beginning the search. This allowed B3 to open directly the right file to locate the buggy code.
4.2
Filling the gaps
The previous section showed that the BL tool information has an active role in developer’s choices, but the support offered has limitations. As observed, the BL tool support information sometimes is not enough to complete the localization of the bug, and therefore participants seek additional sources, either formal (like design documentation) or social (like turning for help to the reporter). The following episodes illustrate how the information from the BL tool needs to be supplemented by other kinds of information. Episode 4 is taken from task 4 of participant C1: locate BR 54095 given all the information (ranked files, confidence scores, file summaries). C1 starts out confident about the tool’s suggestions and deems all types of support information useful. However, for the last steps of the localization, they found it necessary to identify more
Pablo Diaz Pedreira, Tamara Lopez, and Michel Wermelinger
details to continue. So while the support information was welcome, it fell short and additional documentation was requested. C1 starts to relate the changes mentioned in the BR with the file recommendations and summaries, and adds: “That DefaultServlet [rank 1 file], I can read that and learn what that is [reads the file summary].” The confidence score is then evaluated: “This confidence [score] seems a bit low [75/100] because it seems 100% obvious that that’s the code that has been changed [refers to the patch mentioned in the BR].” C1 follows by searching the terms “gzip” and “compression” in DefaultServlet. At the end of the task, rather than being confident on where the bug is, C1 explained: “In real life, I probably spent a lot of time going through the changes... I’d just roll the git diff to figure out what he’s actually done to get an overview of the approach. But it’s also to me like an architectural decision, so it might be a case where I was looking at a software engineering design discussion.” Similar observations were mentioned throughout the task: “But I also want the bigger picture: Where this fits in, in the architecture of the tool.”, “You would need the documentation... The architectural design of the system.” In episodes 5 and 6, the participants only have access to the file recommendation list, and the BL tool is perceived differently. Episode 5 shows that BL tools are not always enough to guide and support developers on their own, even when accurate. Episode 5 is taken from task 1 of participant D1: locate BR 54095 (the same as in episode 4) given the ranked files. D1 starts being confused about the BR: I’m saying this description doesn’t tell me what I’m meant to do. [...] If I were actually a member of this project, I might know what happens [in the BR].” The confusion motivates D1 to look at the BL tool information, but seems to not help: “So that also means that I don’t feel able to interpret these file recommendations. I mean, these are a bunch of files, they could be relevant in some way. [...] I could open up one of these files and look at it, but I wouldn’t know what I would be looking for, how it related to the task that I meant to perform.” At the end of the task, the participant was still unclear about the bug and was not able to find any potentially buggy code. The file recommendation list alone does not seem to help, despite D1 relying on it and the BL tool ranking first the fixed file. Episode 6 is taken from task 1 of participant B2: locate BR 54144 given the ranked files. In this episode, the BR lacked context and the tool was not accurate, so B2 would resort to asking the reporter. B2 has a number of questions about the BR: “So it’s sort of clear, but I’m missing context. [...] Why are they erased now? What problems is this creating along the line? I try to visualise what else it [the bug] is impacting.” The evaluation of the BL tool by B2 was negative: “It was not very useful. Having a list of files, while I was trying to locate if there’s any of these calls happening in those files... I could have saved time if I started from the place that it is pointing to [Out file mentioned in the BR].” The BL tool file recommendations for this BR do not contain any buggy file. In turn, B2 perceives that the tool output wasted their time. Nonetheless, B2 managed to make progress in the task, commenting on how they would continue: “If I read this [the BR], I will have an idea where to start looking like I’ve done at the end now, but I would have to feed the person that wrote this file with a couple of questions.” These episodes show that developers rely on contextual information from other sources, both technical and social when finding
Toward an Understanding of Developer Behaviour while Using Bug Localization Tools
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
bugs. They demonstrate that the information from the tool cannot always fill that gap. The lack of additional documentation, reading support, and contextual information from the BR is a limitation of current tool approaches. Confidence scores and file summaries may compensate somewhat for this limitation in the early stages of the localization, as suggested by the contrast in tool perception in episode 4 (full support information) compared to the other two.
BL tool information is a part of the iterative comprehension process that characterizes bug localization. The additional thoughts, effort, and increase in source code familiarity, after trying to locate the bug for a while, may allow the participants to elicit new information from both the BR and the tool. This in turn results in following a new direction (episode 7) or reinterpreting parts of the BR and tool information (episode 8). This means that tool suggestions are not necessarily useful only at one time, presumably at the beginning of the task, but rather they evolve with the information developers gather and understanding they gain about the problem.
4.3
Reinterpreting available information
So far, all participants have started the tasks by reading the BR in varying levels of detail. Occasionally during the task, the participant gets stuck, resorts to reading the BR again, and ends up changing their approach to the task. In the process, the way participants engage with the BL tool is also affected. Revisiting the BR clarifies the situation for the participant. Sometimes the BL tool has led the participant astray, and sometimes the BL tool is the mechanism that gets the participant moving forward again. Episode 7, like episode 6, is taken from task 1 of participant B2: locate BR 54144 given the ranked list. In episode 7, after following the tool recommendations, B2 decides to stop checking files and reads the BR again: “I don’t think this has anything to do with that [the BR]. [...] Let me read the task again.” During the BR revisit, B2 finds a file reference missed in the first read: “OK, maybe I should start from this one [points to file Out in the BR]. If I can find it [the bug], and it’s not too late. [...] The code that the plugin generates calls value.toString(). Let me see if that’s true. [opens Out.java] There are any call here? OK, it’s there.” B2 then is able to continue the task reading file Out and searching for “toString”. B2 comments at the end of the task: “I think I’ve managed to locate now, with the description, where [the BR] was pointing me to and that I’ve neglected [...] until the end.” Episode 8 is taken from task 3 of participant B3: locate BR 54095 given the recommended files and their summaries. B3 seems stuck while reading DefaultServlet, the first ranked file: “Not entirely sure where [in the source code] it [the BR] is talking about at the moment”. Then B3 spends 3 minutes looking at the BL tool before deciding to go back to the BR: “Making sure that I’m... Make sure I understand the problem sufficiently.” While reading the BR, B3 starts filtering the tool recommendations with the aid of the file summaries: “Unless the bug is about the samples [files ranked 3 and 4], but it doesn’t, it doesn’t look that way. I do feel like a DefaultServlet behaviour. Somewhere about the DefaultServer.” After reading the BR, B3 checks file DefaultServlet, and discards it. B3 goes back to the tool suggestions and moves to the rank 2 file, AbstractHttp11Processor.java. There, B3 starts to find parts of the code that match the expectations of where the bug should be. In reflecting on the BL tool for this task, B3 states: “First, I didn’t find it useful [the BL tool], but in the end helped me find the problem because it led me to the file [AbstractHttp11Processor]. There is no way I would have spotted it otherwise.” The second reading of the BR seemed important to reframe the tool’s suggestions. File AbstractHttp11Processor is not a buggy file for the BR. Nonetheless, the participant tried searching within DefaultServlet during the task, even before episode 8, and after arriving at AbstractHttp11Processor B3 made progress in their understanding of the bug. The tool was perceived as beneficial and ultimately allowed B3 to move forward.
5
Limitations
In bringing together responses from different participants, the findings in this report provide a constrained view on problem solving during bug localization. Some aspects of the study design may have influenced the interpretation. For example, though the study was designed to reflect tool use in natural settings, time restrictions may have pushed participants to reject tool output or to change objectives. In addition, the lack of familiarity with Tomcat, while planned for in the task design, may have affected behaviour. Finally, the analysis presented here is preliminary, and was primarily performed on a subset of the sessions. As a task-directed study, time and difficulty constraints cannot be removed. These limitations were mitigated by making the tasks non-performative and being flexible about time. The interpretation made here will be checked through deeper engagement with the full dataset. The study does not cover all possible scenarios, like multiple fixed files in the 10 recommendations shown. Extra scenarios were discarded after the pilot phase, to avoid participant fatigue [14]. The study uses one BL tool and one LLM because we are interested in how users engage with the information such tools produce, rather than evaluating the tools’ accuracy. Using any other BL tool and LLM would similarly support the research aim.
6
Discussion
The preliminary analysis presented here is based on early patterns identified in the data. The patterns themselves were chosen as a point of departure to provide context for the data analysis and interpretation of the tool interaction. It is apparent that the developers engage with the tool, and use it in various ways as described in Section 4. By inspecting a series of episodes, we noted instances of how software developers use BL tool information to find the reported error. First, by guiding the decision of where to start searching following a stack trace (episodes 1, 2, 4). Second, by reinforcing the information available in the BR (episodes 1, 4). Third, by offering an alternative option when the developer gets stuck (episode 8). We also noted some of the limitations BL tools face in practical scenarios. They are ignored (episode 3), they are not able to support an already confused developer (episode 5) [15], and are occasionally misleading (episodes 6, 7). One known problem in BL is the low quality of BRs [11], partly because not all the information needed is present in them [1]. Episodes 4, 5, and 6 confirm that developers still require additional information even when they use tools. However, comparing episode 4 with 5 and 6 suggest that additional support information could
EASE 2026, Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom
alleviate this need [15], at least in early stages of bug localization. Future analysis work may provide additional evidence to support this possibility. Previous work found that developers require a high level of accuracy for the tools to be satisfactory and therefore adopted [8, 11]. However, during the study, no participant stated any problem with the tool’s accuracy. To further support this, in the data analysed so far, there was no engagement with the confidence score (used here as a proxy for accuracy) other than the one in episode 4. One explanation could be that participants may form an idea of the tool’s accuracy during task 1, when they only have access to file paths. Further data needs to be analysed to support this premise. In contrast, file summaries were used in episodes 2, 4, 8, and were considered for the participant’s reasoning as demonstrated with quotations. This may suggest that, in contrast to previous evidence based on survey data [11], the barrier to using BL tools is not strictly low accuracy.
7
Conclusion
To our knowledge, this research is the first qualitative empirical study to examine how developers engage with bug localization tools. By observing developers on realistic tasks supported by varying amounts of information (ranked file list, confidence scores, file summaries), we identify how BL tools shape behaviour, and how developers integrate recommendations into their reasoning. The main contribution gives insight beyond tool accuracy: tool usage itself, social context information, and problem solving challenges. We document the rational and informational strategies developers employ when interacting with BL tools, highlighting developers’ need for additional sources of information, and showing that BR re-reads help overcome problem solving challenges. Our findings show that developers do not rely on recommendations alone; instead, they combine BL outputs with purposeful searches and contextualize results through the bug report. Even features intended to help, such as confidence scores or LLM-generated summaries, influence behaviour in nuanced ways, sometimes improving navigation but also occasionally misleading. These findings underscore the importance of BL tools to communicate uncertainty, rationale, and context more effectively. Future work will expand the analysis, documenting additional strategies developers use with BL tools, how developers relate the BR with the BL tool information and the source code, and refine how multiple sources of information influence developer behaviour. Ultimately, this research contributes to understanding of tool assisted bug localization practices, identifies aspects of this complex problem space that developers need more help with, and shows the limitations of BL tools in real-world scenarios.
Supplementary Files https://doi.org/10.6084/m9.figshare.31449289
Acknowledgements We thank the developers who participated in our study. We thank the reviewers for their helpful comments. The research protocol was approved by The Open University Human Research Ethics Committee, ref. 2025-1006-1. This work was partially supported by
Pablo Diaz Pedreira, Tamara Lopez, and Michel Wermelinger
project Empirical Data-Driven Bug Localization, funded by Huawei Technologies (Ireland).
References [1] Jorge Aranda and Gina Venolia. 2009. The secret life of bugs: Going past the errors and omissions in software repositories. In Proc. 31st Int’l Conf. on Software Engineering. IEEE, 298–308. [2] Andrea J. Bingham. 2023. From Data Management to Actionable Findings: A FivePhase Process of Qualitative Data Analysis. International Journal of Qualitative Methods 22 (Oct. 2023), 1–11. [3] L. Braz, C. Aeberhard, G. Çalikli, and A. Bacchelli. 2022. Less is More: Supporting Developers in Vulnerability Detection during Code Review. In Proc. 44th Int’l Conf. on Software Engineering. ACM, 1317–1329. [4] Oscar Chaparro, Juan Manuel Florez, and Andrian Marcus. 2019. Using bug descriptions to reformulate queries during text-retrieval-based bug localization. Empirical Software Engineering 24, 5 (Oct. 2019), 2947–3007. [5] Bogdan Dit, Meghan Revelle, Malcom Gethers, and Denys Poshyvanyk. 2013. Feature location in source code: A taxonomy and survey. Journal of Software: Evolution and Process 25, 1 (Nov. 2013), 53–95. [6] Abram Hindle and Curtis Onuczko. 2019. Preventing duplicate bug reports by continuously querying bug reports. Empirical Software Engineering 24, 2 (April 2019), 902–936. [7] Thomas Hirsch and Birgit Hofer. 2021. What we can learn from how programmers debug their code. In Proc. 8th Int’l Workshop on Softw. Eng. Research and Industrial Practice. IEEE, 37–40. [8] Pavneet Singh Kochharand, Xin Xia, David Lo, and Shanping Li. 2016. Practitioners’ expectations on automated fault localization. In Proc. 25th Int’l Symposium on Software Testing and Analysis. ACM, 165–176. [9] An Ngoc Lam, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N. Nguyen. 2017. Bug Localization with Combination of Deep Learning and Information Retrieval. In Proc. 25th Int’l Conf. on Program Comprehension. IEEE, 218–229. [10] Tien-Duy B. Le, Ferdian Thung, and David Lo. 2017. Will this localization tool be effective for this bug? Mitigating the impact of unreliability of information retrieval based bug localization tools. Empirical Software Engineering 22, 4 (Aug. 2017), 2237–2279. [11] Wei Li, Qingan Li, Yunlong Ming, Weijiao Dai, Shi Ying, and Mengting Yuan. 2022. An empirical study of the effectiveness of IR-based bug localization for large-scale industrial projects. Empirical Software Engineering 27, 2 (2022), 47. [12] Zhengmao Luo, Wenyao Wang, and Caichun Cen. 2023. Improving Bug Localization With Effective Contrastive Learning Representation. IEEE Access 11 (2023), 32523–32533. [13] Binhang Qi, Hailong Sun, Wei Yuan, Hongyu Zhang, and Xiangxin Meng. 2022. DreamLoc: A Deep Relevance Matching-Based Framework for bug Localization. IEEE Transactions on Reliability 71, 1 (March 2022), 235–249. [14] Paige Rodeghero, Cheng Liu, Paul W. McBurney, and Collin McMillan. 2015. An Eye-Tracking Study of Java Programmers and Application to Source Code Summarization. IEEE Transactions on Software Engineering 41, 11 (Nov. 2015), 1038–1054. [15] Jonathan Sillito, Gail C. Murphy, and Kris De Volder. 2008. Asking and Answering Questions during a Programming Change Task. IEEE Transactions on Software Engineering 34, 4 (July 2008), 434–451. [16] Shaohua Wang and David Lo. 2016. AmaLgam+: Composing Rich Information Sources for Accurate Bug Localization. Journal of Software: Evolution and Process 28, 10 (Oct. 2016), 921–942. [17] Jacqueline Whalley and Nadia Kasto. 2014. A qualitative think-aloud study of novice programmers’ code writing strategies. In Conf. on Innovation & Technology in Computer Science Education. ACM, 279–284. [18] Xi Xiao, Renjie Xiao, Qing Li, Jianhui Lv, Shunyan Cui, and Qixu Liu. 2023. BugRadar: Bug localization by knowledge graph link prediction. Information and Software Technology 162 (Oct. 2023), 107274. [19] Yan Xiao, Jacky Keung, Qing Mi, and Kwabena E. Bennin. 2017. Improving Bug Localization with an Enhanced Convolutional Neural Network. In Proc. 24th Asia-Pacific Software Engineering Conf. IEEE, 338–347. [20] Xin Ye, Razvan Bunescu, and Chang Liu. 2014. Learning to rank relevant files for bug reports using domain knowledge. In 22nd ACM SIGSOFT Int’l Symposium on Foundations of Software Engineering. ACM, 689–699. [21] Xia Zhang, Ziye Zhu, and Yun Li. 2023. Enhancing Bug Localization through Bug Report Summarization. In Proc. Int’l Conf. on Data Mining. IEEE, 1541–1546. [22] Jian Zhou, Hongyu Zhang, and David Lo. 2012. Where should the bugs be fixed? More accurate information retrieval-based bug localization based on bug reports. In Proc. 34th Int’l Conf. on Software Engineering. IEEE, 14–24. [23] Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schroter, and Cathrin Weiss. 2010. What Makes a Good Bug Report? IEEE Transactions on Software Engineering 36, 5 (Sept. 2010), 618–643.