Automatic Assistance to Reduce Information Overload during Software Issue Report Understanding

Noname manuscript No. (will be inserted by the editor)

Automatic Assistance to Reduce Information Overload during Software Issue Report Understanding

arXiv:2611.07109v1 [cs.SE] 4 Nov 2026

Salma Begum Tamanna · Gias Uddin · Junaed Younus Khan · Lan Xia · Longyu Zhang

Received: date / Accepted: date

Abstract Modern issue tracking systems are used to log an issue and track it from its inception to closure. Issue reports with diverse discussions from stakeholders can be large and verbose, making them challenging to analyze quickly. Our developer interviews revealed that they struggled with the diverse information in issue reports and demanded automated techniques to classify and summarize information. We designed a tool, iSum (issue summarizer), that helps with the automatic classification and summarization of information types discussed in issue reports. The summarizer offers two views: 1. Developer View takes an issue report as input and outputs visualized summaries of information in the report. 2. Manager View takes a suite of all issue reports under a given software project as inputs and shows the prevalence and trends of different information types across the reports. We evaluated iSum through three studies. 1. In an empirical study over two benchmarks, iSum offers, on average, 11-13% higher F1-scores than state-of-the-art baselines to detect information types. 2. We conducted a user study in task-based setup with 30 developers to determine iSum’s effectiveness in issue exploration scenarios. Results showed increased productivity and accuracy with iSum compared to manual inspection. 3. We assessed iSum’s feasibility for development and manSalma Begum Tamanna University of Calgary E-mail: tamanna.sb13@gmail.com Gias Uddin York University E-mail: guddin@yorku.ca Junaed Younus Khan Bangladesh University of Engineering & Technology E-mail: junaed@cse.buet.ac.bd Lan Xia · Longyu Zhang IBM Canada E-mail: lan xia@ca.ibm.com, longyu.zhang@ibm.com

2

1

INTRODUCTION

agement tasks. It proved useful to both developers and project managers for issue comprehension and trend monitoring. Keywords Issue Report, Information Type, Summarization, Detection 1 Introduction Modern issue tracking systems (ITS) are crucial components in software development to log an issue/bug/defect about a software product and track the progress of the issue management from its inception to closure. Issue reports are often the cornerstone for team performance observability and monitoring in a software organization [28, 35, 41]. Various tasks are performed to manage issues in ITS, which can broadly be divided into four types: issue understanding, monitoring, triaging, and finally fixing. A significant body of research in software engineering (SE) has dedicated to utilize issue reports for software requirement analysis [26], to determine software design decisions [80], to understand the life cycle of bugs and their influence over different SE stakeholders [8], to categorize issue reports (e.g., features, bugs, or questions) [15,35,44,52,64–66,73,77,91,92], to develop tools and techniques for automated issue triaging and fixing [5,36,49,62,76,81,84–86,88] and others. An important prerequisite to support all such activities is issue understanding. Issue understanding denotes the activities of stakeholders to understand the contents of an issue. Various types of information exist in an issue report [10], such as comments about bug reproduction, solution discussion, etc. However, the abundance of issue reports or the plethora of information within a single large and verbose issue report can lead to information overload, making them difficult to analyze and comprehend effectively. Additionally, it is not clear which information in an issue report can help in understanding the issue and whether developers need tool support to understand large issues quickly and efficiently. This paper focuses on helping software developers and team leads during their understanding of an issue. The project started as part of a collaboration with an industrial team from a large international software company. The team collaborates with OSS developers in several of their open-source industrial projects, such as the design and operationalization of runtime systems (e.g., OpenJ9). Our discussions with the team find that issue understanding is often challenging, especially when issue reports are quite large and verbose. Due to the conversation among project stakeholders, these discussion threads accumulate a wealth of knowledge over time regarding the particular problem or the project, and stakeholders in the project can use those various pieces of information according to their needs. A triager may seek out interested or possible contributors, developers may wish to find out the issue replication circumstances; or managers may monitor the solution progress in the thread of conversation. However, issue discussion threads can grow long and complex [22] over time and overload the discussion with diverse information, making it challenging for project stakeholders to follow and locate the information they’re

3

seeking. In light of this, we recognized the necessity of automatic summarization of various information types as a way to support the understanding of issue reports. We began our study by interviewing 10 software professionals to gain insight into their direct experiences with issue discussion forums. We explored how they interact with issue reports and identified the challenges they face during issue understanding. Additionally, we assessed the demand for automated assistance tools that enable rapid yet efficient understanding of issues and examined the specific informational needs of different stakeholders. Our findings revealed that developers struggle with the diverse information discussed in issue reports and expressed the need for automated techniques to classify and summarize various information types, such as bug reproduction or solution discussions. To address this need, we introduce iSum (issue summarizer), an interactive toolkit that offers visualized summaries of the different information types in issue reports. To generate these summaries, iSum features a suite of classification models that can automatically detect information types in issue reports: 1. Stand-alone: as stand-alone models, we experimented with LightGBM [38], BERT, and RoBERTa. Among them, LightGBM surpassed the state-of-the-art (SOTA) model [10] of Random Forest (RF) by 6-8% in overall F1 score. 2. Ensemble: We then used these three stand-alone models in an ensemble with the baseline model to generate labels and pass them to the ensemble classifier with textual and conversational features. As the ensemble classifier, again, we experimented with Random Forest, LightGBM, BERT, and RoBERTa. The best ensemble model in iSum achieved 11-13% higher accuracy than the existing SOTA model [10]. We released iSum as an open-source GitHub plug-in, making it available for everyday industrial use and for other researchers to analyze information types in issue reports. In addition to analyzing iSum’s performance across two datasets, we conducted a user study to evaluate the effectiveness of iSum in supporting developers during real-world issue understanding scenarios. In a task-based setup involving 30 developers, we observed that developers were more productive and accurate when using iSum to assess issue reports for understanding or fixing bugs, compared to their performance without it. In a subsequent phase, we evaluated the feasibility of deploying iSum for developers and software teams to enhance their daily issue report comprehension and management tasks. We conducted interviews with 10 industrial and OSS developers, providing them with access to iSum while demonstrating its capabilities. We found that iSum was beneficial for both developers and project managers, aiding in understanding individual issue reports and monitoring trends and discussions across project issue reports. To the best of our knowledge, this is the first study to analyze the importance of issue report understanding using information types and assess its feasibility in professional settings by consulting both industrial and OSS developers.

4 2

UNDERSTANDING AUTOMATIC ASSISTANCE NEEDS FOR ISSUE REPORT UNDERSTANDING

Table 1: Demographic info of interview participants by years in industry, role and geographic location and team size P#

#Yrs

Role

Location

Team Size

P01 P02 P03 P04 P05 P06 P07 P08 P09 P10

2 15 4.5 5 3.5 4 4 4 15 15

Developer Project Manager Developer Developer & Project Manager Developer Developer Developer QA Engineer Developer QA Engineer

Asia Asia Europe Asia Asia Asia Asia North America North America Asia

8 140 18 7 3 2 35 50 6 11

2 Understanding Automatic Assistance Needs for Issue Report Understanding Past literature has claimed that navigating complex and lengthy issue discussions to find relevant information is challenging, especially for those unfamiliar with the issue’s context, and established a correlation between extended discussions, increased stakeholder involvement, and delays in issue resolution [10, 61]. They’ve also observed that redundant questions often pop up, even after they’ve been answered before in these lengthy issue discussion threads. However, there is a gap in understanding, from a practical perspective, how developers and project managers interact with issues in their everyday work environments without the aid of automated tools. We conducted a study to address this gap by examining the hands-on experiences of these key stakeholders and complementing the existing body of knowledge by adding a layer of user-centric analysis. We seek to uncover how open source software (OSS) and industry professionals engage with issue discussion forums, their perspectives on the importance of different types of information in these reports, and the challenges they encounter. This interview aims to evaluate the need for automated assistance tools, with the findings helping to customize these tools to better address the specific needs and challenges faced by users.

2.1 Study Methodology A series of semi-structured interviews have been conducted by the first author with 10 professional software engineers. We invited seven software engineers from our personal contacts, and the rest of the respondents were from our industry partner, IBM. These participants are situated in three different countries (Canada, Bangladesh, and Ireland) from three continents, and their roles spread across the spectrum of engineering and leadership roles. Some of our participants were working in large corporations like Amazon, Samsung, and IBM, and one respondent has their own software startup. We have

2.2

Issue Report Understanding Processes

5

Table 2: Interview Questions: Understanding Issue Report Exploration Approaches and Challenges Q# 1 2 3 4 5 6

Questions

Topic

Do you go through the issue discussion along with the issue details after the assignment? Do you keep yourself up to date with the issue discussion in ITS for the assigned issues? Do you often find that issue discussions contain some necessary information? Have you faced troubles in finding information from lengthy or complex issue threads? Do you think automated info detection from issue discussion can help stakeholders? What type of information do you mostly search for?

Process Process Process Challenge Assistance Needs Assistance Needs

listed the details of the participants in Table 1. (1) Among the participants, two have experience of working in the software industry as project manager. Six of the participants were in the developer role only and remaining two are QA Engineers. (2) 7 participants have industry experience of 2–5 years, while other three have around 15 years of experience. (3) Five respondents work/last worked in a team with less than 10 members. Others have/had a team of 11–140 members. (4) Among all the interviewees, four experts identified themselves as female and rest are male. (5) Everyone has experience of using ITS for issue management; though seven members directly use GitHub; others use JIRA or other internal ITS. All interviews were carried out and recorded via Microsoft Teams to facilitate transcription. Due to the limitations of auto-transcription accuracy, each interview underwent manual correction. Following the collection of demographic information, participants were asked to provide feedback on predefined interview questions, all of which were closed-ended. The response options were as follows: Q1–Q5 were binary (Yes/No), and Q6 allowed multi-selection from 16 predefined information types. Table 2 presents the interview questions.

2.2 Issue Report Understanding Processes Participants aimed to read an issue report contents and to monitor ongoing discussions in the report during their fixing of a bug or the implementation of a feature as denoted in the report. 2.2.1 Reading 80% Participants usually read the whole issue thread 10% Participants sometimes skip issue description 10% Participants sometimes skip issue discussion The typical approach among developers when they are assigned an issue is to read the entire issue thread comprehensively to understand the context and

6 2

UNDERSTANDING AUTOMATIC ASSISTANCE NEEDS FOR ISSUE REPORT UNDERSTANDING

specifics of the issue. This method is commonly adopted by 80% of participants, indicating that they examine the entire discussion when they receive an issue. However, there are instances where more experienced developers might deviate from this approach. For instance, two participants mentioned that they might skip parts of the issue report—such as the description or the discussion—if they feel they already have a sufficient understanding of the issue from the other details provided. Participant P03 noted, ”I used to see all the technical details and discussion forum when I was new, but after some years, I see the issue title first, and if it’s meaningful and it seems good to me that the issue title is enough for me, then I don’t go to all the technical details” P03. Additionally, those who have roles in project management tend to briefly review new issues to understand the project’s current status but do not engage deeply with the content unless necessary. Despite varying degrees of engagement with the content of the issues, the consensus among all participants is that issue discussions can be a valuable resource. As P02 emphasized, ”If anyone is interested to know what’s happening in this issue, they should go through each line of the issue” - P02. 100% Participants consider issue discussions a valuable resource of information

2.2.2 Monitoring 80% Participants always stay up to date with issue discussion 10% Participants check updates occasionally Nearly all participants (80%) follow updates on issues they are involved with. This proactive approach is crucial for keeping track of the resolution process, as highlighted by Participant P04, who stated, ”That’s necessary, because that way we have to keep track of everything that’s been going on” - P04. Two participants even subscribe to email notifications to ensure they don’t miss any updates. In smaller teams, the dynamic changes slightly, with developers often preferring direct communication over discussing issues through information tracking systems (ITS). Summary of Processes to Understand Issue Report. The usual practice of the developers is to go through the whole discussion thread of their assigned issues and keep up to date with the ongoing issue discussion since oftentimes they find valuable information from the discussion.

2.3 Issue Report Understanding Challenges 80% Participants frequently faced difficulties in issue understanding 10% Participants rarely faced difficulties in issue understanding When exploring the experiences of participants with navigating lengthy and

2.4

Needs for automatic assistance for information understanding

7

complex issue threads,the feedback highlighted a common challenge: difficulties in extracting necessary information from these extended conversations. This difficulty was echoed by nearly 80% of the respondents. One participant, P04, described the frustration: ”It actually happens quite frequently. Sometimes I go through the whole forum, but don’t get any results, or there is no clear answer that it has solved or not” - P04. This statement captures the essence of the challenge faced by many developers and project managers—diving through extensive discussions without guarantee of finding the information they need. Contrarily, participant P09 had a different experience, stating that such difficulties occurred only rarely. This variation in experience points to the subjective nature of the issue, possibly influenced by the individual’s familiarity with the project or their ability to efficiently navigate through discussions. P10 shared insights into the practical approach taken when faced with voluminous threads; repeatedly reading through dense material to ensure no critical information is overlooked: ”All comments may not be necessary but I don’t know which is important and which is not. So, I need to go over everything, and when I’m going through all the comments, it’s possible that while I’m just glancing, I might have overlooked the important points. Or sometimes I get lost in reading such lengthy things and I repeat reading it” - P10. This response illustrates the depth of the challenge, where even seasoned professionals find themselves re-reading and scrutinizing discussions to ensure no critical information is missed. Summary of Challenges during Understanding Issue Report. The primary challenge identified by participants in understanding issue reports was the difficulty of extracting essential information from long and complex discussion threads. Nearly 80% struggled with this often and finding it hard to locate the required information. This led to repetitive reading and scanning of conversations, with some participants worrying they might miss critical details.

2.4 Needs for automatic assistance for information understanding The participants wished for automatic assistance to automatically detect and summarize various information types discussed in issue reports (eg., bug reproduction, solution discussion, etc.). 100% Participants support for automation detection and summarization of diverse information

2.4.1 Relevant Information Identification To further understand which information is relevant to stakeholders in their work, we provide them with a list of information types, drawn from the taxonomy built by Arya et al. [10]. These information types are listed in Table 3

8 2

UNDERSTANDING AUTOMATIC ASSISTANCE NEEDS FOR ISSUE REPORT UNDERSTANDING

Table 3: Definition of information types found on issue discussion (Content Mgt = Issue Content Management) Info Type

Definition (based on [10])

Examples (from OpenJ9 Issue Reports)

Action on Issue

Comments about actions perform for issue management like closing, reopening, labeling and others Any question, statement or report regarding the reproduction of the bug About assigning or asking members for resolving an issue along with volunteer contribution or unwillingness The expected/ideal situation from user’s perspective or personal preference and opinion

“Closing, since Java 11 Windows is now compiling with MSVC 2017.”

Bug tion

Reproduc-

Contribution and Commitment Expected haviour

Be-

Future Plan

Any long term plan related to the issue

Investigation and Exploration

Suggesting investigation ideas or reporting results about the cause of the issue Redirecting issue discussions to another place Why an issue needs to resolved or a feature need to be added or changed

Content Mgt Motivation

Observed Behaviour

Bug

Questions or reports about the observed behaviour of a bug

Potential New Issues and Requests

Comments involving identifying new bugs or need of new features or any change while investigating an issue

Social Conversation

Any kind of small talk or showing gratitude or disappointment or other emotions. Suggestion or statement about design, implementation, constraints, challenges and references for solution Any question, statements or report about testing process, environment and results

Solution Discussion Testing

Task Progress Solution Usage Workarounds

Asking or providing update about the progress of the task towards the solution Question or suggestion on how to use the new solution. Temporary solution before the official one

“I still cannot reproduce this issue.” “please assign to me.”

“I expect it’s invalid to access the variable outside of holding the mutex, as the value can be modified by other threads.” “We should design the data record to be extensible in the future.” “We need to see the bytecodes to see why the check in the verifier is failing.” “Is it possible to move the discussion to a slack channel?” “OpenJDK is modifying Windows compilation for Java 11+ to use VS2017 and OpenJ9 should support the same.” “Crash or OutOfMemoryError when using AsyncGetCallTrace()” “Unfortunately I am now hitting another problem: if I run long enough, the flight service starts consuming all the CPU on the machine” “Thanks for looking into this!” “I think the assert should just be removed.” “Did something change with the SharedClasses.SCM23.MultiCL 0 test?” “Any update on this?” “Is that not the official way to use BN?” #10848 provides a “temporary fix for the 0.23 release.”

with definitions and examples. This list offers a consistent framework that helps stakeholders quickly identify and select relevant categories, organizing their thoughts more effectively. It also allows us to arrange information into familiar categories, reducing cognitive load and helping users focus on the most

2.4

Needs for automatic assistance for information understanding

9

pertinent information. Also, by knowing which information types are most sought after, the tools can be tailored to prioritize and highlight these types. Responses showed that most respondents face bug related issues; therefore, they are most interested in finding Observed Bug Behaviour, Bug Reproduction, Solution Discussion and Expected Behaviour related comments; but for feature enhancement issues, they look for motivation. One respondent said, in case where a quick solution is needed, like when the module owner is not available, they sometimes look for Workarounds. Though some inform for task progress, they use other management software; some still search the comment section to learn about Task Progress. However, no one is interested in reading the Social Conversation. Rather, one suggested to categorize Social Conversation related comments to identify the expressions or toxicity. Figure 1 shows the distribution of the information types that participants find important.

2.4.2 Relevant information Summarization When asked about the potential benefits of automated assistance for issue summarization, all participants voiced strong support for such technology. The consensus highlights a widespread challenge in current practices—manually navigating through extensive threads is not only time-consuming but also prone to errors, as critical details may be missed or overlooked. Participant P10 expressed the urgency for automation by stating, ”I may overlook the important points and I may get lost reading through and I will have to repeat reading, which is time consuming. So anything that is time-consuming can be automated, that is the basic rule of automation” - P10. This sentiment underscores the common frustration of needing to revisit lengthy threads multiple times to ensure no essential information has been missed. Participant P06 added another dimension to the discussion by emphasizing the need for categorization based on task progress or actions taken by other team members: ”I need to know what is the task progress in it or what other people have done on that. So I think if there is anything to distinguish among all the data, that would be better.” This remark illustrates a desire for structured information that can streamline the process of understanding task statuses and historical actions within a project. All feel this assistance can help all project stakeholders, but two emphasized that it would only help project managers, as commented by P09, ” I would have just read the issue, but I think there is a need to categorize it from a project point of view. That’s to me, maybe more meaningful” - P09. Summary of Needs for Automatics Assistance for information Understanding. Due to the information overload in lengthy and complex issues, project stakeholders feel the necessity of automatic identification of issue discussion information types. From the issue discussion, stakeholders mostly check for bug and solution related information.

3 10 NEEDS FOR INFORMATION DETECTION AND SUMMARIZATION IN ISSUE REPORTS Observed Bug Behaviour Bug Reproduction Solution Discussion Expected Behaviour Investigation and Exploration Task Progress Motivation Potential New Issues. Workarounds Action on Issue Solution Usage Contribution and Commitment

70 70 60 50 40 40 30 30 30 20 20 10 20

40 60 Frequency (%)

Fig. 1: Importance of information types to the participants

3 Needs for Information Detection and Summarization in Issue Reports The feedback from the interviews demonstrates that issue reports are not just mere notifications of problems; they contain detailed discussions of collaborative problem-solving efforts. Stakeholders rely on these discussions to extract valuable information for issue resolution. We consulted with our industrial partner about the study findings and found that the findings resonated with their difficulties in understanding the issue report. For instance, an OpenJ9 issue thread (refer to Figure 2), filled with 155 comments, contains crucial details like bug reproduction steps, suggestions for investigation, and a final solution. For example, a stakeholder posted a comment stating, ”New information—I just hit the same issue on Linux (Ubuntu 19.10).” This insight could be crucial for understanding the scope of the problem and potentially identifying any platform-specific factors contributing to the issue. But amidst all these comments, finding the important insights needed to understand the issue can be like searching for a needle in a haystack. The complexity of navigating such dense information is further compounded by the frustrations stakeholders experience when forced to read or re-read lengthy discussions. As encapsulated by Participant P02’s remark, ”when an issue has, for example, 40 comments, sometimes it’s annoying to read those comments.” Responses from our interview, presented in Section 2.3, underscore the inefficiencies and potential for oversight inherent in manual navigation of issue discussions, which not only slows down the resolution process but also increases the risk of missing crucial information. Recognizing the challenges outlined by participants, there emerges a clear and compelling need for automated tools that can streamline the understanding of issue discussions. During the interviews, all participants responded positively to the idea of automated assistance, particularly the automated identification of information types within issue discussions, indicating a unanimous agreement on its potential benefits. Such tools would mitigate the burdens

11

Fig. 2: Issue discussion thread of OpenJ9 #7684 where important information are spread in the discussion

of manual navigation and enhance the accuracy and efficiency of information retrieval from discussions, as further discussed in Section 2.4. Prior research, such as that by Arya et al. [10], has recognized the importance of categorizing information types within issue discussions. Their motivation stemmed from the understanding that issue reports often contain diverse information types, each serving different stakeholders’ needs, from developers seeking bug reproduction steps to project managers looking for task progress updates. Drawing from these insights, it becomes evident that navigating dense, information-rich discussions, as seen in the OpenJ9 project thread, poses a common challenge. Therefore, we focus on developing a more accurate automated assistance that can detect the diverse information from issue discussion and ease issue understanding.

4 iSum: Automatic Summarization of Information Types in Issue Reports We have designed iSum (Issue Summarizer), an end-to-end pipeline/framework as a suite of several toolkits that can be used to automatically detect information types discussed in an issue report or across issue reports. The tool iSum then summarizes the detected information types and their associated discussions in interactive and visualized dashboards.

4 12 ISUM: AUTOMATIC SUMMARIZATION OF INFORMATION TYPES IN ISSUE REPORTS

The design of iSum is motivated by our findings from the interviews (see Section 2) and our exploration of the information overload problems during software issue report understanding, as we undergone these problems with our industrial partners over a span of more than a year (see Section 3). In this section, we present the overall framework of iSum and the modules that comprise the framework (see Sections 4.1 - 4.3). We then discuss the implementation architecture of iSum that we deployed as a plug-in in collaboration with our industrial partner and that we used to assess the effectiveness of iSum in real-world tasks.

4.1 iSum Framework Issue Summarizer (iSum) is a toolkit that outputs visualized summaries of the various information types present in the issue reports. For this generation of information type summary, iSum employs a Machine Learning (ML) model that can automatically identify the information type in each sentence of an issue report. We take the catalog of 16 information types from Arya et al. [10]. The iSum framework has four major components (see Figure 3): two to collect and process issue reports and two to detect and summarize information types in the reports. Report Collector and Preprocessor: The collector component is used to download or fetch issue discussion threads from project repository. The processor module retrieves total 14 metadata as conversational features from the sentences of an issue discussion thread. We followed the guidelines set by Arya et al. [10] for defining these metadata. These conversational features include participant information, sentence length related information, structural (position) and temporal (time) information and existence of code snippets in a sentence. For the saved project, sentences and metadata of all issues in the repository get stored in a database and for a single issue report, these information directly go to the next component. Information Type Detector and Summarizer: This detector component is used to detect information type for each sentence in an issue report (discussed in Section 5). The summarizer component provides two dashboards of summaries of information types for the issue reports (discussed in Section 4.3).

Issue Report

Dashboard

Report Collector

Report Preprocessor

Information Type Summarizer

Database

Information Type Detector

Fig. 3: Schematic diagram of the iSum framework

4.2

Classification of Information Types

13

4.2 Classification of Information Types To classify the types of information within issue discussions effectively, we took a multi-faceted approach. First, we carefully selected four ML models renowned for their distinct strengths: Random Forest, LightGBM, BERT, and RoBERTa. Each model was chosen for its unique capabilities in handling various aspects of the classification task. Random Forest was selected as a baseline following Arya et al. [10] because it effectively captures conversational patterns. LightGBM was selected for its ability to efficiently handle highdimensional sparse TF-IDF features and capture non-linear interactions with conversational features, particularly on small and imbalanced datasets, while providing built-in regularization [38]. BERT [17] was chosen for its bidirectional attention mechanism, which captures rich contextual semantics crucial for distinguishing similar information types such as Motivation and Expected Behaviour. Finally, RoBERTa [51] was included as it builds upon BERT with improved pre-training strategies, yielding more robust text representations. Following model selection, we conducted extensive feature engineering to optimize the performance of RF and LightGBM and fine-tuned BERT and RoBERTa. Following this, we employed an ensemble technique to integrate the predictions of these standalone models, enhancing the overall classification accuracy. Notably, we also designated these same four models to serve as the final classifier within our ensemble framework. This decision allows users of iSum to utilize either a single model or an ensemble model according to their specific requirements, ensuring flexibility and adaptability in addressing diverse information classification needs. We will discuss more about the models in section 5 but discuss the feature engineering here. 4.2.1 Feature Engineering For feature engineering, we incorporated a range of features that span both textual and non-textual data from issue reports. Depending on the context, our model utilizes either one, both, or none of these features. Prior to feature extraction, we conducted text preprocessing. Text Preprocessing. Initially, we preprocessed the raw text from issue discussions by replacing code snippets with the keyword CODE and trimming sentences to remove unnecessary whitespace, ensuring cleaner data for analysis. Textual Features. For transforming the text into a format suitable for ML models, we utilized a Term Frequency-Inverse Document Frequency (TFIDF) Vectorizer. This technique converts text data into numerical feature vectors, emphasizing the importance of words based on their frequency across documents but penalized if they appear too commonly across all documents. This helps in highlighting words that are uniquely significant to certain types of discussions in the issue reports. We set max features to 10000 to limit the number of most significant words considered across the documents. This constraint is designed to prevent the model from being overwhelmed with data

4 14 ISUM: AUTOMATIC SUMMARIZATION OF INFORMATION TYPES IN ISSUE REPORTS

Table 4: Conversational Features Used in Our Study (CL = Collaborator, norm = normalized) Category

Participant

Feature Name

Description

Value Range

AA

Author’s association with the repository

BEGAUTH

Indicates if the author also posted the original issue

{Owner, CL, Member, Other} {True, False}

LEN

Total character count of the sentence Ratio of sentence length to the longest sentence in the thread Ratio of sentence length to the longest sentence in the comment

Positive Numbers (0, 1]

Position of sentence within its comment Position of sentence within the entire conversation Indicates if the sentence is in the first comment Indicates if the sentence is in the last comment

(0, 1]

Time from the start of the conversation to the comment, norm. Time from the comment to the end of the conversation, norm Time from the previous comment to the current comment, norm Time from the current comment to the next comment, norm

[0, 1]

Indicates whether the comment contains a code snippet

{True, False}

Length TLEN CLEN TLOC Structural

CLOC FIRST TURN LAST TURN TPOS1

Temporal

TPOS2 PPAU

NPAU Code

HAS CODE

(0, 1]

(0, 1] {True, False} {True, False}

[0, 1] [0, 1]

[0, 1]

and to focus on the most informative features, enhancing both efficiency and effectiveness. Conversational Features. To capture the context within which each sentence appears in the discussion, we incorporated a set of conversational features outlined in [10], as listed in Table 4: 1. Participant Features: Including the author’s association (AA) with the repository and whether the author of the comment also initiated the issue (BEGAUTH). 2. Length Features: Such as the total length of the sentence (LEN), the relative length compared to the longest sentence in the thread (TLEN), and within the same comment (CLEN). 3. Structural Features: These include the location of the sentence within the comment (TLOC) and within the entire conversation (CLOC), along with

4.3

Summarization of Information Types

15

flags indicating whether a sentence appears in the first (FIRST TURN) or last comment (LAST TURN). 4. Temporal Features: Measure the timing of a comment within the conversation, such as the time elapsed since the start of the discussion (TPOS1), time until the end of the discussion (TPOS2), and the time intervals between adjacent comments (PPAU and NPAU). 5. Code Feature: A binary flag (HAS CODE) indicating the presence of a code snippet within a comment, which can be crucial for understanding the technical nature of a discussion. 4.3 Summarization of Information Types The iSum summarizer assists the developers/managers with a suite of summarization of the detected information types in issue reports. The summaries can be of three types, as described below: – Visuals. We show two types of visuals as follows. 1. Distribution. We show the overall distribution of information types at sentence in an issue report and across issue reports. This is suitable to get a quick overview of the type of discussions happening in an issue or in a project. 2. Trends. We show the distribution of information types in a project. This is suitable to understand whether certain types of discussions are more prevalent in recent days/months compared to before. – Text Summaries. We show a summary of all textual comments for a given information type. The summary is created using the TextRank algorithm [54]. This approach enables a quick overview of comments related to a specific information type, as TextRank is a lightweight, unsupervised extractive method that requires no additional training data and performs reliably on short, domain-specific software engineering texts such as issue comments. – Text Highlighter. We highlight all comments for a given information type, so that users can check those without reading the entire contents of the issue report. At this moment, iSum offers two types of dashboards to showcase the summaries for the stakeholders. – Developer View corresponds to the summaries per issue. – Manager View corresponds to the summaries across all issues. Figure 4 shows screenshots of the two dashboards in iSum UI (top: Developer, bottom: Manager View). The views are discussed below. 4.3.1 Developer View In any issue page, it displays the distribution of information types present in the issue discussion in a pie or bar chart depending on the default settings

4 16 ISUM: AUTOMATIC SUMMARIZATION OF INFORMATION TYPES IN ISSUE REPORTS

Fig. 4: The iSum dashboards to summarize information types in developer and manager views

4.4

iSum Deployment and Industrial Adoption

17

of the tool. Tooltip of each slice of the pie chart shows the total number of information type along with name of the information type. Legend maps between the color and name of information type and the current percentage of the distribution. Clicking on any slice of the chart highlights the sentences of the information type and shows the top 5 sentences (if any) as the summary in a textarea below. These top 5 sentences are selected by the TextRank algorithm according to their centrality, that is, the sentences most representative of all sentences of that type, rather than the first 5 or the 5 most recent sentences. This tool also allows users to personalize the visualization. Users can select which information types are displayed in the charts, and the same color is used consistently for both the highlighted sentences and the corresponding chart slice on the GitHub page. Users can also save their preferred settings so that the selected information types persist across sessions. 4.3.2 Manager View At the start page of the saved project repository, iSum displays two charts: (1) the information type distribution over all issue discussions of that project as pie or bar chart and (2) the evolution of the prevalence of information types over the years from beginning of the project to current date with multiple line charts; each corresponds for each information type. Single legend will map between the color and the information type of both charts. Users are also able to filter issue to display the data about all, open or closed issues using the button displayed.

4.4 iSum Deployment and Industrial Adoption We built iSum as a browser extension tool. The tool is activated when a user goes to a GitHub project that contains issue reports. As such, iSum can offer the two dashboards within a GitHub page: the developer view at the per issue level of a project and the manager view at the main page of the project in GitHub. To build this extension tool, we have exploited a simple client-server architecture that uses a relational database for storing issue reports and is developed on top of the Python Django Web Framework. The communication between Django webserver and iSum report processor is handled via Rest-based middleware to transfer data in JSON format over HTTP protocol. We do not need an website for output visualization since the UI elements are added on the GitHub issue/repository page. Figure 5 exhibits the client-server architecture of iSum. There are three major components of the iSum: 1) Report Preprocessor: Handles processing of the issue reports and hosts all the techniques to summarize issue discussion information types. 2) Database: A data storage component for storing issue discussion sentences with its metadata. 3) REST Web Server: The middleware between the Report Preprocessor and the repository. Since we follow client server architecture for iSum deployment,

18

5

PERFORMANCE OF INFORMATION TYPE CLASSIFICATION IN ISUM

Report Preprocessor GitHub Repo Information Type Distribution API

Highlighter API

Summary API

Trend Chart API

Fig. 5: The client server architecture of iSum the deployment only needs a server for handling preprocessing of the data and hosting algorithms and techniques for issue and project summarization. Any OSS user or any organization that are involved on open-source industrial projects can easily deploy the tool in a local server or a server owned by them. 5 Performance of Information Type Classification in iSum Arya et al. [10] utilized traditional shallow ML models like random forest and logistic regression for classifying information types in issue discussions. However, there is scope for further enhancement in accuracy and efficiency by employing more advanced models and techniques. Enhanced models leveraging advanced ML techniques such as ensemble methods could provide significant improvements. Ensemble methods combine predictions from multiple models to improve accuracy and reliability over individual models. Given the complex and varied nature of issue report data, ensemble models can better capture the nuances and reduce the likelihood of misclassification. In this section, we answer the following two research questions about the development of these models for information type classification in issue reports and evaluate their performance with two benchmark datasets. RQ1. How do the stand-alone models perform in iSum? RQ2. Can we improve it based on an ensemble of models? 5.1 Studied Datasets We used two benchmark datasets. First, we used the benchmark dataset previously developed by Arya et al. [10]. We refer to this benchmark as Dataset 1. The models developed by Arya et al. [10] are our baselines against which we check our new models. The dataset did not have any contents from the issues of language runtime systems, the software our industrial partner is working on. As such, we also produced another benchmark by analyzing the issues in the runtime systems (OpenJ9). The code and issues of runtime systems

5.1

Studied Datasets

19

Table 5: Dataset used in developed ML models (#Sen = # of Sentences, #Type = # of Information Types) Dataset

Source

Dataset 1 [10] Dataset 2

TensorFlow, scikit-learn, Spacy OpenJ9

#Sen

#Type

4330 4241

13 12

are open source and available on iSum GitHub repository. As such, this new benchmark can also be used by the researchers to develop new models for information types. We refer to this benchmark as Dataset 2. In both datasets, a GitHub issue contains a thread—the chronological sequence of comments posted on that issue. Each comment is written by a single participant and may contain one or more sentences; the sentence is the unit of classification in our work. We use conversation and thread synonymously to refer to the full sequence of comments within a single issue.An issue has multiple threads, each consist- ing of one or multiple comments, each consisting of one or multiple sentences, Dataset 1. We reused the dataset collected by Arya et al. [10]. It consists of 15 complex issues, taken from three Artificial Intelligence (AI) libraries hosted on GitHub, namely, TensorFlow, scikit-learn, and spaCy; 5 issues from each repository. Initially, this dataset consists of total 4656 sentences of 16 types of information. However, sentences which have multiple labels are removed along with the sentences labeled with the information types, which have very few sentences associated with them. Future Plan, Content Management and Testing-Related; these three labels have sentences which covers only 1% of whole dataset and thus removed. So the final dataset has total 4330 sentences containing 13 informaton types. Dataset 2. For this one, we have collected sentences from total 21 closed issues in OpenJ9 GitHub repository. We have chosen this repository since it supports the interest of our industrial partner IBM Canada. The rules set for the selection of issues are similar to the previous work [10]. Closed issues are chosen to get the full discussion flow of an issue, although these information types are applicable for open issues too. From all the closed issues of OpenJ9, the ones with the most comments up to June 28, 2022 are chosen for preparing this dataset. This dataset has been labeled by the first author of this manuscript, strictly adhering to the label definitions established by Arya et al. [10], without any modifications. For this dataset also, labels with few sentences are removed along with the sentences with multilabels. The final dataset contains 4241 sentences of 12 information types, leaving out sentences of the following four labels: Future Plan, Potential New Issues and Requests, Content Management, and Solution Usage. Dataset-level statistics of the selected issue reports from TensorFlow, scikitlearn, spaCy, and OpenJ9 are summarized in Table 6. For each project, we report: (i) the total, mean, and variance of the number of sentences per thread

20

5

PERFORMANCE OF INFORMATION TYPE CLASSIFICATION IN ISUM

Table 6: Statistics of studied datasets for benchmark projects Dataset

Project Total

Sentences Mean Var

Comments Total Mean Var

Dataset 1

Scikit-learn SpaCy TensorFlow

1401 829 2100

280 166 420

1759 6587 21165

429 173 751

86 35 150

338 148 3092

Dataset 2

OpenJ9

4241

202

4194

1599

76

612

(i.e., thread length in sentences); and (ii) the total, mean, and variance of the number of comments in the dataset.

5.2 Model Validation and Evaluation 5.2.1 Validation Approach We used a stratified k-fold cross-validation (k = 10) on a given benchmark to train and test each model we studied. Given the relatively small size of our dataset, we chose this validation approach since this method is particularly advantageous for small datasets, as it maximizes the use of available data by allowing each data point to be used for both training and testing. It also reduces bias and variance by evaluating the model across multiple train–test splits and averaging the results, making the performance estimate less dependent on any single data partition. With a single holdout split, the estimate of performance depends heavily on how that one split happened to partition the data: if the test set is unrepresentative (for example, missing some rare information types), the estimate can be both biased and high-variance. In 10fold cross-validation, each data point serves as test data exactly once and as training data nine times, so every region of the input space contributes to both training and evaluation, which tends to lower bias. Averaging the performance over 10 folds also smooths out randomness from any one unlucky split, which reduces variance in the estimate. Empirically, using around 10 folds is often a good trade-off: fewer folds (like 2–3) can give higher bias because the training sets are small, while very many folds (like leave-one-out) can increase variance because each test set is extremely small.

5.2.2 Metrics Used We measured three metrics to report the performance of each model, Precision, Recall, and F1-score. Precision (P ) is the ratio of correctly predicted examples to all predicted examples for a given class. Recall (R) is the ratio of the number of correctly predicted examples and all examples of a given class. F1-score (F 1) is the harmonic mean of precision and recall.

5.3

Performance Analysis of Stand-alone Models (RQ1)

P =

21

TP P ∗R TP , R= , F 1 = 2x TP + FP TP + FN P +R

TP = Correctly classified as an information type, FP = Incorrectly classified as a type, TN = Correctly classified as not a type, FN = Incorrectly classified as not an information type. We report these metrics using three averaging strategies: macro average, which computes the metric for each class independently and takes the unweighted mean, treating all information types equally regardless of their frequency; micro average, which computes metrics globally across all classes and is equivalent to overall accuracy in multi-class classification and weighted average, which weights each class by its support (number of instances), reflecting overall model performance on the dataset.

5.3 Performance Analysis of Stand-alone Models (RQ1) 5.3.1 Motivation. As part of our study on shallow ML models, we first adopted the best-performing model identified by Arya et al. [10] as our baseline classifier. Additionally, we explored another more advanced shallow ML model, LightGBM, to further enhance our ensemble classifier. We chose this model to take advantage of its highly efficient gradient-boosting decision tree architecture. It handles sparse data naturally and is faster with lower memory usage compared to other gradient boosting models. Given its high performance on structured data and its ability to handle feature interactions effectively, LightGBM was chosen for its efficiency in training and predicting, which is particularly beneficial in scenarios with a limited amount of data [38]. We also focused on utilizing pretrained transformer models (PTM) such as BERT [17] and RoBERTa [51] to build deep ML classifiers, primarily due to their remarkable performance in natural language processing (NLP) tasks. Unlike shallow ML models, which often struggle to capture complex linguistic patterns and dependencies in text data, transformer models excel at understanding contextual information and capturing semantic nuances, thanks to their attention mechanism and deep architecture [79]. 5.3.2 Approach. Here, we detail the development and optimization of the individual ML models—LightGBM, BERT, RoBERTa—and the baseline model for classifying information types in issue reports. BERT and RoBERTa perform deterministically with fixed seeds (no sampling) while Random Forest and LightGBM are nondeterministic due to bagging/subsampling. All models are trained on individual sentences, with each sentence treated as a separate example.

22

5

PERFORMANCE OF INFORMATION TYPE CLASSIFICATION IN ISUM

Random Forest (Baseline). Arya et al. [10] explored a variety of configurations and feature sets to optimize the classification of information types in issue discussions. They implemented different balancing techniques, such as adjusting class weights or employing SMOTE (Synthetic Minority Over-Sampling Technique) to address class imbalances, alongside experimenting with textual and conversational features to comprehensively capture the semantics and contextual metadata of discussions. They primarily utilized Random Forest and Logistic Regression, which are commonly favored for text classification tasks due to their robustness and effectiveness. However, they excluded other classifiers like Support Vector Machines and Naive Bayes after preliminary testing indicated inferior performance and longer training times. Based on their findings, we adopted their best-performing model setup—Random Forest using class weight adjustments and conversational features—as our baseline. For this classifier, they considered following hyperparameters in their optimization process: 1. min samples split: The minimum number of data points required to split a node. The values explored included 2, 5, and 10, aiming to determine the best threshold for creating new branches in the decision trees. 2. n estimators: The number of trees in the forest. By testing different quantities, such as 10, 50, and 100, they aimed to balance the model’s complexity with its ability to generalize across different types of data. LightGBM (Light Gradient-Boosting Machine). For this model, we represent the sentence text using TF-IDF features (top 10,000 terms) and concatenate them with conversational features to form the final input to the model. TF-IDF is chosen because it performs well on small, high-dimensional text datasets and captures sparse lexical patterns that complement the semantic information learned by BERT/RoBERTa in the ensemble. To optimize the LightGBM model, we used Optuna [4], an automatic hyperparameter tuning tool that employs Bayesian optimization to efficiently determine the best model parameters. Optuna was tasked with tuning the parameters listed in Table 7, while we fixed parameters such as objective to multiclass and metric to multi logloss for this multi-class classification problem. Additionally, we addressed class imbalance by setting the class weight parameter to balanced, ensuring that the model pays more attention to underrepresented classes. We also set gbdt (standard Gradient Boosting Decision Tree) as the boosting type. In addition to the above parameters, several other hyperparameters are also important in our setting because we train on relatively small, high-dimensional, and imbalanced text datasets. The feature fraction and bagging fraction parameters regularize the model when using up to 10,000 TF-IDF features, reducing overfitting and improving training speed by subsampling features and instances at each iteration. Furthermore, num leaves, min child samples, and lambda l1/l2 jointly control tree complexity and weight magnitudes, which is crucial to avoid overfitting given the limited number of labeled sentences in both datasets. We also enable feature pre filter to discard unused features early and tune bagging fraction for stochastic subsampling, further preventing overfitting. The model training architecture is shown in Figure 6.

5.3

Performance Analysis of Stand-alone Models (RQ1)

23

Feature Generator Input

Text Preprocessor

TF-IDF Vectorization

Metadata Extraction

Hyperparameter Tuner

Trainer & Evaluator

Output

Fig. 6: LightGBM training architecture Table 7: HyperParameters for LightGBM classifiers HyperParameter

Definition

num classes objective

Defines the number of target classes Specifies the learning task and the corresponding learning objective Performance assessment metric to be minimized or maximized during training The type of algorithm to use for boosting Adjusts the weight of each class contribution to the objective

metric boosting type class weight feature pre filter lambda l1 lambda l2 feature fraction bagging fraction bagging freq min child samples num leaves

Setting multiclass multi logloss gbdt balanced

Whether to pre-filter features before building the dataset L1 regularization : effective in feature selection by shrinking some coefficients to zero L2 regularization : effective at combating the model’s overfitting The fraction of features to be used at each iteration The fraction of the data to be used for each iteration Frequency for performing bagging at every N iterations Minimum number of data points needed in a leaf The maximum number of leaves in one tree

BERT (Bidirectional Encoder Representations from Transformers). BERT [17] is a powerful pre-trained model that uses transformer architecture to process words in relation to all other words in a sentence rather than one-by-one in order. This allows BERT to understand the context of a word based on all of its surroundings, which is a significant advantage for classifying information types in textual data where the context in which words appear is crucial. The bidirectional nature of BERT makes it adept at understanding the nuances and complexities of language in issue reports. For instance, distinguishing between Motivation and Expected Behaviour often requires examining the context following the key phrase. Consider the examples: “I expect it to fail because the input is invalid,” which corresponds to Expected Behaviour, and “We should support this use case because users often need . . . ,” which reflects Motivation. In these cases, phrases such as “because the input is invalid” and “because users often need” provide crucial contextual cues that appear later in the sentence. A bidirectional encoder can attend to both the preceding and following context simultaneously, enabling it to better disambiguate between these labels.

24

5

PERFORMANCE OF INFORMATION TYPE CLASSIFICATION IN ISUM

Table 8: Architecture details of BERT and RoBERTa model (L = Layers, H = Hidden, D = Heads, P = Parameters) Architecture

Pre-trained Checkpoint

L

H

D

P

BERT RoBERTa

bert-base-uncased roberta-base

12 12

768 768

12 12

110M 125M

Table 9: Stand-alone models and inputs Model

Input

Random Forest (Baseline) LightGBM BERT ROBERTA

Conversational Features Textual + Conversational Features Preprocessed Text Preprocessed Text

RoBERTa (Robustly Optimized BERT Approach). RoBERTa [51] builds on BERT by modifying key hyperparameters, removing the next-sentence pretraining objective and training with much larger mini-batches and learning rates. This results in an improved model that outperforms BERT on many benchmarking tests. We chose RoBERTa because it offers all the benefits of BERT but with refinements that increase its accuracy and robustness in understanding textual data. The comparison between architecture details of BERT and RoBERTa Model is presented in Table 8. As input to our deep ML models, we utilized preprocessed text (see Table 9), which was tokenized using either the bert-base-uncased or roberta-based tokenizer, depending on the selected model variant. The text preprocessing details are provided in Section 4.2.1. Additionally, we generated a validation set by extracting 5% of the training data. To address class imbalances, we computed class weights based on the frequency of each label category within N the training data as Weightc = K·n , where N is the total number of training c examples, K is the number of classes, and nc is the number of examples in class c. This corresponds to class weight = balanced in our shallow classifiers. These weights were then incorporated into the model training process to mitigate bias towards the majority class. Our model training pipeline utilized a custom trainer class, which computed the loss function while considering the assigned class weights during the forward pass. The hyperparameters for fine-tuning were selected based on recommendations from the original BERT [17] and RoBERTa [51] papers and further optimized via grid search over learning rate {1e-5, 2e-5, 3e-5} and batch size {8, 16}. This process yielded a final configuration of learning rate = 2e-5 and batch size = 16. The final layer is a dense linear classification head on the [CLS] embedding, producing a dense logit vector with one value per class (13 for Dataset 1, 12 for Dataset 2). Softmax converts the logits into class probabilities, and the predicted label is the argmax. The training architectures of these models are shown in Figure 7.

5.3

Performance Analysis of Stand-alone Models (RQ1)

Input

Text Preprocessor

BERT / RoBERTa Tokenizer

Hyperparameter Tuner

25

Class Weight Calculator

Trainer & Evaluator

Output

Fig. 7: BERT/RoBERTa fine-tuning architecture Table 10: Performance of stand-alone ML models on dataset 1. Dataset 1 Information Type

Baseline (RF) Pr Rc F1

LightGBM Pr Rc F1

Pr

BERT Rc F1

RoBERTa Pr Rc F1

Sup

Action on Issue Bug Reproduction Contribution & Commit. Expected Behaviour Investigation and Explr. Motivation Observed Bug Behav. Potential New Issues. Social Conversation Solution Discussion Task Progress Usage Workarounds

0.45 0.39 0.53 0.47

0.42 0.5

0.9 0.46 0.6 0.68 0.62 0.65

0.76 0.67 0.69 0.51 0.5 0.5

0.7 0.7 0.69 0.49 0.59 0.53

61 245

0.25 0.19

0.21

0.56 0.19

0.58 0.48 0.51

0.53 0.52 0.51

83

0.42 0.28

0.33

0.59 0.41 0.46

0.34 0.32

0.33

0.42 0.42

0.42

124

0.6

0.62

0.68 0.65 0.66

0.43 0.47

0.45

0.46 0.5

0.48

377

0.56 0.53 0.54 0.56 0.7 0.62

0.63 0.47 0.53 0.75 0.69 0.69

0.31 0.36 0.4 0.27

0.34 0.32

0.38 0.41 0.33 0.28

0.39 0.3

288 131

0.67 0.66

0.66

0.74 0.68

0.7

0.23 0.23

0.23

0.26 0.21

0.23

230

0.63 0.62

0.63

0.71 0.71

0.71

0.82 0.78

0.8

0.84 0.81 0.83

798

0.68 0.71

0.69

0.67 0.82 0.74

0.61 0.58

0.59

0.63 0.55

0.59

1411

0.27 0.14 0.65 0.82 0.58 0.45

0.18 0.73 0.49

0.5 0.16 0.23 0.75 0.8 0.77 0.84 0.44 0.56

0.47 0.43 0.49 0.55 0.31 0.31

0.44 0.51 0.31

0.5 0.5 0.48 0.46 0.6 0.52 0.33 0.38 0.35

125 368 89

Micro Average Macro Average Weighted Average

0.64 0.64 0.56 0.54 0.61 0.62

0.64 0.54 0.61

0.68 0.68 0.68 0.69 0.54 0.58 0.68 0.68 0.67

0.54 0.54 0.48 0.46 0.55 0.54

0.54 0.46 0.54

0.56 0.56 0.49 0.5 0.57 0.56

4330 4330 4330

0.65

0.26

0.56 0.48 0.56

5.3.3 Results In Tables 10 and 11, we detail the performance metrics—precision, recall, and F1-score—for each information type and overall, comparing stand-alone ML models across two datasets. Tables 12 and 13 report the performance of the same models on each fold of datasets 1 and 2. Notably, LightGBM surpasses the baseline model (Random Forest) [10], achieving the highest F1 score in Dataset 1 at 0.67, showcasing the strength of gradient boosting techniques in processing feature-dense textual data. This surpasses the baseline’s F1 score of 0.61. BERT and RoBERTa, however, scored lower with F1 scores of 0.54 and 0.56, respectively. In Dataset 2, all models improved upon the baseline, with scores ranging between 0.61-0.63, suggesting that dataset 2 benefits more from text embeddings, with LightGBM again leading. This variation in performance between the datasets suggests that they differ significantly in content and structure. Dataset 2, unlike Dataset 1, appears to benefit more from text embeddings, likely due to its content having more specialized vocabulary or unique linguistic patterns. These embeddings can grasp the subtle meanings of words within the specific context of the discussions, providing an advantage in understanding and classifying the information types more accurately. The F1 scores for LightGBM, BERT, and RoBERTa across various labels provide insightful reflections on each model’s handling of different types

26

5

PERFORMANCE OF INFORMATION TYPE CLASSIFICATION IN ISUM

Table 11: Performance of stand-alone ML models on dataset 2 Dataset 2 Information Type

Baseline (RF) Pr Rc F1

LightGBM Pr Rc F1

Pr

BERT Rc F1

RoBERTa Pr Rc F1

Sup

Action on Issue Bug Reproduction Contribution & Commit. Expected Behaviour Investigation and Explr. Motivation Observed Bug Behav Social Conversation Solution Discussion Task Progress Testing Workarounds

0.28 0.2 0.23 0.4 0.18 0.24

0.66 0.44 0.65 0.4

0.51 0.49

0.7 0.75 0.72 0.58 0.53 0.54

0.66 0.79 0.72 0.56 0.57 0.57

96 176

0.23 0.11 0.15

0.73 0.36

0.47

0.68 0.59

0.6

0.67 0.65 0.64

109

0.33 0.21 0.25

0.12 0.08

0.1

0.24 0.17

0.19

0.43 0.33 0.32

24

0.63 0.83 0.71

0.65 0.89 0.75

0.66 0.68

0.67

0.69 0.67

0.68

1891

0.56 0.44 0.46 0.37 0.3 0.31

0.83 0.56 0.65 0.48 0.27 0.34

0.28 0.22 0.23 0.23

0.24 0.23

0.17 0.11 0.28 0.31

0.13 0.29

18 81

0.43 0.37

0.4

0.62 0.44

0.52

0.66 0.56

0.6

0.68 0.58 0.62

469

0.62 0.58

0.6

0.7

0.6

0.64

0.54 0.58

0.56

0.54 0.6

0.57

856

0.33 0.16 0.21 0.57 0.31 0.39 0.31 0.18 0.23

0.43 0.16 0.68 0.51 0.72 0.59

0.23 0.58 0.62

0.44 0.43 0.65 0.58 0.64 0.59

0.43 0.61 0.6

0.46 0.48 0.47 0.64 0.58 0.6 0.71 0.7 0.67

233 261 27

Micro Average Macro Average Weighted Average

0.58 0.58 0.58 0.42 0.32 0.34 0.55 0.58 0.55

0.65 0.65 0.65 0.61 0.44 0.49 0.64 0.65 0.63

0.61 0.61 0.52 0.49 0.61 0.61

0.61 0.5 0.6

0.62 0.62 0.53 0.52 0.63 0.62

4241 4241 4241

0.62 0.52 0.62

of information in issue discussions. BERT and RoBERTa, known for their deep learning capabilities and handling of natural language through contextual embeddings, show better performance in Action on Issue, Contribution and Commitment, Social Conversation, and Task Progress in both datasets. These categories benefit from the models’ ability to understand natural language. However, there are some cases where text-based models get confused due to the vocabulary ambiguity. ’This issue should not be labeled as a stopship/blocker issue for 0.29.1 due to the presence of a workaround’ ; this sentence gets labeled as Workarounds though it should be of Action on Issue. Because of the word workaround, the text-based models got confused. On the other hand, shallow ML models typically achieve better performance on Investigation and Exploration, Motivation, Observed Bug Behaviour and Solution Discussion in both datasets, indicating their potential in leveraging conversational features like code snippets, the length of the discussion, the position of sentences within the conversation and others. For example, while training, Observed Bug Behaviour are mostly found on the start of the conversation. These models learned to give more importance to the positions of the sentences in the thread when labeling this class. Also, Investigation and Exploration, Solution Discussion and Testing give importance to the existence of code. Again, Solution Discussion related sentences are much longer than Social conversation and Contribution and Commitment related sentences. Investigation and Exploration comments generally come from the authors who are contributors to the repository. However, these models struggle with Contribution and Commitment and Task Progress, where they may not have sufficient distinctive textual or conversational cues to differentiate between categories effectively. This also indicates a limitation in the model’s ability to capture the more subtle nuances of language that could be critical for these information types.

5.3

Performance Analysis of Stand-alone Models (RQ1)

27

Table 12: Performance of stand-alone ML models on each fold of dataset 1. Fold 1 2 3 4 5 6 7 8 9 10

Baseline (RF) Pr Rc F1 0.60 0.62 0.61 0.63 0.63 0.62 0.64 0.65 0.64 0.64 0.64 0.63 0.63 0.65 0.64 0.62 0.62 0.62 0.66 0.67 0.66 0.60 0.61 0.60 0.65 0.64 0.64 0.64 0.64 0.64

Pr 0.69 0.68 0.67 0.68 0.68 0.67 0.76 0.64 0.69 0.68

LightGBM Rc F1 0.69 0.68 0.69 0.67 0.67 0.66 0.68 0.67 0.68 0.67 0.66 0.65 0.73 0.72 0.64 0.63 0.70 0.69 0.67 0.66

Mean Variance

0.63 0.64 0.63 0.0003 0.0003 0.0003

0.68 0.68 0.67 0.0008 0.0005 0.0005

Pr 0.53 0.52 0.55 0.55 0.57 0.54 0.55 0.55 0.59 0.57

BERT Rc 0.53 0.50 0.53 0.55 0.55 0.55 0.54 0.53 0.58 0.58

F1 0.53 0.50 0.54 0.55 0.55 0.54 0.54 0.53 0.58 0.57

0.55 0.54 0.54 0.0004 0.0006 0.0004

Pr 0.56 0.55 0.59 0.60 0.55 0.55 0.57 0.58 0.59 0.57

RoBERTa Rc F1 0.56 0.55 0.55 0.55 0.57 0.57 0.55 0.56 0.54 0.54 0.55 0.55 0.56 0.56 0.57 0.57 0.58 0.58 0.57 0.57

0.57 0.56 0.56 0.0003 0.0002 0.0001

Table 13: Performance of stand-alone ML models on each fold of dataset 2. Fold 1 2 3 4 5 6 7 8 9 10

Baseline (RF) Pr Rc F1 0.54 0.57 0.54 0.51 0.55 0.52 0.56 0.58 0.56 0.56 0.57 0.55 0.53 0.59 0.55 0.57 0.59 0.57 0.54 0.58 0.54 0.58 0.59 0.56 0.55 0.59 0.56 0.54 0.57 0.54

Pr 0.64 0.65 0.61 0.68 0.62 0.69 0.62 0.65 0.65 0.63

LightGBM Rc F1 0.64 0.62 0.67 0.64 0.63 0.60 0.68 0.66 0.65 0.62 0.68 0.66 0.64 0.61 0.65 0.63 0.66 0.64 0.64 0.61

Mean Variance

0.548 0.578 0.549 0.0004 0.0002 0.0002

0.644 0.654 0.629 0.0006 0.0003 0.0004

Pr 0.61 0.60 0.56 0.63 0.58 0.64 0.61 0.64 0.59 0.62

BERT Rc 0.61 0.59 0.54 0.64 0.58 0.64 0.60 0.64 0.59 0.61

F1 0.60 0.59 0.54 0.63 0.58 0.64 0.61 0.64 0.59 0.61

0.61 0.61 0.60 0.0007 0.0009 0.0008

Pr 0.60 0.60 0.58 0.67 0.62 0.64 0.66 0.66 0.65 0.59

RoBERTa Rc F1 0.60 0.59 0.60 0.60 0.57 0.57 0.66 0.66 0.61 0.61 0.63 0.63 0.65 0.65 0.65 0.65 0.64 0.64 0.59 0.59

0.63 0.62 0.62 0.0009 0.0008 0.0009

In all cases, LightGBM outperforms Random Forest due to its efficient gradient boosting method, which corrects previous trees’ errors, handles sparse data effectively and uses textual features. Again, RoBERTa shows better performance than BERT due to its improved pre-training regime, including training with larger mini-batches and removing the next-sentence prediction task, which enables it to capture linguistic nuances more effectively. Our fine-tuning setup for BERT and RoBERTa remains largely comparable, using the same weighted cross-entropy objective with only minor differences in learning rate. We further examine recall to understand how well each model identifies critical information types. On Dataset 1, LightGBM achieves strong recall for Solution Discussion (0.82) and Observed Bug Behaviour (0.69), demonstrating effective detection of these essential categories. Bug Reproduction recall is moderate at 0.62, and Workarounds recall at 0.44 reflects the challenge of detecting this relatively rare category (support=89). Task Progress recall varies across models (0.14–0.50), consistent with its small support and subtle linguistic cues. On Dataset 2, recall for several types decreases slightly, with Bug Reproduction at 0.40 for LightGBM, Observed Bug Behaviour at 0.27, and Motivation at 0.56, reflecting the added challenge of the specialized OpenJ9 vocabulary - a pattern consistent across all models on this dataset. 5.3.4 Misclassification Analysis Table 14 presents how misclassifications by one stand-alone model can be corrected by other models, with the last column (≥ 1) indicating how the

28

5

PERFORMANCE OF INFORMATION TYPE CLASSIFICATION IN ISUM

Table 14: How the misclassification of a classifier can be corrected by one or at least one classifier Stand-alone Models

RF

LGBM

BERT

RoBERTa

≥1

47% 36%

49% 39% 28%

68% 64% 75% 74%

54% 43% 30%

69% 62% 66% 65%

Dataset 1 Random Forest LightGBM BERT RoBERTa

34% 25% 58% 58%

56% 56%

26%

Dataset 2 Random Forest LightGBM BERT RoBERTa

35% 21% 47% 49%

46% 48%

50% 39% 28%

misclassification of a model can be corrected by the correct classification of at least one of the models. This analysis underlines the potential benefits of using an ensemble approach where the weaknesses of one model are compensated by the strengths of others. For dataset 1, the data shows that the misclassifications made by BERT and RoBERTa are more frequently corrected by other models, with both achieving correction rates of over 70% when at least one other model is considered. This suggests that while BERT and RoBERTa might be prone to specific errors, their weaknesses are effectively covered by the capabilities of the other models, particularly Random Forest and LightGBM. Random Forest and LightGBM themselves show a moderate level of error correction by others, indicating a more balanced level of dependency across models. For dataset 2, the correction rates are generally lower, which might be indicative of more challenging or ambiguous data that even complementary models struggle to correct. However, for both datasets, all models still benefit from corrections by others, with rates over 60%. This reflects the ongoing relevance of an ensemble approach, especially in complex datasets where no single model is perfect. Overall, these results justify the use of an ensemble approach in scenarios where diverse models can leverage their unique strengths, leading to improved overall performance by filling in each other’s gaps. Summary of RQ1. Among the stand-alone models, in both datasets, LightGBM achieves the highest F1-score for information type detection in both datasets, surpassing SOTA model [10] by 6-8%.

5.4 Performance Analysis of Ensemble Models (RQ2) 5.4.1 Motivation. To further enhance the performance of the overall issue discussion information type classification, we thought of leveraging distinct strengths and mitigating

5.4

Performance Analysis of Ensemble Models (RQ2)

29

individual weaknesses of each stand-alone model. Also, Table 14 shows how the misclassification of a classifier can be corrected by one or at least one classifier. Therefore, we used these four models - Random Forest [10], LightGBM, BERT, RoBERTa, to make an ensemble classifier for information type detection. This approach is suitable when the development environment can deploy multiple models that form an ensemble with the hope of producing better performance than the stand-alone models. The models are chosen because no single one sees all the signal in our data, and each fills a gap the others leave open. Random Forest and LightGBM work from explicit, engineered features and combine trees differently (bagging vs. boosting), giving each a different bias-variance profile and access to different features: Random Forest is robust to noisy, heterogeneous, low-dimensional metadata without assuming linearity [13], while LightGBM’s histogram-based leaf-wise boosting scales to the high dimensional sparse TFIDF features and captures non-linear feature interactions, with built-in L1/L2 regularization (Table 7) controlling overfitting on our limited labeled data [21, 38]. Gradient-boosted trees also remain competitive with deep models on small, high-dimensional sparse problems like ours [23]. BERT [17] and RoBERTa [51] see the raw sentence text directly and learn context-dependent meaning through self-attention, rather than relying on hand-engineered features. This is what lets them catch cases the tree models miss: two sentences with identical surface vocabulary but different meaning depending on context (e.g., our Motivation vs. Expected Behaviour example). Conversely, the tree models catch what the transformers miss, namely conversational and structural cues like turn position or author role that never enter the transformers’ input at all. We include both BERT and RoBERTa because RoBERTa differs from BERT only in pretraining regime, not architecture [51], which is enough on its own to produce decorrelated errors, since ensembling benefits from base learners disagreeing rather than from architectural novelty [18,45,83]. Our per-class results (Tables 10 - 11) confirm this empirically: BERT and RoBERTa are strongest on different information types, and no model dominates across all types, which is the practical evidence that the ensemble’s complementary design pays off. 5.4.2 Approach. To construct the ensemble classifier, the predicted labels from four distinct classifiers are combined with textual and conversational features from the dataset to feed into a ML classifier. This classifier then generates the final prediction. We use hard predicted labels from stand-alone models rather than their predicted probabilities, to keep the ensemble simple and reduce overfitting risks on our ∼4k sentences. Passing soft probabilities can propagate correlated errors from similar base models and amplify rather than diversify their predictions. We perform one 10-fold cross-validation per dataset. There is no separate, second cross-validation for the ensemble. In each fold, base

30

5

PERFORMANCE OF INFORMATION TYPE CLASSIFICATION IN ISUM

Text

BERT

Text

RoBERTa

Metadata

LightGBM

Predicted Labels

Random Forest Metadata

Dataset Metadata

Random Forest

Final Prediction

Fig. 8: The architecture of ensemble classifier

models train on the other nine folds. They predict only on the held-out fold. So their predictions feeding the ensemble are always out-of-fold. The ensemble is trained and evaluated on this same fold structure. No sentence’s ensemble prediction relies on a base model trained on that sentence. This holds independently for each dataset. The two datasets are never mixed. The structure of this ensemble approach is illustrated in Figure 8. For the final classification step, we have experimented with a variety of classifiers, including Random Forest, LightGBM, BERT, and RoBERTa. The technique of automatic hyperparameter tuning previously detailed in Section 5.3 is applied to the shallow ML models. In contrast, for deep learning models like BERT and RoBERTa, we enhance input integration by prefixing each data point with predicted labels from stand-alone models, formatted as PRED ModelName Label. These models are subsequently fine-tuned with carefully selected hyperparameters to optimize performance with batch size 8 and epoch 10. As learning rate, we chose 5e-5 and 3e-5 for BERT and RoBERTa respectively. 5.4.3 Results Tables 15 and 16 demonstrate the efficacy of ensemble classifiers across two distinct datasets. Tables 17 and 18 report the performance of the same models on each fold of datasets 1 and 2. On both datasets, the best ensemble model surpasses the baseline models and all other stand-alone implementations and establish the capability of ensemble methods complementing any stand alone models. In Dataset 1, the ensemble classifiers generally achieve a weighted average F1-score ranging from 0.70 to 0.72, highlighting their consistent performance across diverse information types. Notably, the LightGBM ensemble stands out by achieving the highest F1-score of 0.72, and it notably excels in seven out of thirteen information categories, indicating its particular strength in extracting and classifying nuanced information effectively. Meanwhile, in Dataset 2, the

5.4

Performance Analysis of Ensemble Models (RQ2)

31

Table 15: Performance of ensemble classifiers on dataset 1. (RF = Random Forest, LG = LightGBM, B = BERT, RB = RoBERTa) Dataset 1 Information Type

Ensemble (RF) Pr Rc F1

Ensemble (LG) Pr Rc F1

Ensemble (B) Pr Rc F1

Ensemble (RB) Pr Rc F1

Sup

Action on Issue Bug Reproduction Contribution & Commit. Expected Behaviour Investigation and Explr. Motivation Observed Bug Behav. Potential New Issues. Social Conversation Solution Discussion Task Progress Usage Workarounds

0.69 0.8 0.64 0.68

0.72 0.65

0.83 0.72 0.72 0.69

0.74 0.7

0.76 0.69 0.68 0.64

0.71 0.66

0.61 0.75 0.64 0.75

0.65 0.69

61 245

0.56 0.41

0.45

0.64 0.41

0.46

0.48 0.47

0.47

0.47 0.65

0.54

83

0.58 0.47

0.51

0.64 0.52

0.57

0.46 0.43

0.43

0.53 0.56

0.54

124

0.67 0.67

0.67

0.7

0.72

0.7

0.68 0.66

0.67

0.63 0.69

0.66

377

0.6 0.61 0.69 0.73

0.6 0.7

0.66 0.56 0.77 0.71

0.6 0.72

0.56 0.58 0.76 0.67

0.57 0.69

0.56 0.61 0.75 0.7

0.58 0.71

288 131

0.73 0.69

0.7

0.72 0.65

0.67

0.67 0.65

0.66

0.66 0.71

0.68

230

0.83 0.74

0.79

0.82 0.79

0.8

0.84 0.78 0.81

0.85 0.77

0.8

798

0.75 0.79

0.77

0.73 0.81

0.77

0.74 0.77

0.76

0.82 0.7

0.75

1411

0.61 0.29 0.74 0.83 0.68 0.6

0.38 0.78 0.62

0.52 0.34 0.74 0.79 0.67 0.52

0.4 0.76 0.58

0.47 0.43 0.73 0.79 0.61 0.56

0.45 0.76 0.58

0.49 0.46 0.73 0.84 0.59 0.69

0.46 0.78 0.62

125 368 89

Micro Average Macro Average Weighted Average

0.72 0.72 0.67 0.64 0.72 0.72

0.72 0.64 0.71

0.73 0.73 0.71 0.63 0.73 0.72

0.73 0.65 0.72

0.71 0.71 0.65 0.62 0.71 0.7

0.71 0.63 0.7

0.71 0.71 0.64 0.68 0.73 0.71

0.71 0.65 0.71

4330 4330 4330

Table 16: Performance of ensemble classifiers on dataset 2. (RF = Random Forest, LG = LightGBM, B = BERT, RB = RoBERTa) Dataset 2 Information Type

Ensemble (RF) Pr Rc F1

Ensemble (LG) Pr Rc F1

Ensemble (B) Pr Rc F1

Ensemble (RB) Pr Rc F1

Sup

Action on Issue Bug Reproduction Contribution & Commit. Expected Behaviour Investigation and Explr. Motivation Observed Bug Behav. Social Conversation Solution Discussion Task Progress Testing Workarounds

0.67 0.79 0.63 0.54

0.72 0.58

0.72 0.78 0.65 0.49

0.74 0.55

0.69 0.77 0.53 0.59

0.71 0.53

0.55 0.82 0.5 0.61

0.65 0.54

96 176

0.71 0.6

0.62

0.76 0.6

0.64

0.61 0.62

0.59

0.54 0.64

0.55

109

0.38 0.17

0.22

0.38 0.17

0.23

0.16 0.25

0.18

0.23 0.37

0.26

24

0.72 0.84

0.77

0.71 0.83

0.76

0.78 0.65

0.71

0.81 0.59

0.68

1891

0.42 0.33 0.39 0.3

0.35 0.33

0.54 0.44 0.49 0.23

0.47 0.3

0.83 0.56 0.65 0.18 0.25 0.2

0.61 0.5 0.21 0.37

0.52 0.26

18 81

0.72 0.53

0.61

0.67 0.56

0.61

0.61 0.62 0.61

0.59 0.62

0.6

469

0.71 0.67

0.69

0.68 0.67

0.68

0.64 0.72

0.68

0.62 0.72

0.66

856

0.53 0.37 0.67 0.6 0.67 0.63

0.43 0.63 0.61

0.5 0.36 0.69 0.58 0.76 0.59

0.41 0.62 0.63

0.4 0.45 0.58 0.65 0.57 0.63

0.42 0.61 0.56

0.45 0.49 0.53 0.67 0.62 0.63

0.46 0.59 0.59

233 261 27

Micro Average Macro Average Weighted Average

0.69 0.69 0.59 0.53 0.69 0.69

0.69 0.54 0.68

0.69 0.69 0.63 0.53 0.68 0.69

0.69 0.55 0.67

0.64 0.64 0.54 0.56 0.67 0.64

0.64 0.53 0.64

0.62 0.62 0.52 0.58 0.66 0.62

0.62 0.53 0.63

4330 4330 4241

weighted average F1-scores for all models hover around 0.64 to 0.68, with Random Forest slightly leading with a 0.68 F1-score, reinforcing the adaptability of this model to different datasets. With respect to recall, the ensemble models show notable improvements over stand-alone models for the most critical information types. On Dataset 1, Ensemble (RF) achieves strong recall for Observed Bug Behaviour (0.73), improving over the best stand-alone LightGBM recall of 0.69, and improves Workarounds recall from 0.44 (standalone LightGBM) to 0.60, meaning fewer workaround comments are missed. Ensemble (LG) achieves the highest recall for Bug Reproduction (0.69), improving over standalone LightGBM (0.62),

32

5

PERFORMANCE OF INFORMATION TYPE CLASSIFICATION IN ISUM

Table 17: Performance of ensemble classifiers on each fold of dataset 1. (RF = Random Forest, LG = LightGBM, B = BERT, RB = RoBERTa) Fold 1 2 3 4 5 6 7 8 9 10

Ensemble RF Pr Rc F1 0.70 0.70 0.70 0.71 0.72 0.71 0.72 0.71 0.71 0.72 0.70 0.70 0.73 0.73 0.72 0.68 0.68 0.67 0.75 0.75 0.75 0.73 0.71 0.71 0.77 0.76 0.75 0.71 0.70 0.70

Ensemble LG Pr Rc F1 0.67 0.67 0.67 0.72 0.72 0.72 0.71 0.69 0.69 0.72 0.71 0.71 0.74 0.73 0.72 0.69 0.68 0.67 0.73 0.71 0.72 0.69 0.68 0.68 0.75 0.75 0.74 0.70 0.69 0.69

Ensemble B Pr Rc F1 0.69 0.69 0.68 0.72 0.71 0.71 0.71 0.71 0.70 0.74 0.72 0.72 0.73 0.72 0.72 0.67 0.67 0.67 0.73 0.73 0.73 0.70 0.69 0.69 0.74 0.73 0.73 0.69 0.68 0.68

Ensemble RB Pr Rc F1 0.73 0.72 0.72 0.73 0.71 0.72 0.72 0.68 0.69 0.73 0.68 0.69 0.72 0.70 0.70 0.71 0.70 0.70 0.76 0.74 0.75 0.72 0.68 0.68 0.76 0.75 0.75 0.72 0.71 0.71

Mean Variance

0.72 0.72 0.71 0.0006 0.0005 0.0005

0.71 0.70 0.70 0.0006 0.0006 0.0005

0.71 0.71 0.70 0.0005 0.0004 0.0004

0.73 0.71 0.71 0.0003 0.0005 0.0005

Table 18: Performance of ensemble classifiers on each fold of dataset 2. (RF = Random Forest, LG = LightGBM, B = BERT, RB = RoBERTa) Fold 1 2 3 4 5 6 7 8 9 10

Ensemble RF Pr Rc F1 0.65 0.66 0.65 0.64 0.67 0.65 0.65 0.65 0.64 0.72 0.72 0.71 0.71 0.71 0.70 0.72 0.73 0.72 0.70 0.71 0.70 0.73 0.71 0.71 0.69 0.69 0.68 0.67 0.67 0.66

Ensemble LG Pr Rc F1 0.65 0.66 0.65 0.66 0.68 0.66 0.66 0.65 0.64 0.73 0.74 0.73 0.68 0.68 0.67 0.67 0.66 0.66 0.69 0.71 0.69 0.69 0.69 0.68 0.70 0.71 0.69 0.68 0.68 0.67

Ensemble B Pr Rc F1 0.63 0.64 0.62 0.65 0.67 0.66 0.63 0.55 0.56 0.71 0.67 0.68 0.66 0.65 0.65 0.73 0.69 0.70 0.69 0.68 0.68 0.65 0.63 0.63 0.66 0.61 0.61 0.65 0.61 0.62

Ensemble RB Pr Rc F1 0.64 0.64 0.64 0.63 0.64 0.63 0.63 0.61 0.62 0.70 0.66 0.67 0.64 0.58 0.59 0.69 0.59 0.60 0.70 0.65 0.66 0.71 0.65 0.66 0.67 0.61 0.61 0.64 0.59 0.60

Mean Variance

0.688 0.692 0.682 0.0010 0.0007 0.0008

0.681 0.686 0.674 0.0005 0.0007 0.0006

0.670 0.640 0.640 0.0010 0.0016 0.0015

0.670 0.620 0.630 0.0009 0.0008 0.0007

and matches Solution Discussion recall (0.81) which is comparable to standalone LightGBM (0.82). On Dataset 2, Ensemble (RF) achieves the highest recall for Action on Issue (0.79) and Investigation and Exploration (0.84), while Ensemble (LG) leads on Investigation and Exploration recall (0.83). Solution Discussion recall improves from 0.60 (standalone LightGBM) to a range of 0.67–0.72 across ensemble models, reflecting consistent detection of solutionrelated discussions. Some information types such as Observed Bug Behaviour, Task Progress, and Expected Behaviour on Dataset 2 show lower recall, which is expected given their relatively small support in the dataset. Overall, ensemble models consistently improve recall for the most critical information types, reducing the risk of missing important information in issue discussions. 5.4.4 Misclassification Analysis Table 19 presents correction rates across various categories for both Dataset 1 and Dataset 2 using ensemble classifiers. Within each category, Each value represents the percentage corrections over the best stand-alone model. Each ensemble classifier shows varying levels of corrections in different categories, reflecting their strengths and weaknesses. For example, in Dataset 1, Contribution and Commitment sees substantial corrections with RoBERTa, achieving a 56.72% improvement, suggesting its effectiveness in understanding nuanced textual data in this category. RoBERTa and BERT often lead in terms of corrections percentages of the stand-alone LightGBM model.

33

Table 19: How the misclassification of the best stand-alone model can be corrected ensemble classifier Information Type

Ensemble (RF)

Action on Issue Bug Reproduction Contribution & Commit. Expected Behaviour Investigation and Explr. Motivation Observed Bug Behav. Potential New Issues. Social Conversation Solution Discussion Task Progress Usage Workarounds

51.52% 22.83% 28.36% 23.29% 20.77% 25.32% 14.63% 14.86% 43.48% 28.52% 25.71% 13.51% 16.00%

Ensemble (LG)

Ensemble (B)

Ensemble (RB)

42.42% 28.26% 35.82% 17.81% 25.38% 31.82% 14.63% 21.62% 45.65% 32.42% 35.24% 22.97% 26.00%

54.55% 41.30% 56.72% 32.88% 26.92% 35.06% 19.51% 24.32% 39.13% 15.63% 40.00% 33.78% 48.00%

64.81% 35.24% 45.71% 27.27% 14.55% 12.50% 5.08% 40.38% 43.66% 37.76% 36.22% 18.18%

72.22% 38.10% 51.43% 36.36% 7.98% 0.00% 16.95% 40.38% 46.61% 42.35% 40.16% 18.18%

Dataset 1 63.64% 20.65% 28.36% 19.18% 16.92% 34.42% 24.39% 21.62% 28.70% 25.78% 20.95% 24.32% 30.00%

Dataset 2 Action on Issue Bug Reproduction Contribution & Commit. Expected Behaviour Investigation and Explr. Motivation Observed Bug Behav. Social Conversation Solution Discussion Task Progress Testing Workarounds

62.96% 19.05% 41.43% 13.64% 32.86% 12.50% 1.69% 29.23% 33.33% 26.02% 25.98% 9.09%

64.81% 27.62% 45.71% 18.18% 37.09% 0.00% 3.39% 25.00% 33.63% 29.59% 29.13% 18.18%

Summary of RQ2. The ensemble model achieves the highest F1-score for information type detection in both datasets, surpassing SOTA model [10] by 11-13%.

6 Effectiveness of iSum In this section, we aim to assess the effectiveness of iSum through an empirical user study focused on the task of issue understanding. Participants will engage in tasks involving the use of iSum and compare their experiences with and without the tool. Based on the results, we answer the following research questions: RQ3. How effective is iSum to support developers during bug report understanding? RQ4. Can iSum also help to explore feature enhancement requests in issue reports?

34

6

EFFECTIVENESS OF ISUM

6.1 Participants and Data Collection In this user study, we recruited 30 participants, predominantly comprised of developers (27 participants), alongside QA engineers (2 participants) and a project manager (1 participant). These participants were recruited through personal connections. Their experience in the software industry varied, with 50% having between 4 and 5 years of experience and 14 participants with 0 to 3 years. Each participant was assigned to one of two groups, with each group receiving a different arrangement of tasks involving the use of iSum. The study did not have participants perform two iterations on the same issue. Group 1 analyzed a bug report without the aid of iSum and a feature proposal with iSum, while Group 2 received the inverse setup—employing iSum for the bug report and proceeding without it for the feature proposal. This structure was chosen to distinguish the effects of iSum on both the speed and quality of information extraction and comprehension across various types of issue discussions. Both groups contain an equal number (15) of participants. The study was conducted using Google Forms, with separate forms for each group. Participants initially engaged in manual understanding of an issue report and rated the perceived difficulties faced during this process, followed by another issue understanding with iSum and rating the usefulness of the tool. Prior to using iSum, they received a guide explaining each information type and how the tool functions. To prevent fatigue, participants were encouraged to take short breaks between tasks. In addition to completing coding tasks, participants were asked to assess the complexity and load required for each task using the NASA Task Load Index (TLX) [25], which evaluates subjective workload. Participants gave free-form textual answers to task questions like ”What bug behavior was observed?” or ”What solutions were discussed?” They weren’t asked to classify individual sentences—just respond naturally. Authors manually identified relevant sentences for each info type.

6.2 Variables to Assess Task Performance To measure the effectiveness of iSum in facilitating issue report understanding, we designed a set of dependent variables that capture various aspects of task performance. These metrics allow us to quantitatively assess the improvements provided by iSum in comparison to traditional methods of issue report understanding. Correctness. This metric evaluates how accurately participants identified the classified information types within the issue discussions. Correctness (C) was assessed by comparing participants’ responses against a predefined set of answers. To establish the ground truth, the same author who labeled Dataset 2 conducted a thorough manual analysis of the issue discussions. Sentences related to the needed information type are labeled and periodically reviewed for consistency and accuracy against the established definitions of information types. Upon this, candidate answers have been created. Before similarity anal-

6.3

Statistical Analysis

35

ysis, each response is manually segmented under each information type, and for information types where multiple answers may exist, the response is deemed correct if it successfully identifies one or several of the expected answers. The Correctness score (C) is calculated using the following formula: C=

Number of Correctly Identified Information Types Expected Number of Information Types

(1)

To automate the evaluation of responses, we utilize the SentenceTransformer library, employing the allMiniLM-L6-v2 model [30] to encode the actual and expected answers into embeddings. Subsequently, we gauge the semantic similarity between these embeddings by computing the cosine similarity to compare against the threshold of 0.7 [67] and assessing how closely the actual answer aligns with the expected summary. For example, in the TensorFlow #33030 memory leak task asking “What solutions were discussed?”, our manually curated ground-truth exemplar was: “Calling tf.keras.backend.clear session() after every model.predict() fixes the issue” (from issue comments). A participant responded: “After using tf.keras.backend.clear session(), the memory leak is fixed.” The SentenceTransformer cosine similarity exceeded the 0.7 threshold, marking the response as correct for the Solution Discussion information type and awarding full task credit—demonstrating semantic matching rather than verbatim reproduction. Task Load. We used the NASA TLX metrics values for computing the task loads of participants. Participants rated the task load required on a scale from 1 to 100, where higher values indicate more task load. This self-reported metric reflects the cognitive and physical demand of the task, with and without iSum. Six aspects were assessed : 1. Mental Demand: The extent of mental and perceptual activity required 2. Physical Demand: The level of physical activity involved 3. Temporal Demand: The pressure of time constraints during task execution 4. Performance: The success in achieving task goals and satisfaction with the outcomes 5. Effort: The overall effort needed to achieve the performance level 6. Frustration Level: The emotional state experienced, ranging from stress to contentment. Finally, the overall TLX score is the average of each participant’s TLX score, T LXi : P6 T LXi =

j=1 Si,j

6 Si,j = score for dimension j given by participant i

(2)

6.3 Statistical Analysis To assess the differences between the different settings in terms of correctness and task load, we employed the Mann-Whitney U test and Cliff’s Delta. Mann-Whitney U test. The Mann-Whitney U test [9] was chosen for its suitability in analyzing non-normally distributed data, a common characteristic in experimental data in human factors research. This test evaluates

36

6

EFFECTIVENESS OF ISUM

whether two independent samples were selected from populations having the same distribution. Specifically, it was used to determine if the differences in Correctness scores and Task Load ratings between the two settings were statistically significant. A significance level (p-value) of less than 0.05 was considered indicative of a statistically significant difference between the two groups. Cliff ’s Delta. This non-parametric measure of effect size determines the degree to which two distributions differ. This metric ranges from -1 to 1, where values close to 0 suggest similar distributions, and values approaching -1 or 1 indicate greater disparity. To interpret Cliff’s Delta, we adopt thresholds based on the recommendations by Romano et al [71].  Negligible (N), if |δ| ≤ 0.147    Small (S), if 0.147 < |δ| ≤ 0.33 Effect Size = Medium (M), if 0.33 < |δ| ≤ 0.474    Large (L), if 0.474 < |δ| ≤ 1 6.4 Effectiveness of iSum in Bug Report Understanding (RQ3) 6.4.1 Study Design In the task design section of this user study, we selected a bug report, issue #33030 from TensorFlow GitHub repository to evaluate the effectiveness of our tool, iSum, in navigating complex discussion threads. To ensure a focused and time-efficient assessment, each participant was instructed to complete their assigned tasks within a strict five-minute window. This setup aimed to simulate real-world conditions where quick issue resolution is often necessary. Table 20 presents the details of the task setting. Participants were assigned the role of a software developer and provided with a bug report that included a lengthy and complex discussion thread. Their task was to identify the bug’s behavior as described in the report, investigate potential causes mentioned by stakeholders, if any, and identify any proposed solutions or workarounds discussed. This task aimed to evaluate participants’ ability to extract these four types of information from issue reports within a 5-minute time frame. Table 20 presents the related information types for this task. Half of the participants performed the task with the assistance of iSum (BI), while the other half did it without (BM). The performance difference in finding the correct information were used to assess iSum’s effectiveness in facilitating quick and thorough understanding of technical issues from complex information threads, which is critical for timely and effective bug resolution. 6.4.2 Study Results The feedback from participants as illustrated in Figure 9, indicates that extracting relevant information and understanding the overall context of an issue is challenging without the aid of information type classification. Most participants found it difficult, with 11 rating it as high difficulty. On the other

6.4

Effectiveness of iSum in Bug Report Understanding (RQ3)

37

Table 20: Task details. Here, G# = Group #, Feat. = Feature request Task

Issue Type

iSum Used

G#

Task Details

Related Information Type

BM BI

Bug Bug

No Yes

G1 G2

Investigate the bug’s behavior and possible causes. Look for proposed solutions or workarounds.

Obs. Bug Behaviour, Investigation. , Solution Discussion, Workaround

FM FI

Feat. Feat.

No Yes

G2 G1

Understand the motivations behind the proposal and the expected outcomes. Look for contribution interest or assignments, new challenges or feature requests and monitor the task progress

Motivation, Expected Behaviour, Potential New Issues and Requests , Contribution and Commitment, Task Progress

Neutral

Difficult

Not Difficult

20%

0

20%

40%

Not Useful

60%

20%

0

Neutral

Useful

20%

40%

60%

Fig. 9: (a) Manual analysis difficulty Perception (b) iSum usefulness perception for bug report understanding 1

100

0.8

80

0.6

60

0.4

40

0.2

20

0

0 BM BI

BM BI

Fig. 10: (a) Correctness of the task and (b) Task load spent across settings for bug report understanding. Here, BI = Bug report understanding with iSum and BM = without iSum. hand, the effectiveness of iSum in aiding this process received mostly positive feedback. Out of the participants, 10 reported a positive impact, suggesting that iSum facilitated the extraction and understanding of relevant information within issue discussions. Meanwhile, 2 participants felt neutral about its effectiveness, and 3 experienced a negative impact, indicating variability in how iSum’s classifications met individual needs or expectations in understanding issue reports. In Figure 10, we show the correctness of the tasks related to bug report understanding and task load across the two settings (manual analysis and analysis using iSum) using two box plots, one for each setting. In comparing the correctness of information type identification between manual analysis and the use of iSum, the data reveals that the mean correctness score increased from 0.45 (BM) to 0.63 (BI) for bug report understanding after using iSum and enhances participants’ ability to correctly identify information. For instance, while the solution to the problem of training for one epoch at a time, saving the model, and reloading it in subsequent iterations was buried within a heap of

38

6

EFFECTIVENESS OF ISUM

Table 21: Results of Wilcoxon Mann-Whitney U test and Cliff’s δ effect size for the pair-wise comparisons of the correctness and task load on bug understanding task without and with the assistance of iSum (BM vs BI) Metric

p-value

Cliff ’s δ

Correctness Average TLX

0.012 0.007

-0.489 L 0.587 L

Mental Demand Physical Demand Temporal Demand Performance Effort Frustration Level

0.003 0.573 0.001 0.028 0.001 0.004

0.640 L 0.124 N 0.698 L 0.471 M 0.711 L 0.622 L

comments, iSum was able to effectively surface this solution. One participant noted, ”easier to find what I need,” highlighting the tool’s effectiveness in locating pertinent details. However, the median for both BM and BI is 0.5. The improvement with iSum is therefore more visible in the mean than the median, reflecting that iSum helped more participants achieve higher correctness scores. Furthermore, when participants explored bug reports manually, the median task load was significantly higher at 77.50 compared to 53.00 when using iSum. This highlights iSum’s effectiveness in reducing the cognitive load and time required to navigate complex bug reports. In Table 21, we present the statistical results from the Wilcoxon MannWhitney U tests and Cliff’s δ calculations to show the comparison between the Manual method and iSum across different metrics of bug report understandings. The p-value of 0.012 indicates a statistically significant difference in correctness between Manual and iSum, with a Cliff’s δ of -0.489. This suggests that iSum significantly improves correctness in bug report understanding compared to the manual method, with a large effect size indicating substantial improvement. Again, the p-value of 0.007 suggests a significant difference in overall task load, with a very high Cliff’s δ of 0.587. This indicates that iSum greatly reduces the task load required in bug report understanding, showing a strong effect where iSum is much more efficient than the manual approach. Furthermore, the performance dimension shows a Cliff’s δ of 0.471, which is classified as Medium (M) according to the thresholds of Romano et al. [71], where a Large effect requires |δ| > 0.474. However, the value is borderline M/L, only 0.003 below the Large threshold. The relatively higher p-value (0.028 vs., for example, 0.001 for Temporal Demand) likely reflects greater variability in how participants perceived their own performance satisfaction. Even when iSum helped participants locate information more efficiently, some participants remained uncertain about the completeness of their answers, resulting in more variable self-reported performance scores compared to more objective dimensions such as Effort or Temporal Demand. Nevertheless, the result remains statistically significant (p < 0.05) with a near-large effect size, indicating that

6.5

Effectiveness of iSum in Feature Request Understanding (RQ4)

39

iSum meaningfully reduced perceived performance load. Summary of RQ3. Participants found it challenging to extract relevant information and understand the overall context of an issue without the aid of information type classification and iSum positively impacted the process for most participants. The use of iSum resulted in increased correctness of information type identification and reduced cognitive load required to navigate complex bug report compared to manual analysis.

6.5 Effectiveness of iSum in Feature Request Understanding (RQ4) 6.5.1 Study Design For this task, we selected a feature request, issue #206, from TensorFlow and also gave participants a five-minute window to complete understanding task as detailed in Table 20. In this task, participants assumed the role of a project manager reviewing a feature proposal. Again, this challenge involved delving into a detailed discussion thread related to the feature, understanding the motivations behind the proposal and the outcomes it aimed to achieve. Furthermore, they need to look for any potential challenges or additional feature requests that may discuss in the comment section; also monitor any sign of enthusiasm, willingness to contribute, or preliminary task assignments within the discussion. Lastly, they need to monitor the progress of tasks. For this feature understanding also, there are two task settings, understanding with (FI) or without iSum (FM) and extracting key insights relevant to the feature proposal. Both studies did not have participants perform two iterations on the same issue. We employed a between-subjects design: G1 analyzed the bug report manually (BM) and the feature report with iSum (FI), while G2 analyzed the bug report with iSum (BI) and the feature report manually (FM). Therefore, each participant encountered each issue only once. 6.5.2 Study Results

Not Difficult

0

20%

Neutral

40%

Difficult

60%

80%

Not Useful

60%

40%

20%

Neutral

0

Useful

20%

40%

Fig. 11: (a) Manual analysis difficulty perception (b) iSum usefulness perception for feature report understanding In the user study focused on feature report understanding, the response for evaluating the difficulty of extracting relevant information and understanding

40

6

EFFECTIVENESS OF ISUM

the overall context without the aid of information type classification was overwhelming. This is illustrated in Figure 11, with 14 out of 15 participants rating the difficulty as high. For assessing the effectiveness of the iSum classifications for feature report understanding, participants provided mixed feedback. Out of 15 participants, 6 reported a positive impact, stating that iSum classifications aided in extracting relevant information and understanding the overall context of the issue. Another 6 participants remained neutral, indicating neither significant benefit nor drawback from using iSum. Remaining three participants rated iSum as “Not useful” mainly because of misclassified or missing tags and perceived mismatches between the tool’s labels and what they needed for nuanced feature discussions (e.g., side questions not tagged as Potential New Issues and Requests). The correctness and task load spent for the tasks related to feature report understanding are shown in Figure 12. In this task, iSum only slightly improves the mean correctness (0.49 (FM) to 0.55 (FI)) and its impact on the challenging scenarios was limited. For instance, during feature report understanding, one participant raised a side question regarding the optimal approach for achieving a specific type of indexing. But iSum failed to tag it as Potential Issues and Requests related comment. Consequently, participants relying on iSum may have missed this information, as highlighted by one user’s comment: ”Wrong tags make users lose information.” This indicates that iSum’s performance may be hindered when discussions deviate from conventional issue-related topics or lack explicit tags. However, for the understanding of feature reports in the user study, the task load required also saw a decrease when using iSum compared to manual methods; underscoring the reduction in workload and task load that iSum offers when managing and understanding feature reports. The median task load score for manual understanding was 74.17, which dropped to 59.00 with the use of iSum.

1

100

0.8

80

0.6

60

0.4

40

0.2

20

0

0 FM FI

FM FI

Fig. 12: (a) Correctness of the task and (b) Task load spent across settings for feature report understanding. Here, FI = Feature request understanding with iSum and FM = without iSum.

41

Table 22: Results of Wilcoxon Mann-Whitney U test and Cliff’s δ effect size for the pair-wise comparisons of the correctness and task load on feature report understanding task with and without the assistance of iSum (FM vs FI) Metric

p-value

Cliff ’s δ

Correctness Average TLX

0.445 0.119

-0.160 S 0.320 S

Mental Demand Physical Demand Temporal Demand Performance Effort Frustration Level

0.139 0.298 0.195 0.133 0.156 0.219

0.320 S 0.227 S 0.280 S 0.324 S 0.307 S 0.267 S

Again, the statistical results from the Wilcoxon Mann-Whitney U tests and Cliff’s δ calculations comparing the Manual method and iSum across various metrics of feature report understandings are presented in Table 22. With a p-value of 0.445, there is no statistically significant difference in correctness between the two methods for feature report understanding. However, the Cliff’s δ of -0.160, classified as small (S), points towards a slight tendency for the manual method (FM) to potentially perform better in terms of correctness. Additionally, the p-value of 0.119 indicates no significant difference in task load, yet the high Cliff’s δ of 0.320, indicates a small effect size favoring a reduction in task load when using iSum. This suggests that iSum potentially reduces task load considerably in feature report understanding, although the results are not statistically significant. Summary of RQ4. Feedback on the effectiveness of iSum for feature request understanding was mixed, with some participants reporting positive impacts on information extraction and understanding, while others remained neutral. Although iSum slightly improved the mean correctness of tasks related to feature report understanding, its impact on challenging scenarios was limited. Nevertheless, using iSum resulted in reduced task load and workload compared to manual understanding methods.

7 Usability and Feasibility of iSum In this section, we extend our user study with 10 industrial and OSS software engineers to investigate the practical usability, feasibility, and potential enhancements of iSum in real-world software development contexts. We show them this tool, explain its features and usage. By soliciting feedback and insights from industrial developers, the study sought to assess how well iSum aligns with the needs, preferences, and workflows of professionals working in diverse software development environments. Additionally, the study aimed to identify potential areas for improvement and refinement to ensure that iSum

42

7

USABILITY AND FEASIBILITY OF ISUM

Table 23: Interview Questions: Understanding Usability and Feasibility of iSum (C/O=Close/Open question) Q#

Questions

O/C

1 2 3

Usability (RQ6.1) Do you find the iSum tool useful? Who do you think can benefit from the tool? Will you or your team use this tool?

C C C

4 5 6

Feasibility (RQ6.2) What are the strengths of this tool? What are the shortcomings of this tool? Do you have any suggestions for improving this tool?

O O O

meets the evolving needs of its users and remains a valuable asset in the software development toolkit. We aim to address the following research question: RQ5. How do industrial developers perceive the usefulness of iSum? We divide this research question into two sub questions as follows: RQ5.1. How do the developers consider of the usability of iSum? RQ5.2. How do the developers consider of the feasibility of iSum in their daily tasks?

7.1 Study Methodology In this study, we conducted another series of semi-structured interviews with the same 10 participants from the previous phase (refer to Section 2). This group comprised 10 professional software engineers, including six developers, two QA engineers, and two project managers. They bring varied levels of experience to the table, actively working within the software industry for several years. These interviews were conducted and recorded via Microsoft Teams, using automated transcription for data capture. The discussions focused on two primary topics: the usability and feasibility of iSum. The interview questions were a mix of closed and open-ended formats. Q1 and Q3 used binary yes/no response options, while Q2 was administered as a multi-select question with a predefined list of options. Q4-Q6 are open-ended questions. For closed-ended questions, responses were quantified, while an open-coding approach was applied to open-ended questions. We conducted open coding of each interview using an online qualitative data analysis software, Ligre [50]. This involved systematically reviewing the interview transcripts, identifying recurring themes, and assigning relevant codes to segments of text to categorize and organize the data. Table 23 outlines the interview questions.

7.2

Usability of iSum (RQ5.1)

43

7.2 Usability of iSum (RQ5.1) 7.2.1 Perceived Usefulness of iSum 80% Participants perceived iSum as useful for both project and issue level summarization 20% Participants perceived iSum as useful for only project level summarization Participants unanimously found the iSum tool to be useful, yet their appreciation of its features varied depending on their roles within the project. Two participants highlighted that summarization at the project level was particularly valuable since they found it best suited for understanding and managing broader project dynamics. Participant P02 articulated this viewpoint by stating, ”I mean from a developer perspective, because I believe in details and I believe developers should look into the details. But from a management perspective, this is awesome.” - P02. In contrast, the majority felt that both project and issue-level summarizations were beneficial, suggesting a wider applicability across different stakeholder needs. 7.2.2 Probable Beneficiaries of iSum 80% Participants consider project manager as a beneficiary of iSum 70% Participants consider developer as a beneficiary of iSum 20% Participants consider newbie as a beneficiary of iSum Participants think this tool would be beneficial for three kinds of stakeholders: developers, project managers and anyone who is new to the project. Since developers are often confronted with the challenge of reading through extensive discussions to uncover essential details that impact their work, participants think this process can be frustrating and time-consuming and lead to delays and inefficiencies in issue resolution. P04 encapsulated this struggle, remarking, ”Sometimes while developing, going through all the discussion is a bit frustrating or takes a lot of time. So it will benefit the developers.” - P04. Conversely, P09 emphasized the daily importance of tracking the project’s development and how iSum can assist with that: ”It’s more for the project manager to understand the project on the daily basis.” - P09. Participants also noted that new team members, who may not be familiar with the project’s history or specific issue threads, can use iSum to quickly understand the project’s current state. P09 explained the advantage clearly, ”Those who new to the project, doesn’t know anything about the project; they can quickly get some information, some insight from the tool to understand what’s going on.” - P09. 7.2.3 Willingness to use iSum 90% Participants are willing to use iSum 10% Participants are not willing to use iSum A significant majority (90%) of participants expressed willingness to adopt the

44

7

USABILITY AND FEASIBILITY OF ISUM

iSum tool, highlighting its potential value across multiple roles. One added, they might like to use it as a base and personalize according to their need. Only one participant (P02) did not want to use it since, at that time, it lacked personalization options. We added personalization options after this interview and conducted the remaining eight interviews.

7.3 Feasibility of iSum (RQ5.2) 7.3.1 Strengths of iSum 80% Participants think iSum saves time in finding information 40% Participants think iSum can provide an initial overview of the project/issue Eight participants think the tool’s success is in saving time in finding information from issue discussion. They preferred the highlighting and summary features the most. The customisation was also a favourite because everyone has various preferences depending on their role and practice. Four of them remarked on how helpful the tool is for getting a quick overview of the project or issue. One participant encapsulates both advantages in the following statement: ”The person concerned are getting information within few clicks and that is beneficial for developers. They can go to the specific issue and check the comment box or discussion and for project managers, they can use their filters, then create brief overview of his project.”-P07. 7.3.2 Shortcomings of iSum 60% Participants consider inaccuracy of information detection as a shortcoming of iSum 40% Participants think iSum needs integration of more information 10% Participants think iSum may not help assigned developers 10% Participants faces difficulty in understanding information type Six participants are concerned about the inaccuracy of the tool. One participant found the names of certain information types to be unclear, asking questions like ”Expected behavior? Motivation. What’s the difference between them?” - P04. This participant felt that users needed sufficient explanations. This provides us with a suggestion to include the definition of information types in the tool’s usage guide. Another argued, usually assigned developers are well aware of the issue discussion. In that case, issue summarization might not help them, rather can help the ones who want to take an overview. 7.3.3 Suggestions for Improvements 75% Participants suggest to introduce and integrate more information 25% Participants suggest to expand for other platform

45

We put some of the suggestions of first two participants into practice, as follows: (1) first, in an issue report, we incorporated the information type summary generator that P01 suggested and then we conducted the interview with P02 using the revised tool (2) customisation options were added in response to P02’s request and the tool was then finalized for the following eight interviews . Here we have listed only the suggestions from these eight participants. Six participants reiterated the desire for additional information, consistent with their earlier feedback about the tool’s shortcomings. They wanted features like keyword highlights and timelines to be incorporated, reflecting the recurring need for comprehensive data retrieval. Two participants advocated for expanding the tool to support other ITS, such as the Unity forum, by stating, ”I hope there is one in for Unity.” - P04. Summary of RQ5. Participants found iSum helpful for summarising issues and projects and believe it might benefit developers, project managers, and anybody else new to the project. Most participants expressed their willingness to use this tool. Participants claim that this tool’s key advantage is its ability to save time when looking up necessary information and provide a brief summary of a project or problem. Lastly, nine out of ten individuals said they would like to use the tool. The participants suggest that iSum can be more effective if the detection performance of the information types can be further improved. Also, they suggested to enhance the tool for different ITS and asked for more information retrieval from the issue discussion, besides showing the information types.

8 Threats to Validity Internal validity threats refer to the authors’ biases while conducting a study. One threat to internal validity is inherent in the participant selection bias of our user studies. We used personal connections to recruit participants, which may not fully represent the broader developer population. Additionally, the scoring criterion in the effectiveness study — where a response is deemed correct if it identifies any one of multiple possible correct answers — is a lenient criterion that may lead to optimistic effectiveness results. However, we argue this reflects real-world issue understanding in scenarios where identifying even one correct answer is practically useful. We acknowledge this as a limitation and recommend a stricter evaluation criterion in future work. A further threat to internal validity concerns our evaluation protocol: sentencelevel StratifiedKFold does not guarantee issue-level disjointness. We quantified this cross-fold overlap for our built dataset (D2): 10 of the 4,241 test sentences (0.24%) appear verbatim here. On inspection, these are within-issue repeated or quoted fragments, for example a command or a quoted benchmark line that recurs later in the same thread. Excluding these sentences from evaluation leaves performance essentially unchanged: the change is less than 0.01

46

8

THREATS TO VALIDITY

for every model we evaluated. Verbatim cross-fold overlap therefore does not inflate our reported results. We further acknowledge that keeping whole issues together during splitting would remove this overlap entirely, and that some residual issue-level overlap is not fully captured by a text-level analysis. Construct validity threats refer to errors in the data collection process. We used the same dataset developed and used by Arya et al. [10] that was prepared through exhaustive manual labeling by the authors. To create Dataset 2, hours were spent manually annotating the selected issue discussions of the OpenJ9 project repository following the same guidelines used for building Dataset 1, with the first author strictly adhering to the label definitions established by Arya et al [10]. We chose the most commented closed issues and used the same definition of information types for labeling. However, selecting only the most commented closed issues may not fully represent the entire distribution of issue reports, as shorter or still-open threads may contain different distributions of information types. Furthermore, Dataset 2 was annotated by a single author without independent double-coding or inter-rater agreement measures (e.g., Cohen’s κ). Although the annotation process was frequently reviewed and discussed with all co-authors, including two industry partners from IBM, the absence of formal inter-rater agreement is a limitation. Due to the first author’s personal and professional experience bias, interpretation of the issue comments of Dataset 2 may differ in some ways from Dataset 1. The open coding of the feasibility interviews was conducted individually by the first author, which may introduce subjective bias, though all codes and themes were subsequently reviewed and refined collaboratively with co-authors. Additionally, sentences with multiple labels were removed to maintain a single-label classification problem, which may exclude some of the most information-dense content and reduce alignment with real-world issue discussions. To quantify the severity of this exclusion, we recovered the 293 multi-label sentences (6.3% of Dataset 1) that were removed during dataset construction. Because these sentences already carry expert multi-label annotations in the source corpus, no new annotation was required. We evaluated iSum’s LightGBM classifier on these sentences under a leave-one-issue-out protocol, so that each sentence is classified by a model that never observed its issue. 10 of the 293 sentences also carry a rare label that was removed during construction; we set these aside and report the remaining 283 sentences whose labels all fall within the modeled set. Of these 283 multi-label sentences, iSum detected all information types in only 9 (3%) and at least one type in 97 (34%), missing at least one type in 274 (97%). This confirms that excluding multi-label sentences is a genuine limitation for information-dense content, and that a simple sentence-splitting workaround would not recover the full label set. We did not repeat this analysis on Dataset 2 because multi-label sentences are negligible there (26 sentences, 0.6% of Dataset 2). Additionally, BERT and RoBERTa classify each sentence in isolation, which is an imperfect operationalization of information type for discourse-dependent sentences. We quantified this with an error analysis using ±1 sentence context under issue-grouped 5-fold cross-validation: it yields a small net improvement on Dataset 1 (weighted F1 0.325 to 0.361) but a net

47

decline on Dataset 2 (0.480 to 0.398), helping discussion-oriented types while hurting short, self-contained ones. This confirms single-sentence classification as a genuine but selective construct-validity limitation rather than a uniform one. External validity threats concern the generalizability of the findings of this study. While more responses would have offered more proof of generalizability, we note that we already observed saturation in our manual coding of themes and labels during open coding. Our effectiveness study considered only one bug report and one feature request from TensorFlow, which limits generalization to other kinds of bugs, features, projects, and issue structures. Furthermore, user study participants were predominantly early-career developers, with only one participant having over five years of experience, which limits our ability to generalize iSum’s effectiveness to highly experienced users. We added a cross-dataset zero-shot evaluation, with no target-project fine- tuning, using our actual models (BERT, RoBERTa, LightGBM, and Random Forest). Each model is trained on one dataset and tested on the other, over the 11 labels shared between Dataset 1 and Dataset 2, and we report weighted F1. Training on Dataset 1 and testing on Dataset 2 (OpenJ9), BERT reaches 0.414, RoBERTa 0.431, LightGBM 0.238, and Random Forest (conversational features only) 0.202; training on Dataset 2 and testing on Dataset 1 gives 0.435, 0.444, 0.326, and 0.271, respectively. An ensemble of BERT and RoBERTa improves on either transformer alone in both directions (0.434 and 0.465), whereas adding LightGBM and Random Forest to that ensemble lowers the average rather than raising it. These results show that the transformer models generalize substantially better across datasets than the shallow models, whose reliance on dataset- specific conversational and lexical patterns does not transfer to an unseen project. iSum therefore drops moderately on a fully unseen project in the zero-shot setting, but retains meaningful cross-project signal through its trans- former components.

9 Related Work Related work to our study can broadly be divided into three types: 1. Information overload in SE, 2. Issue report summarization, and 3. Issue discussion exploration. We briefly discuss some of the related works in each category:

9.1 Information Overload in SE Information overload refers to the phenomenon where stakeholders are exposed to an excess of information, much of which is irrelevant to their current tasks. This critical challenge often affects stakeholders’ productivity and decisionmaking capabilities [37]. Developers and other project stakeholders are overwhelmed by the sheer volume of information from documentation, version control systems, and bug reports. As software systems grow in complexity,

48

9

RELATED WORK

the volume of data that stakeholders must process and manage also increases exponentially. Studies such as those by Meyer et al. [53] examine how developers manage and prioritize incoming information from various sources like documentation, version control systems, and bug reports. Their findings suggest that developers often experience cognitive overload, which can reduce the efficiency of debugging and coding activities. As new technologies emerge, software engineers must continuously learn new programming languages, frameworks, and tools. Vasilescu et al. [78] investigated how the adoption of multiple tools and platforms leads to information overload, which can delay decisionmaking and introduce errors in software development. Research by Bjørn and Ngwenyama [12] looks into how distributed teams handle the coordination of information and the overload it can create, often leading to miscommunication and project delays. Pennington et al. [63] conducted a study to investigate how information overload can impact risk management decisions in software projects. Several software tools have been developed to alleviate information overload by filtering information based on context. One such tool is Mylar [39], which integrates with the Eclipse IDE and focuses on filtering irrelevant data using task context. Semantic knowledge management systems, explored by Demirsoy et al. [16], can also address information overload by structuring and categorizing software knowledge. Modern ITS often manage thousands of tickets per project, making it challenging for developers to prioritize and comprehend the most critical information. Ko et al. [42] investigated how developers navigate bug databases and found that the sheer volume of entries often overwhelms them, leading to inefficiencies in addressing issues. Anvik et al. [7] highlighted how duplicate and redundant bug reports clutter the system, reducing the efficiency of bug triage. They recommended automated techniques to identify and merge duplicate reports. Baysal et al. [11] explored information overload in issue tracking systems (ITS), highlighting how developers often experience frustration due to the difficulty in accessing necessary information for specific tasks. They point out that the relentless influx of new data, most of which is irrelevant to the immediate needs of developers, complicates the situation further. This excess of non-essential information can lead to critical issues being overlooked. To overcome this, they propose a developer-centric approach to issue tracking, which employs a grounded theory analysis of developer comments to design customized, task-specific dashboards. We also aim to alleviate information overload challenges in ITS by automating the detection and extraction of relevant information from issue discussions and aiding stakeholders in understanding issue reports efficiently and effectively.

9.2 Issue Discussion Exploration Rath and Mäder [69] discovered three patterns of conversation in issue threads between stakeholders: (1) monologue (2) feedback and (3) collaboration. All of these conversation may contain important information to help developers in

9.3

Issue Report Summarization

49

their issue resolution process and managers to keep track of the issue. Several works have been done to investigate the issue discussions to retrieve focused information. Viviani et al. [80] analyzed pull request discussion to retrieve design related information. Agrawal et al. [1] investigated types of visual contents present in issue discussions. Sillito et al. [74] examined developer queries, while Breu et al. [14] investigated potential impact of the developer-posted queries on issue resolution procedure and Huang et al. [29] created a model to forecast the likelihood of questions being raised in issue reports. Significant body of research are dedicated to sentiment analysis [24, 31, 57, 60, 72, 87] of issue discussion, track down uncivil [20] and toxic [55] comments or presence of human values [59]. Instead of retrieving a single type of information from the issue discussion, our work tries to capture different types of information to support issue exploration. Multiple works are also present in the literature to support multiple information retrieval from issue discussion. Li et al. [47, 48] categorized review comments and introduced classification algorithms to better understand the code review process. Arya et al. [10] uncovered 16 information types from issue discussion and later employed shallow ML models for detection. Paing et al. [61] integrated Arya et al.’s classification model [10] to create a tool that can assist Empirical Software Engineering (ESE) researchers in finding issues containing query keywords. While existing research has made significant advancements, our study focuses on offering better classifying models that surpass SOTA tools [10] for the detection of information types and developing a framework for quick issue exploration to support developers and project managers. We also conduct an evaluation study to assess the effectiveness of our framework in real-world settings. By incorporating feedback from actual users through this evaluation, we aim to ensure that our tools are not only technologically advanced but also practically useful, aligning with the detailed and diverse needs of OSS stakeholders.

9.3 Issue Report Summarization ITSs are proven to be rich sources of diverse information about software projects which can be utilized in different software activities. However, these issue reports can be extensive and verbose, leading to information overload. This overload makes it challenging for stakeholders to quickly analyze and extract relevant information. To help developers, testers, and project leads sift through issue reports efficiently, several studies have focused on automated summarization techniques to provide concise summaries of report contents [43, 58]. Rastkar et al. [68] proposed an approach that utilizes NLP and machine learning methods to generate concise summaries of bug reports. Their work demonstrated that machine learning models trained on annotated datasets could effectively identify the most relevant parts of an issue report. Li et al. [46] proposed DeepSum utilizing unsupervised deep neural networks, while Jiang et al. [32] introduced a supervised algorithm called Logistic Regression with Crowdsourced Attributes (LRCA). Jindal et al. [33] explored techniques for

50

10

FUTURE WORK

extracting meaningful keywords and key-phrases to generate summaries with relevant scores. Mukhtar & Lee [56] combine classification and summarization techniques to automatically generate concise summaries of bug reports. Practical AI tools such as Jira AI Summary and Issue Summarizer [75] demonstrate how AI can summarize ticket descriptions and summaries in Jira workflows. Kogents Jira AI Integration [3] discusses integrating AI into Jira platforms for ticket classification and issue management, while the Eesel AI Blog [2] explores best use cases of AI for automating ticket categorization and summarization in software teams. However, the same summary cannot be applicable for all tasks and meet the diverse requirements of the project stakeholders. Therefore, to serve the multifaceted issue summarization and contribute to it, we focused on investigating issue discussion threads and information identification. Recent works also focus on automating bug triage, classification, and workflow support as part of summarization for stakeholder efficiency. Kiashemshaki et al. [40] propose an instruction-tuned LLM for automating bug triage with strong shortlist quality. Zhao et al. [90] review 234 triage papers, analyzing concepts, architectures, datasets, metrics, and obstacles. Zhang et al. [89] introduce BTAL, a BERT-TextCNN approach with an adaptive loss to handle imbalance. Andrade et al. [6] conducted an empirical study evaluating multiple traditional classifiers including SVM for classifying bug reports as bugs, feature requests, or mistakes. Jyothsna and Prasad [34] proposed an ensemble-based approach combining SVM, Random Forest, and Logistic Regression for classifying bug reports by their nature (e.g., GUI, performance, configuration), demonstrating the effectiveness of ensemble methods for bug report classification. However, their work operates at the report level rather than classifying individual sentences into fine-grained information types, as in our work. Nevertheless, their findings reinforce the value of ensemble methods in software engineering classification tasks, which is consistent with our results showing that heterogeneous ensembles outperform stand-alone models. As an additional stand-alone baseline, we also implemented a linear-kernel SVM, motivated by their observation that the linear kernel outperforms the Radial Basis Function (RBF) kernel for issue report classification. We tuned C ∈ {1, 10, 100, 1000} with class weight = balanced, under the same stratified 10-fold protocol, features, and evaluation as our main models. SVM reached a weighted F1 of only 0.42 on Dataset 1 and 0.47 on Dataset 2 (versus 0.61/0.55 for Random Forest and 0.67/0.63 for LightGBM), as its linear boundary struggles with our fine-grained, imbalanced sentence-level task. Given this, we did not include SVM in the main evaluation tables and provide full results in the supplementary material.

10 Future Work Our current BERT and RoBERTa models classify individual sentences without surrounding context. Because naive multi-sentence context can help some information types but hurt others, future work should condition context use on

51

information type rather than uniformly widening the input window. Also, we could explore integrating conversational features directly into BERT/RoBERTa inputs via custom fusion layers. Replacing TF-IDF with BM25 [70] or hybrid sparse+dense representations in LightGBM is a promising direction, particularly for handling specialized software engineering vocabularies. We also plan to explore alternative encodings for ensemble predictions and a simpler hybrid approach of feeding RoBERTa predictions directly into LightGBM as an alternative to the full ensemble. Regarding model selection, we plan to explore code-aware encoders such as CodeBERT [19] and modern encoders like ModernBERT [82], which may better handle code-heavy issue discussions. TabPFN [27] is a promising alternative to LightGBM for our tabular feature space. Instruction-tuned LLMs could be evaluated both as end-to-end classifiers for information type detection and as additional ensemble members or dedicated summarization components, subject to deployment constraints such as latency, cost, and data governance. iSum currently uses TextRank for extractive summarization, and future work could explore more modern summarization techniques, including abstractive and hybrid approaches. We also plan to extend iSum’s summaries to include similar related issues and offer deep-dive views of issue reports by information type, showing other relevant comments within each type. Multi-project leave-one-out cross-validation, where entire projects are held out for evaluation, would provide stronger evidence of cross-project generalization. We also plan to evaluate iSum on a broader distribution of issues, including shorter and still-open threads, to address the current bias towards highly commented closed issues. Sub-sentence labeling is a promising direction to capture multi-label information-dense sentences currently excluded from our datasets. A more rigorous taxonomy refinement via clustering could also produce more orthogonal information types, reducing overlaps between categories such as Investigation and Exploration and Solution Discussion. The information type labels and BERT/RoBERTa embeddings from iSum could serve as semantic anchors for cross-issue search and retrieval tasks. For example, a developer searching for how a similar bug was reproduced in past issues could query specifically over Bug Reproduction classified comments, improving precision over full-text search. Dense retrieval pipelines leveraging our model embeddings could also surface related or duplicate issues more effectively. iSum’s 14 conversational features and 16 information types also enable rich analytics for optimizing issue writing and management. Future extensions could compute correlations between information type features and issue resolution time - for example, correlating SUM(Bug Reproduction LEN) with issue open time to reveal how detailed reproduction steps impact resolution speed, or correlating SUM(Expected Behaviour LEN) with SUM(Action on Issue LEN) to highlight requirement-specification mismatches that lead to delays. The existing Manager View could be extended with resolution-time regressions or prescriptive alerts. For annotation and study methodology improvements, Dataset 2 should be re-annotated involving multiple annotators with adjudication and inter-rater

52

12

DECLARATIONS

agreement reported (e.g., Cohen’s κ). The feasibility interview open coding should be independently double-coded in future replications. Collecting separate feasibility feedback for bug reports and feature requests would provide more targeted insights, given iSum’s stronger performance on bug reports. A stricter correctness criterion requiring identification of all expected answers should also be evaluated alongside the current lenient criterion Finally, future evaluations should recruit a more balanced pool of participants across experience levels and evaluate iSum on more issue reports across multiple projects. A Likert-scale rating of information type importance would provide more fine-grained insights than the current binary selection. Understanding how iSum integrates with and complements existing developer workflows and toolchains - such as git commit messages, pull request discussions, and project management dashboards - is an important direction. We also plan to expand iSum to support other issue tracking systems beyond GitHub, such as JIRA, as suggested by participants.

11 Conclusions Issue report discussion provides a variety of information that may be useful to developers, managers and team leads, yet exploring these detailed discussions can be time-consuming and cumbersome. Recognizing the challenges stakeholders face with navigating extensive issue discussions, we designed a tool named iSum that provides issue and project summarization using information types found in issue discussions. The effectiveness of iSum was first established through performance improvements over SOTA [10] tools in accurately detecting various information types within issue discussions. Subsequently, we conducted a user study involving tasks that required efficient understanding of two types of issue reports—bug and feature—both with and without the assistance of iSum. The results from this study highlighted that iSum supports more effective and efficient understanding of bug reports between the two, significantly enhancing stakeholders’ ability to understand issues within a short time frame. Our future work will focus on extending iSum’s capabilities by incorporating richer summary views, improving classification performance for feature requests, and evaluating iSum across a broader range of projects and issue types to further validate its generalizability.

12 Declarations Data Availability. The code and data used for this paper are shared in this repository: https://github.com/tamanna037/ISum Conflict of Interest. The authors have no conflict of interest. Ethics Approval. This study was reviewed and received ethics clearance from the Conjoint Faculties Research Ethics Board of the University of Calgary. Informed Consent All participants provided verbal informed consent prior

53

to participation in the study. Clinical Trial Number Not applicable. Funding. This study was supported by IBM Canada. Author Contributions S.B. Tamanna contributed to the investigation, data curation, methodology design, development, and preparation of the manuscript. J.Y. Khan assisted with certain aspects of the ML model development. G. Uddin supervised the study and contributed to manuscript review and editing. L. Xia and L. Zhang contributed to project administration and manuscript review. All authors have read and approved the final version of the manuscript.

References 1. V. Agrawal, Y.-H. Lin, and J. Cheng. Understanding the characteristics of visual contents in open source issue discussions: A case study of jupyter notebook. In The International Conference on Evaluation and Assessment in Software Engineering 2022, EASE 2022, page 249–254, New York, NY, USA, 2022. Association for Computing Machinery. 2. E. AI. Jira ai summary: Best use cases for ticket categorization and summarization. https://www.eesel.ai/blog/ jira-ai-summary-best-use-cases-for-ticket-categorization-and-summarization, 2025. Accessed: March 2026. 3. K. AI. Jira ai integration. https://kogents.ai/platforms/jira-ai-integration/, 2025. Accessed: March 2026. 4. T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, page 2623–2631, New York, NY, USA, 2019. Association for Computing Machinery. 5. I. Alazzam, A. Aleroud, Z. A. Latifah, and G. Karabatis. Automatic bug triage in software systems using graph neighborhood relations for feature augmentation. IEEE Transactions on Computational Social Systems, 7:1288–1303, 2020. 6. R. Andrade, C. Teixeira, N. Laranjeiro, and M. Vieira. An empirical study on the classification of bug reports with machine learning, 2025. 7. J. Anvik, L. Hiew, and G. C. Murphy. Coping with an open bug repository. In Proceedings of the 2005 OOPSLA Workshop on Eclipse Technology EXchange, eclipse ’05, page 35–39, New York, NY, USA, 2005. Association for Computing Machinery. 8. J. Aranda and G. Venolia. The secret life of bugs: Going past the errors and omissions in software repositories. In 2009 IEEE 31st International Conference on Software Engineering, pages 298–308, 2009. 9. A. Arcuri and L. Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (ICSE), pages 1–10. IEEE, 2011. 10. D. Arya, W. Wang, J. L. C. Guo, and J. Cheng. Analysis and detection of information types of open source software issue discussions. In Proceedings of the 41st International Conference on Software Engineering, ICSE ’19, page 454–464. IEEE Press, 2019. 11. O. Baysal, R. Holmes, and M. W. Godfrey. No issue left behind: reducing information overload in issue tracking. FSE 2014, page 666–677, New York, NY, USA, 2014. Association for Computing Machinery. 12. P. Bjørn and O. K. Ngwenyama. Virtual team collaboration: building shared meaning, resolving breakdowns and creating translucence. Information Systems Journal, 19, 2009. 13. L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001. 14. S. Breu, R. Premraj, J. Sillito, and T. Zimmermann. Frequently asked questions in bug reports. 04 2009. 15. I. Chawla and S. K. Singh. An automated approach for bug categorization using fuzzy logic. In Proceedings of the 8th India Software Engineering Conference, ISEC ’15, page 90–99, New York, NY, USA, 2015. Association for Computing Machinery.

54

12

DECLARATIONS

16. A. Demirsoy. Using semantic knowledge management systems to overcome information overload problems in software engineering. 2013. 17. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019. 18. T. G. Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems (MCS), Lecture Notes in Computer Science, volume 1857, pages 1–15. Springer, 2000. 19. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP, pages 1536–1547, 2020. 20. I. Ferreira, J. Cheng, and B. Adams. The ”shut the f**k up” phenomenon: Characterizing incivility in open source code review discussions. Proc. ACM Hum.-Comput. Interact., 5(CSCW2), oct 2021. 21. J. H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189–1232, 2001. 22. G. Gousios, M.-A. Storey, and A. Bacchelli. Work practices and challenges in pull-based development: The contributor’s perspective. In Proceedings of the 38th International Conference on Software Engineering, ICSE ’16, page 285–296, New York, NY, USA, 2016. Association for Computing Machinery. 23. L. Grinsztajn, E. Oyallon, and G. Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 35, pages 507–520, 2022. 24. E. Guzman, D. Azócar, and Y. Li. Sentiment analysis of commit comments in github: An empirical study. In Proceedings of the 11th Working Conference on Mining Software Repositories, MSR 2014, page 352–355, New York, NY, USA, 2014. Association for Computing Machinery. 25. S. G. Hart and L. E. Staveland. Development of nasa-tlx (task load index): Results of empirical and theoretical research. Advances in Psychology, 52:139–183, 1988. 26. P. Heck and A. Zaidman. A framework for quality assessment of just-in-time requirements: The case of open source feature requests. Requir. Eng., 22(4):453–473, nov 2017. 27. N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848, 2022. 28. Q. Huang, D. Lo, X. Xia, Q. Wang, and S. Li. Which packages would be affected by this bug report? In 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), pages 124–135, 2017. 29. Y. Huang, D. A. Costa, F. Zhang, and Y. Zou. An empirical study on the issue reports with questions raised during the issue resolving process. Empirical Softw. Engg., 24(2):718–750, apr 2019. 30. Hugging Face. all-MiniLM-L6-v2. Webpage. https://huggingface.co/ sentence-transformers/all-MiniLM-L6-v2. Accessed: March 2026. 31. M. R. Islam and M. F. Zibran. Towards understanding and exploiting developers’ emotional variations in software engineering. In 2016 IEEE 14th International Conference on Software Engineering Research, Management and Applications (SERA), pages 185– 192, 2016. 32. H. Jiang, X. Li, Z. Ren, J. Xuan, and Z. Jin. Toward better summarizing bug reports with crowdsourcing elicited attributes. IEEE Transactions on Reliability, 68(1):2–22, 2019. 33. S. G. Jindal and A. Kaur. Automatic keyword and sentence-based text summarization for software bug reports. IEEE Access, 8:65352–65370, 2020. 34. C. S. Jyothsna and B. S. Prasad. Smart bug classification using nlp and ensemble ml techniques. International Journal of Novel Research and Development, 10(11):a603– a608, Nov 2025. 35. R. Kallis, A. Di Sorbo, G. Canfora, and S. Panichella. Predicting issue types on github. Science of Computer Programming, 205:102598, 12 2020. 36. M. R. Kardoost, M. R. Moosavi, and R. Akbari. Devranker: An effective approach to rank developers for bug report assignment. 2022 12th International Conference on Computer and Knowledge Engineering (ICCKE), pages 030–035, 2022.

55 37. A. Kashada, A. Isnoun, and N. Aldali. Effect of information overload on decision’s quality, efficiency and time. 05:53–58, 2020. 38. G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightgbm: a highly efficient gradient boosting decision tree. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 3149–3157, Red Hook, NY, USA, 2017. Curran Associates Inc. 39. M. Kersten and G. C. Murphy. Using task context to improve programmer productivity. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering, SIGSOFT ’06/FSE-14, page 1–11, New York, NY, USA, 2006. Association for Computing Machinery. 40. K. Kiashemshaki, A. Khosravani, A. Hosseinpour, and A. Akhavan. Automated bug triaging using instruction-tuned large language models. arXiv preprint, 2508.21156, 2025. 41. R. Kikas, M. Dumas, and D. Pfahl. Using dynamic and contextual features to predict issue lifetime in github projects. In 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR), pages 291–302, 2016. 42. A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Transactions on Software Engineering, 32(12):971–987, 2006. 43. R. Krasniqi. Extractive summarization of related bug-fixing comments in support of bug repair. 2021 IEEE/ACM International Workshop on Automated Program Repair (APR), pages 31–32, 2021. 44. A. Kukkar and R. Mohana. A supervised bug report classification with incorporate and textual field knowledge. Procedia Computer Science, 132:352–361, 01 2018. 45. L. I. Kuncheva and C. J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51(2):181–207, 2003. 46. X. Li, H. Jiang, D. Liu, Z. Ren, and G. Li. Unsupervised deep bug report summarization. 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC), pages 144–14411, 2018. 47. Z. Li, Y. Yu, G. Yin, T. Wang, Q. Fan, and H. Wang. Automatic classification of review comments in pull-based development model. pages 572–577, 07 2017. 48. Z. Li, Y. Yu, G. Yin, T. Wang, and H. Wang. What are they talking about? analyzing code reviews in pull-based development model. Journal of Computer Science and Technology, 32:1060–1075, 2017. 49. Z. Li and H. Zhong. Revisiting textual feature of bug-triage approach. 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1183–1185, 2021. 50. LiGRE Software. Ligre: Qualitative data analysis software. https://ligresoftware.com/. Accessed: March 2026. 51. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach, 07 2019. 52. T. Merten, M. Falis, P. Hübner, T. Quirchmayr, S. Bürsner, and B. Paech. Software feature request detection in issue tracking systems. In 2016 IEEE 24th International Requirements Engineering Conference (RE), pages 166–175, 2016. 53. A. Meyer, T. Fritz, G. Murphy, and T. Zimmermann. Software developers’ perceptions of productivity. 2014. 54. R. Mihalcea and P. Tarau. Textrank: Bringing order into text. In Empirical Methods in Natural Language Processing, page 404–411, 2004. 55. C. Miller, S. Cohen, D. Klug, B. Vasilescu, and C. Kstner. Did you miss my comment or what?; understanding toxicity in open source discussions. In 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), pages 710–722, 2022. 56. S. Mukhtar and S. Lee. A combination of classification and summarization techniques for bug report summarization. In Proceedings of the 2nd International Workshop on Intelligent Software Engineering (ISE 2023), volume 3655 of CEUR Workshop Proceedings, Seoul, South Korea, 2023. CEUR-WS.org. 57. A. Murgia, M. Ortu, P. Tourani, B. Adams, and S. Demeyer. An exploratory qualitative and quantitative analysis of emotions in issue report comments of open source systems. Empirical Softw. Engg., 23(1):521–564, feb 2018.

56

12

DECLARATIONS

58. M. I. Nawaz Tarar, F. Ahmed, and W. H. Butt. Automated summarization of bug reports to speed-up software development/maintenance process by using natural language processing (nlp). In 2020 15th International Conference on Computer Science and Education (ICCSE), pages 483–488, 2020. 59. A. Nurwidyantoro, M. Shahin, M. Chaudron, W. Hussain, R. Shams, H. Perera, G. Oliver, and J. Whittle. Human values in software development artefacts: A case study on issue discussions in three android applications. Information and Software Technology, 141, 09 2021. 60. M. Ortu, T. Hall, M. Marchesi, R. Tonelli, D. Bowes, and G. Destefanis. Mining communication patterns in software development: A github analysis. In Proceedings of the 14th International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE’18, page 70–79, New York, NY, USA, 2018. Association for Computing Machinery. 61. Y. Paing, T. Vélez, and R. Khatchadourian. Quertci: A tool integrating github issue querying with comment classification. 02 2022. 62. X. Peng, P. Zhou, J. Liu, and X. Chen. Improving bug triage with relevant search. In International Conference on Software Engineering and Knowledge Engineering, 2017. 63. R. Pennington and B. Tuttle. The effects of information overload on software project risk assessment. Decision Sciences, 38:489–526, 2007. 64. N. Pingclasai, H. Hata, and K.-i. Matsumoto. Classifying bug reports to bugs and other requests using topic modeling. In 2013 20th Asia-Pacific Software Engineering Conference (APSEC), volume 2, pages 13–18, 2013. 65. J. Polpinij. A method of non-bug report identification from bug report repository. Artif. Life Robot., 26(3):318–328, aug 2021. 66. H. Qin and X. Sun. Classifying bug reports into bugs and non-bugs using lstm. In Proceedings of the 10th Asia-Pacific Symposium on Internetware, Internetware ’18, New York, NY, USA, 2018. Association for Computing Machinery. 67. F. Rahutomo, T. Kitasuka, and M. Aritsugi. Semantic cosine similarity. 2012. 68. S. Rastkar, G. C. Murphy, and G. Murray. Automatic summarization of bug reports. IEEE Transactions on Software Engineering, 40(4):366–380, 2014. 69. M. Rath and P. Mäder. Request for comments: Conversation patterns in issue tracking systems of open-source projects. In Proceedings of the 35th Annual ACM Symposium on Applied Computing, SAC ’20, page 1414–1417, New York, NY, USA, 2020. Association for Computing Machinery. 70. S. E. Robertson and H. Zaragoza. The probabilistic relevance framework: BM25 and beyond. In Foundations and Trends in Information Retrieval, volume 3, pages 333–389. now Publishers, Inc., 2009. 71. J. Romano, J. D. Kromrey, J. Coraggio, J. Skowronek, and L. Devine. Exploring methods for evaluating group differences on the nsse and other surveys: Are the t-test and cohens d indices the most appropriate choices? In Proc. Annual meeting of the Southern Association for Institutional Research, page 51, 2006. 72. A. Sanei, J. Cheng, and B. Adams. The impacts of sentiments and tones in communitygenerated issue discussions. pages 1–10, 05 2021. 73. M. L. Siddiq and J. C. S. Santos. Bert-based github issue report classification. In 2022 IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE), pages 33–36, 2022. 74. J. Sillito, G. C. Murphy, and K. De Volder. Questions programmers ask during software evolution tasks. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering, SIGSOFT ’06/FSE-14, page 23–34, New York, NY, USA, 2006. Association for Computing Machinery. 75. R. W. Software. Jira ai summary & issue summariser. https://www. roguewombatsoftware.com/blog/jira-ai-summary-issue-summariser, 2025. Accessed: March 2026. 76. Z. Tao, J. Chen, H. Jiang, X. Luo, and X. Xia. Bug report enrichment with application of automated fixer recommendation. 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC), pages 230–240, 2017. 77. P. Terdchanakul, H. Hata, P. Phannachitta, and K. Matsumoto. Bug or not? bug report classification using n-gram idf. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 534–538, 2017.

57 78. B. Vasilescu, Y. Yu, H. Wang, P. T. Devanbu, and V. Filkov. Quality and productivity outcomes relating to continuous integration in github. Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015. 79. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. 80. G. Viviani, C. Janik-Jones, M. Famelis, X. Xia, and G. C. Murphy. What design topics do developers discuss? In Proceedings of the 26th Conference on Program Comprehension, ICPC ’18, page 328–331, New York, NY, USA, 2018. Association for Computing Machinery. 81. H. Wang and Q. Li. Effective bug triage based on a hybrid neural network. 2021 28th Asia-Pacific Software Engineering Conference (APSEC), pages 82–91, 2021. 82. B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, et al. Modernbert: Smarter, better, faster, longer — a modern bidirectional encoder for fast inference and long contexts. arXiv preprint arXiv:2412.13663, 2024. 83. D. H. Wolpert. Stacked generalization. Neural Networks, 5(2):241–259, 1992. 84. S. Xi, Y. Yao, X. Xiao, F. Xu, and J. Lu. An effective approach for routing the bug reports to the right fixers. Proceedings of the 10th Asia-Pacific Symposium on Internetware, 2018. 85. S. Xi, Y. Yao, X. Xiao, F. Xu, and J. Lu. Bug triaging based on tossing sequence modeling. Journal of Computer Science and Technology, 34:942 – 956, 2019. 86. X. Xia, D. Lo, Y. Ding, J. M. Al-Kofahi, T. N. Nguyen, and X. Wang. Improving automated bug triaging with specialized topic model. IEEE Transactions on Software Engineering, 43(3):272–297, 2017. 87. B. Yang, X. Wei, and C. Liu. Sentiments analysis in github repositories: An empirical study. In 2017 24th Asia-Pacific Software Engineering Conference Workshops (APSECW), pages 84–89, 2017. 88. W. Zhang, Y. Cui, and T. Yoshida. En-lda: An novel approach to automatic bug report assignment with entropy optimized latent dirichlet allocation. Entropy, 19:173, 2017. 89. Y. Zhang, Y. Sun, Y. Shi, S. Jiang, and G. Yuan. Btal: An imbalance software bug report triage approach based on bert-textcnn. Information and Software Technology, 183:107731, 2025. 90. Y. Zhao, S. Zhang, Y. Wu, Y. Sun, Y. Sun, D. Pei, C. Bansal, and M. Ma. Triage in software engineering: A systematic review of research and practice. arXiv preprint, 2511.08607, 2025. 91. L. Zhifang, W. Kun, Z. Qi, L. Shengzong, Z. Yan, and H. Jianbiao. Classification of open source software bug report based on transfer learning. Expert Systems, n/a(n/a):e13184, 2022. 92. Y. Zhou, Y. Tong, R. Gu, and H. Gall. Combining text mining and data mining for bug report classification. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 311–320, 2014.

Related documents

• LabelMate: An LLM-Driven Framework for Refined Issue Report Labeling#639735
• FGDM: Reasoning Aware Multi-Agentic Framework for Software Bug Detection using Chain of Thought and Tree of Thought Prompting#141570
• Dissecting Bug Triggers and Failure Modes in Modern Agentic Frameworks: An Empirical Study#6032
• VulnSAGE: Adjudicating SAST Reports by Connecting LLM Reasoning to Program Structure#1387324
• VulnSAGE: Adjudicating SAST Reports by Connecting LLM Reasoning to Program Structure#1374957