Characterizing the Usefulness of Code Review Comments in Scientific Software for Software Quality and Scientific Rigor Sharif Ahmed
Nasir U. Eisty
University of Central Arkansas Conway, AR, USA [email protected]
University of Tennessee Knoxville, TN, USA [email protected]
ABSTRACT
1
Context: Innovation thrives on scientific software, with useful code review feedback enhancing its correctness and impact. However, unlike general-purpose commercial and open-source software, the usefulness of code review feedback (CR comment) in scientific software remains largely unstudied. Objective: This paper aims to characterize the usefulness of CR comment in scientific opensource software (Sci-OSS), leveraging existing research on useful CR comment. Method: To achieve this objective, we mine successful Sci-OSS from GitHub, analyze their CR comments with usefulnessrelated features, and compare the findings from prior research on general-purpose commercial and open-source CR comments. Results: The investigation on the usefulness of CR comments in SciOSS confirms many characteristics that prior research identified in general-purpose software. For example, subjective or negative CR comments remain not useful for the Sci-OSS. We also find CR comments which receive negative emoji reactions have a very small correlation with not useful comments, whereas the positive emojis show mixed correlations. Importantly, 6-33% CR comments in Sci-OSS are not useful in our mined repositories. Conclusions: Our investigation into Sci-OSS extends findings from CR comments’ usefulness research on general-purpose software, benefiting developers, scientists, and researchers in the Sci-OSS community.
Modern Code Review (MCR), online asynchronous tool-based source code inspection technique, is crucial despite its significant time costs [13]. According to Bosu et al. [7], developers spend at least a tenth of their time engaging in peer code review activities in both industry and open-source development. Given its essential role, the code review process is beneficial in enhancing overall team efficiency and project quality. The core of the MCR process lies in the CR comments [13].1 CR comments are different from source-code comments and commit messages that are written by code-authors instead of code-reviewers. The effectiveness of the code review process is heavily dependent on the usefulness of these CR comments. Usefulness of CR comments also emerged in Kononenko et al. [18]’s empirical study on the Mozilla code review process. Despite their importance, empirical findings from Bosu et al. [8] reveal that more than 1 out of 3 CR comments at Microsoft are not useful.
CCS CONCEPTS • Software and its engineering ! Open source model; • Information systems ! Open source software.
KEYWORDS Scientific Software, Open Source Software, Code Review Comment, Usefulness, Software Quality ACM Reference Format: Sharif Ahmed and Nasir U. Eisty. 2026. Characterizing the Usefulness of Code Review Comments in Scientific Software for Software Quality and Scientific Rigor . In Proceedings of The Platform for Advanced Scientific Computing Conference (PASC 2026). ACM, New York, NY, USA, 12 pages. https://doi.org/XXXXXXX.XXXXXXX
PASC 2026, June 29 – July 1, 2026, Bern, Switzerland 2026. ACM ISBN 978-1-4503-XXXX-X/18/06 https://doi.org/XXXXXXX.XXXXXXX
INTRODUCTION
Given enough eyeballs, all bugs are shallow —Linus Torvalds [28] This well-known quote captures the core idea behind open source software (OSS): when many people can view, test, and improve code, problems are more likely to be found and fixed quickly. OSS relies on collaboration, transparency, and community involvement to ensure software quality. While many open source tools are built for general use, such as web browsers, operating systems, or development frameworks, some are created specifically to support scientific research. These are known as scientific open-source software (SciOSS). They help researchers analyze data, run experiments, and test new ideas. Although both types of software are open and collaborative, they serve different goals, follow different development practices, and are used by different communities. Moreover, different people, groups, and organizations write and maintain the Sci-OSS. These people may have different expertise, backgrounds, and practices. However, researchers have investigated the usefulness of code review feedback or code review comments (CR comments), identified factors to classify usefulness of CR comments [8, 31]. We also featurized, predicted, compared and contrasted the usefulness of CR comments between commercial and open-source general software in our previous study [3]. Similar to general software, code review is crucial in Sci-OSS because it helps ensure the accuracy, reliability, and trustworthiness of computational results, which are fundamental to scientific progress [16]. Given that Sci-OSS often involves complex algorithms and data analysis, peer review of code allows developers to identify and correct errors, improve readability, and enhance maintainability. This collaborative process not only reduces the likelihood of bugs 1“CR comments” and “Code Review Comments” are interchangeably used in this paper
PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
and inaccuracies in scientific outputs but also promotes knowledge sharing among developers. As a result, implementing systematic peer code review practices contributes to producing high-quality Sci-OSS that can be trusted by the scientific community and supports reproducible and robust scientific outcomes. To the best of our knowledge, none of the prior research has framed the usefulness of CR comments in Sci-OSS. This observation motivates us to look into Sci-OSS, leveraging state-of-the-art techniques for identifying and analyzing CR comments. This paper aims to investigate the usefulness of CR comments in Sci-OSS. It complements our previous work [3] and other existing research conducted in commercial and open-source CR comments [24, 27]. Analyzing the state-of-the-art estimated usefulness of CR comments in Sci-OSS via feature analysis and eXplainable AI (XAI) can inform us what makes a CR comment useful in Sci-OSS. The outcome of the proposed work is based on addressing the following research questions: • To the best of our knowledge, there is no dataset or experiment on usefulness of CR comments in Sci-OSS. So, we ask first research question, RQ1: How useful are the Sci-OSS CR comments when evaluated with state-of-the-art usefulness prediction models? • We have found studies that characterize and describe usefulness of CR comments in general software [3, 8, 31]. To describe the CR comments in Sci-OSS leveraging usefulness research we ask our second research question, RQ2: How do the characteristics of CR comments in Sci-OSS differ from those of useful CR comments in commercial and open-source? • Our recent work [3] observed bipolarity of usefulness characteristics in CR comments from different environment. Therefore we pose our third research question, RQ3: Do CR comments in Sci-OSS projects vary in characteristics across different scientific domains? • Our other work [2] found emoji-features help predicting usefulness of CR comments better, thus we explore our last research question, RQ4: How are emojis perceived in CR comments within Sci-OSS? The rest of the paper is organized as follows. Section 2 provides background and a brief overview of the literature. Section 3 details the methodology for answering our research questions, RQ 1-4. Section 4 reports and discusses our experiment results. Section 5 reviews the threats to the validity of our work, and Section 6 concludes.
2
BACKGROUND AND RELATED WORK
This section provides background and a concise review of the relevant literature.
2.1
Usefulness Definition
In 2014, Pangsakulyanont et al. [25] proposed a definition for CR comments, classifying them as either useful or useless based on how closely the reviewer’s comment matched the author’s commit message for a specific code change. If it was not possible to clearly determine the usefulness of a comment, it was labeled as undetermined. The following year, Bosu et al. [8] studied CR comments at
Sharif Ahmed and Nasir U. Eisty
Microsoft, where seven developers labeled them as useful, somewhat useful, or not useful. For their usefulness prediction model, somewhat useful comments were grouped with useful ones. A CR comment was considered useful if it led to a code change within 1–10 lines of the comment. They found that changes one line away minimized both false positives and false negatives. This definition was later adopted [24, 27] or adapted [31] in three datasets.
2.2
Classifying Usefulness
Initially, Pangsakulyanont et al. [25] employed the Vector Space Model with cosine similarity measure between the commit message and the CR comment on a code-change to classify usefulness. Subsequently, Bosu et al. [8] identified the factors of useful CR comments through an empirical study at Microsoft. To generate features for their usefulness classifier, they utilized textual properties of CR comments and attributes of review activities. Kononenko et al. [19] introduced factors, such as code author and reviewer experience, that impact the quality of code reviews. Rahman et al. [27] collected CR comments from commercial projects and developed a model to predict their usefulness. Their approach involved using textual properties of the CR comments and features related to the developers’ experience. However, unlike Bosu et al. [8], they did not incorporate any features related to the code review activities. In 2018, Efstathiou and Spinellis [14] proposed measuring usefulness using linguistic semantics to address this issue. Concurrently, Meyers et al. [24] used several linguistic features to classify CR comments. Hasan et al. [17] developed an in-house web-based application to reward reviewers for their useful feedback. They did not consider any linguistic features from Meyers et al. [24]. However, they extended the existing features by incorporating features related to code review activities. Furthermore, they presented several features for predicting the usefulness of CR comments from the literature and their work, but these features are limited to their model. For the feature-based usefulness prediction task, Rahman et al. [27] utilized the Mann-Whitney Wilcoxon test to identify differences in features between useful and non-useful CR comments, and Cohen’s D as an effect-size measure. In contrast, Hasan et al. [17] used the Pearson Correlation measure to interpret their input features. Turzo and Bosu [31] employed the Chi-Square test to check the relationship between respondents’ demographics and responses. Recently, we [3] analyzed the features by combining feature analyses from prior works [17, 27]. In that work, our comprehensive feature analyses on cross-datasets uncovered a bipolarity between datasets of CR comments from two different environments.
2.3
Emojis and Usefulness
Emojis and emoticons serve as non-verbal cues and convey emotive or instructive messages to developers. In one of our previous studies [2], we explored the utility of CR comments by examining the sentiments and semantics of emojis within these comments using existing datasets DA⌘ ,D22 , and D>3 . In that paper, emojiaware models outperform the emoji-unaware models. Therefore, overseen emojis in CR comments emerged as an important aspect of CR comments’ usefulness prediction task.
Characterizing the Usefulness of Code Review Comments in Scientific Software for Software Quality and Scientific Rigor PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
2.4
General CR comments
In another of our previous works [1], we identified three available datasets of useful CR comments from general software. We briefly describe them below. • DA⌘ , 2017: This dataset [27] comprises 1,481 CR comments, collected through the GitHub API from four commercial projects of an unspecified company. The authors of the article manually annotated each CR comment as either useful or non-useful, based on the usefulness heuristic of the Microsoft study [8]. • D22 , 2018: This dataset [24] has 3,794 CR comments from Google Chromium Project. The authors used the RESTful API to obtain accessible CR comments on Rietveld (code review tool used in Google Chromium Project) from 2008 to 2016. Next, they automatically identified acted-upon CR comments by considering Rietveld’s “Done” click feature. Though their annotation principle is similar to the DA⌘ [27], it was labeled as acted-upon (i.e., useful) and not-(known to be)-acted upon CR comments. • D>3 , 2023: This dataset [31] contains 2,654 CR comments from OpenDev’s Nova OSS project. The authors used the Gerrit Miner tool for mining CR comments from 2011 to 2022 and manually annotated CR comments with useful and not-useful labels. In this paper, we use all three general-purpose commercial and open-source CR comment datasets (DA⌘ ,D22 ,& D>3 ) to compare our CR comments derived from SciOSS, DB28 .
2.5
male, mostly male, female, mostly female, androgynous, or unknown. After determining the guessed gender, they transformed “mostly male” and “mostly female” into male and female respectively, and unpredicted genders into the “unknown” category. Normalized Data. Ahmed et al. [4] observed the GitHub users’ location data and found that a specific location had multiple values for example ‘US’, ‘USA’, ‘U.S.’, or ‘United States’. Additionally, some GitHub users shared city names without their country names, for example, ‘London’ instead of ‘London, United Kingdom.’ To unify the geo-location of the developers, they used GeoPy [20], an opensource geocoding python client, to obtain a full address from the OpenStreetMap Nominatim geocoder. Lastly, for maximum coverage, they took only the country name from the obtained address as location data for every developer who publicly shared their location information at the time of our GitHub data mining. We derive usefulness-related features from literature [3, 8, 24, 27] to characterize the usefulness of these SciOSS CR comments and contribute DB28 (as detailed in Section 3).
3
METHODOLOGY
Here, we outline our proposed methodology to answer our research questions, RQ1-4, introduced in Section 1. Figure 1 shows the proposed approach for this research objective. As our focus is on the scientific projects, we first select the relevant projects. Next, we collect the data from selected projects. We then answer our research questions through our feature and XAI analyses. We detail each phase below.
SciOSS CR comments
The U.S. Department of Energy2 (DoE) sponsored CASS3 fosters collaboration among diverse Software Stewardship Organizations (SSOs), each responsible for advancing parts of the Sci-OSS ecosystem (e.g., math libraries, data tools). Ahmed et al. [4] explored the scientific open-source software (e.g., math libraries, data tools) and mined 10 projects that are open and active on GitHub. In the process of project selection, authors excluded the projects that are mirrored on GitHub from social coding or other platforms. Next, authors mined the GitHub data using GitHub REST API. Their dataset contain other information in addition to CR comments, such as, pull request number, line number, author association, user type, and GitHub emoji reactions ( ). We name these additional data related to the mined CR comments as metadata throughout this paper. To make our work self-contained, we provide brief overview of approximated and normalized metadata [4], which we use to answer one of our research questions. Approximated Data. GitHub user profiles do not have specific attributes for the developer’s gender, but gender diversity has been studied and found to be an important metric for community engagement and software sustainability. Ahmed et al. [4] employed an open-source gender-guessing technique [12] that is able to predict gender nearly 98% of the time correctly [29]. The tool is able to predict a person’s gender from their name using the categories 2 http://science.energy.gov 3 www.cass.community
Figure 1: Overview of Our Methodology to Answer RQs 1-4
3.1
DB28 Dataset Curation
We adopt CR comments from SciOSS [4], compute usefulness-related hand-crafted features from these CR comments, and answer our research questions RQ1-4. 3.1.1 Feature Data Computation. After collecting the data, we compute the feature values using existing tools and artifacts from our prior work and related research[1–3, 8, 24, 27]. With the obtained data, we compile a dataset for our experiments and contribute it to future research. Importantly, as we are interested in studying the usefulness of scientific CR comments and we have state-of-the-art usefulness predictors, we plan to obtain the predictor-based annotation of our derived scientific CR comments. This will facilitate us to characterize the scientific CR comments.
PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
3.1.2 Prediction-based Usefulness Annotation. This paper aims to investigate scientific CR comments within the context of code review feedback usefulness and characterize scientific CR comments leveraging features and analyses applied for understanding the CR comments. As we have a state-of-the-art usefulness predictor from our previous work [3], we plan to obtain a predictor-based annotation of our derived Sci-OSS CR comments. This paper shows OpenAI’s GPT-4o as the best-performing model to predict the usefulness of code review comments. As GPT-4o is not free and our derived code reviews are many (164,708 CR comments) compared to their adopted datasets (<4,000 CR comments), we will need to pay for the fine-tuning process and then for predicting our scientific code review text. To achieve this, we choose the second-best performing model from our paper [3], which is free of cost and demands less computational time. Finally, we use the usefulness prediction model [3] to predict the usefulness of our derived Sci-OSS CR comments.
3.2
Answering Research Questions
We leverage the experiments from prior research [2, 3, 17, 27] to answer the research questions for scientific CR comments. 3.2.1 RQ1: How useful are the Sci-OSS CR comments when evaluated with state-of-the-art usefulness prediction models? To answer this question, we check the distributions of predicted usefulness of CR comments from our compiled dataset, DB28 . For better insight, we plot the aggregated usefulness ratio of the individual projects within DB28 over the years. 3.2.2 RQ2: How do the characteristics of CR comments in Sci-OSS differ from those of useful CR comments in commercial and opensource? Conducting feature and XAI analyses from prior CR comments analyses [3, 27] on Sci-OSS projects can describe the CR comments in Sci-OSS. Thus, we perform the existing experiments with our newly obtained scientific CR comments.
A. For within-dataset feature analyses, we follow the following process. • We adopt around fifty textual features from our previous works [2, 3], which includes prior works [1, 8, 17, 24], in our experiment. • Next, we preprocess and calculate all the feature values of our DB28 CR comments using our previous implementations and details [2, 3]. • With the computed feature values for each CR comment, we determine whether each of the individual features can distinguish auto-annotated usefulness. To this end, we employ existing statistical measures, the Mann-Whitney U test [23] and Cohen’s D [10], that were used for inspecting usefulnessrelated feature analysis [3, 27]. • We compare and contrast the statistical findings from our scientific CR comments with findings from general-purpose commercial and open-source CR comments. B. For cross-dataset feature analyses, we adhere to the following steps, similar to our previous study [3]. While our within-dataset feature analysis describes the individual feature’s ability to distinguish usefulness within a single dataset, the cross-dataset feature
Sharif Ahmed and Nasir U. Eisty
analysis evaluates whether the correlation between features and usefulness statistically differs from one dataset to another. • Compute Pearson correlation of all the features to the predicted usefulness labels for all datasets. • Take feature correlations on our dataset versus feature correlations on each of the existing datasets and perform the Wilcoxon signed-rank test [32] followed by Cohen’s D [10] effect size measure. We have noticed the bipolarity of two open-source datasets in our prior work’s empirical results [3]. This answer from the Sci-OSS will inform us if the Sci-OSS has similar polarity or not. C. For cross-dataset eXplainable AI (XAI) analysis, we adopt the recent XAI analyses on CR comment-usefulness datasets [3]. This approach provide explanation of a dataset when trained on an explainer AI model. To this end we take following steps. • We consider robust SHapley Additive exPlanation framework [22] to train one of the dataset on a linear explainer model and get explanation while predicting another dataset. • Train the selected explainer model with our curated DB28 dataset and check explanation from predicting usefulness of existing datasets (DA⌘ , D22 , D>3 ) and vice versa. • The obtained SHapely values will provide what contributed in explainer model to predict a CR comment useful or notuseful. • The XAI findings from scientific CR commentsof DB28 will complement the findings of general purpose CR comments from commercial DA⌘ and open-source D22 and D>3 datasets. 3.2.3 RQ3: Do CR comments in Sci-OSS projects in Sci-OSS projects vary in characteristics across different scientific domains? To answer this RQ, we select our derived CR comments in DB28 that come from similar and different scientific domains, such as applied math, programming systems, machine learning. Next, we compare their usefulness and usefulness-feature values using statistical tests following our experiment setup for cross-dataset analysis in RQ2 (Section 3.2.2). Here, we compare the projects within DB28 only. As our dataset has additional metadata from the Sci-OSS repositories (Sec 2.5), we compare and contrast these metadata in terms of CR comment usefulness (Sec 3.1.2). Here, we conduct statisticaltests for feature data (described in Section 3.2.2) on these metadata. We also consider Pearson correlation (X) between these numeric metadata of CR comments and usefulness CR comments. Since some of these metadata are categorical, we consider Chi-Square (j 2 ) [26] instead of Mann-Whitney U and Cramér’s + [11] instead of Cohen’s D and Pearson correlation. Answers to this will provide an additional snapshot of the selected Sci-OSS projects. 3.2.4 RQ4: How are emojis perceived in CR comments within SciOSS?. We examine the emoji sentiments and emoji semantics within our derived Sci-OSS CR comments. CR comment-usefulness literature informs us that emoji semantics are perceived differently in useful CR commentsfor general purpose open and commercial software [2]. • We examine which emojis are used in scientific CR comments comparing the literature.
Characterizing the Usefulness of Code Review Comments in Scientific Software for Software Quality and Scientific Rigor PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
• Next, we preprocess, extract, and normalize emoticons, emojiexpressions, and Unicode emojis to a single format, Unicode emojis. • With unified emojis, we obtain emoji sentiment features and emoji semantic embeddings using Novak’s general sentiment [21], Ahmed & Eisty’s code review sentiments [2], and emoji2vec pre-trained language model [15]. • We then check the features’ correlation to usefulness in emoji-containing CR comments in DB28 . • Finally, we contrast the emoji not-aware CR comment-semantics (fastText [6]) and emoji aware CR comment-semantics (fastText [6]+emoji2vec [15]) for predicting the usefulness of CR comments. We maintain the existing stratified 10-fold crossvalidation setup for this evaluation [2]. Since our dataset has prediction-based annotation, we limit our analysis for prediction-evaluation to DB28 . We additionally employ a statistical paired comparison between the DB28 ’s annotated usefulness and emoji-(non)-aware models’ CR comment-usefulness predictions. For this measure, we consider Wilcoxon’s Signed Rank test [32] with a threshold of p-value< 0.05, followed by Cohen’s D [9] effect-size measure. This research question will extend our knowledge by adding insights for emojis within the context of the Sci-OSS code review feedback.
4
RESULTS AND DISCUSSIONS
This section provides the experimental results corresponding to the procedures described in Section 3 and offers analysis of findings.
Figure 2 shows the yearly aggregated ratio of useful comments of the scientific projects. Most of the projects had 70-80% useful CR comments. The project P5 has only one datapoint in the figure with 100% useful CR comments. After looking into this, we find that this project has only 19 CR comments and all of them are from the year 2023. For P8, the useful comments ratio in year 2021 is the lowest, 50% useful from only 10 CR comments. However, it has 80% useful CR comments in the following years, 2023 with 32,205 CR comments and 2024 with 71,150 CR comments.
4.2
RQ2: Useful CR comments: General vs Sci-OSS
4.2.1 Features. Table 1 shows statistical properties of scientific CR comments from our DB28 along with all three existing usefulness labeled CR comments-datasets, DA⌘ , D22 , and D>3 , from general software. Ahmed & Eisty discovered a bipolarity between D22 and D>3 , so we were curious about DB28 ’s stance, and we find that DB28 is statistically closer to D22 . Overall, we see that CR comments from Sci-OSS has a stronger association with usefulness for code review sentiment, cr_senti, [5] and association with not-usefulness for stop-word-ratio, is confirmatory? than general open-source and commercial software. 4.2.2 Datasets. Table 2 shows that all the projects are significantly different from each other, with Wilcoxon’s signed rank pvalue<0.05, except for DB28 -DA⌘ . The Cohen’s D effect-size is also shows DB28 strongly varies with D>3 , similar to DA⌘ and D22 . 4.2.3 eXplainable AI. Figure 3 shows the words that contributed to predicting usefulness of CR comments from scientific DB28 to general DA⌘ , D22 , and D>3 , and vice versa. This extends the findings from the recent XAI findings [3]. In our XAI analysis, we find that our XAI analysis confirms prior works regarding the contributions of the words line, we, you, good, should, test, etc. We also don’t notice the docstring, API, instance, method, case words that had contributed to XAI analysis across general software in prior work [3]. Therefore, CR comments that specify the source-code line, avoid subjective feedback, and talk about testing are also helpful in Sci-OSS. Following the excerpting of prior works [3, 30], we explore DB28 comments labeled as useful and present a few excerpts below.
Here, y-axis shows yearly aggregated ratio (0–1.0) of useful CR comments across 10 SciOSS projects (2014-2024) Figure 2: Useful-CR Comments in Scientific Software, DB28
4.1
RQ1: Usefulness of CR comments in Sci-OSS
We find 79% of our derived 164,708 CR Comments in DB28 are useful with our prediction based useful annotation described in Section 3.1.2. The project-wise ratio of not-useful CR comments in DB28 ranges from 6% to 33%. This mirrors the findings of a study conducted a decade ago at Microsoft [8], which reported 34.5% of CR comments were not-useful.
• ‘check history of this line’ • ‘This is not the focus of this PR. but the indentation of this line is confusing.’ • ‘nit: trivial braces can be omitted’ • ‘Presumably this line will change depending on the Fortran compiler used. Users will just have to be aware of that.’ • ‘Yep, there should be a `xxxx.c` which needs the cuda header instead’ • ‘ Use xxxx_error_w_msg instead.’ • ‘@xx @xx I'd avoid including `.hpp` in a `.h` but instead include in `.c` that needs CUDA’ • ‘acknowledged.’ • ‘unresolving, @xx should take another look here and acknowledge the solution explicitly’
PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
Sharif Ahmed and Nasir U. Eisty
Table 1: Statistical Properties of Features in DB28 CR comments, Comparing DA⌘ , D22 , and D>3 from [3] Here, effect-sizes denoted by ⇤ are statistically significant (p-value<0.05) effect-sizep-value Pearson Correlation(X) Dataset! DB28 DA⌘ D22 D>3 DB28 DA⌘ D22 D>3 tone 0.03⇤ 0.05 0.17⇤ 0.46⇤ 0.01 -0.03 0.07 -0.17 ⇤ distress 0.01 0.01 0.11⇤ 0.23⇤ 0.00 0.00 0.04 -0.09 empathy 0.01⇤ 0.01 0.11⇤ 0.23⇤ 0.00 0.00 0.04 -0.09 is_toxic 0.01 0.06 0.02 0.19⇤ 0.00 -0.03 0.01 -0.07 gratitude 0 0.01 0.24⇤ 0.15⇤ 0.00 -0.01 -0.10 -0.06 cr_senti 0.29⇤ 0.12⇤ 0.19⇤ 0.09 0.12 0.06 0.08 -0.04 kw_refact_xerox 0.15⇤ 0.04 0.17⇤ 0.09 0.06 0.02 0.07 -0.04 num_propernouns 0.01⇤ 0.09 0.02 0.05 0.00 -0.04 0.01 -0.02 num_interjections 0 0.04 0.07 0.04 0.00 0.02 -0.03 -0.01 polarity 0.27⇤ 0.04 0.25⇤ 0.03 -0.11 -0.02 -0.10 -0.01 kw_msoft_u 0.21⇤ 0.07 0.38⇤ 0 0.09 0.03 0.16 0.00 num_exclamation 0.01⇤ 0.01 0.02 0.01 0.01 -0.01 -0.01 0.00 is_confirmatory 0.22⇤ 0.04 0.07 0.02 0.09 0.02 -0.03 0.01 question_ratio 0.19⇤ 0.2⇤ 0.51⇤ 0.03 -0.08 -0.10 -0.21 0.01 kw_satd_potdar 0.02⇤ 0.06 0.1⇤ 0.04 0.01 -0.03 0.04 0.02 implicature 0.09⇤ 0.01 0.06⇤ 0.06 -0.04 0.00 -0.02 0.02 kw_msoft_nu 0.21⇤ 0.1⇤ 0.26⇤ 0.06 -0.09 -0.05 -0.10 0.02 ⇤ code_word_ratio 0.2 0.15⇤ 0.23⇤ 0.06⇤ 0.08 0.07 0.10 0.02 politeness 0.14⇤ 0.12 0.19⇤ 0.06⇤ -0.06 -0.06 -0.08 0.02 num_Qmark 0.12⇤ 0.18⇤ 0.52⇤ 0.1⇤ -0.05 -0.09 -0.21 0.04 ⇤ ⇤ rd_text 0.21 0.08 0.19 0.14⇤ -0.08 -0.04 -0.08 0.06 kw_refact_problem 0.01 0.09 0.09 0.16⇤ -0.01 0.05 0.03 0.06 avg_chars 0.2⇤ 0.11⇤ 0.16⇤ 0.16⇤ 0.08 0.05 0.07 0.06 formality 0.1⇤ 0.01 0.15⇤ 0.18⇤ 0.04 0.00 0.06 0.07 subjectivity 0.36⇤ 0.15⇤ 0.38⇤ 0.18⇤ -0.15 -0.07 -0.15 0.07 has_out_snippet 0.04⇤ 0.02 0.09⇤ 0.2⇤ 0.02 -0.01 -0.04 0.08 kw_secdev 0.12⇤ 0.01 0.31⇤ 0.24⇤ -0.05 -0.01 -0.13 0.09 avg_punct 0.1⇤ 0.12⇤ 0.04 0.25⇤ 0.04 0.06 0.02 0.10 ⇤ num_sent 0.21 0.02 0.46⇤ 0.25⇤ -0.09 0.01 -0.19 0.10 stop_word_ratio 0.37⇤ 0.02 0.24⇤ 0.26⇤ -0.15 -0.01 -0.10 0.10 num_tentative 0.17⇤ 0.08 0.23⇤ 0.27⇤ -0.07 -0.04 -0.10 0.10 informativeness 0.06⇤ 0.02 0.07 0.28⇤ 0.02 0.01 0.03 0.11 ⇤ programming_words 0.02 0.02 0.12 0.31⇤ 0.01 -0.01 -0.05 0.12 num_adverb 0.27⇤ 0.01 0.39⇤ 0.32⇤ -0.11 -0.01 -0.16 0.12 num_adj 0.13⇤ 0.16⇤ 0.36⇤ 0.33⇤ -0.05 -0.08 -0.15 0.12 num_nouns 0.06⇤ 0.07 0.39⇤ 0.35⇤ -0.02 -0.04 -0.16 0.13 ⇤ num_determinants 0.26 0.09 0.43⇤ 0.37⇤ -0.11 -0.04 -0.17 0.14 ⇤ num_chars 0.11 0.03 0.46⇤ 0.39⇤ -0.05 -0.02 -0.18 0.15 num_words 0.2⇤ 0.07 0.5⇤ 0.39⇤ -0.08 -0.03 -0.20 0.15 ⇤ avg_stopwords 0.29 0.1 0.38⇤ 0.4⇤ -0.12 -0.05 -0.15 0.15 ⇤ ⇤ kw_refact_solution 0.19 0.01 0.41 0.41⇤ -0.08 0.00 -0.17 0.16 ⇤ ⇤ num_verb 0.16 0.03 0.38 0.41⇤ -0.07 0.02 -0.15 0.16 avg_words 0.11⇤ 0.08 0.32⇤ 0.44⇤ -0.04 -0.04 -0.13 0.17
Characterizing the Usefulness of Code Review Comments in Scientific Software for Software Quality and Scientific Rigor PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
Table 2: Comparisons among Features’ Correlations to the Usefulness Here, ⇤ indicates statistically significant (p-value<0.05) Cohen’s D effect-size
p-value
D22
D22
D>3
1.28
D>3
* *
*
DB28
*
*
DA⌘
DA⌘
DB28
1.08
1.28
0.64
0.51 0.24
-
4.3
RQ3: Useful CR comments in Different Scientific Domains
We report analysis from both features’ data, computed from the CR comment, and the metadata, derived from the GitHub repository attributes associated with CR comments, from our compiled DB28 .
Metadata. Table 3 shows the statistical relationship of CR comment-metadata in terms of distinguishing our prediction based useful/not-useful annotations. The column ‘all’ indicates the entire dataset DB28 and P1-10 are ten GitHub SciOSS projects. Though categorical author association and country metadata could distinguish usefulness for individual projects with small effect sizes, overall, they could not distinguish for the entire dataset, DB28 . On the other hand, approximated gender metadata shows a very small but consistent association with CR comment-usefulness across machine learning, applied mathematics, and programming systems projects, P1-10. In Table 3, we also notice non-categorical metadata start line moderately correlates with not-useful, whereas line, position with useful CR comments. The emoji reactions show a small correlation in P1 and P7. We also notice that emoji reactions: , , , and in P8 can statistically distinguish the usefulness of CR comments. Conversely, only in P10 can statistically distinguish the usefulness. As we find a small but consistent correlation for gender, country, and author association, developers should be inclusive during code review activities regardless of other participants’ origin, identity, and association.
Feature-data. Even if the DB28 shows small correlation in Table 1, the individual projects show medium correlation to many features in Table 4. We find negative correlations for subjectivity, polarity, and politeness features and positive correlations for CR-sentiment, formality, informativeness, and codeword ratio in P1. The rest of the projects (P2-10) are with similar or lower correlations. While looking at the type of anonymous projects in DB28 , we find machine learning related Sci-OSS (P1) is showing stronger feature relations with usefulness than the rest of the applied mathematics (P2-4) and programming systems (P5-10) projects. Since machine learning and deep learning subfields are more widely used by opensource developers, P1 might have a similar feature association to general-purpose CR comment(D22 -DA⌘ ). Additionally, we find in Table 4 that only P7 shows a small correlation with not-useful CR comment for features like refactoring problem keyword density, tone, and gratitude. This intrigues us to compare P7 versus P1-6;8-10 repositories. We find P7 is older than other projects; so, we suspect that the people, community, and culture are different than our other selected projects.
Figure 3: Interpreting Scientific and General CR comments using XAI
The excerpts above from DB28 have similar instructions or messages that are also useful in general CR comments, such as refactoring, suggestions, and nitpicking [3].
4.4 RQ4: Emojis and Usefulness in Scientific CR comments In our derived dataset DB28 , we find only 1,269 from 164,708 SciOSS CR comments that contain emojis. The < 1% presence of emojis within CR comments in DB28 is much lower than emojis in DA⌘ (2%), D22 (3%), and D>3 (2.5%). Though the presence of emojis in DB28 is fewer than general, Table 5 shows diverse emojis in CR comments from SciOSS. We find emojis in DB28 mostly overlapping with the emojis found in CR comments from commercial and open-source
PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
Sharif Ahmed and Nasir U. Eisty
Table 3: CR comment-Usefulness vs CR comment-GitHub-Metadata for Various Scientific Software in DB28 Here, effect-sizes denoted by ⇤ are statistically significant (p-value<0.05)
# Metadata (Categorical) DB28 -Projects! author_association country gender project side start_side
P1 .08⇤ .08⇤ .08⇤ 0⇤ 0⇤ 0
# Metadata (Numeric) DB28 -Projects! line original_line original_position original_start_line position pr_num react_+1 react_-1 react_confused react_eyes react_heart react_hooray react_laugh react_rocket react_total_count start_line
P1 .62 .56 .1 0 .63 .17 .4 0 0 0 0 0 0 0 .4 0
P10 .02 .07⇤ .03⇤ 0⇤ .03⇤ 0⇤
effect-sizep-value ! Cramér’s + Chi Square (j 2 ) P2 P3 P4 P5 P6 P7 P8 .03⇤ 0⇤ .03⇤ 0 .01⇤ .05 .03⇤ ⇤ ⇤ ⇤ ⇤ .09 0 .01 0 .06 0 .06⇤ .06⇤ .03⇤ .03⇤ 0 .02⇤ .13⇤ .02⇤ 0⇤ 0 0⇤ 0 0⇤ 0 0⇤ .02⇤ .05 .01⇤ 0 .04⇤ 0⇤ .04⇤ .02 .28⇤ .03⇤ 0 .07 0 .06⇤
P9 .01⇤ .03⇤ .05⇤ 0⇤ .02⇤ .02⇤
all .01 .06 .02⇤ .05⇤ .03⇤ .05⇤
P1 .08 .08 .08
P10 .02 .07 .03
P2 .03 .09 .06
P3 .00 .00 .03
Cramér’s + P4 P6 .03 .01 .01 .06 .03 .02
.00
.03 .00
.02 .02
.05 .28
.01 .03
P10 .03 .01 .05⇤ .12 .04 .01 .02 .08⇤ .05 .03 .01 .03 0 .01 .03 .11
effect-sizep-value ! Cohen’s DMann-Whitney U P2 P3 P4 P6 P7 P8 .03 .07 .06 0 .04 .88 .03⇤ .02 .02 .08⇤ 0 .01 .15 .01 .05⇤ .11⇤ .02 0 .03⇤ .21 0⇤ .07 .94 .1⇤ 0 .01 0 .01⇤ .01⇤ .12 .04 0 .06⇤ .48 .02⇤ .04⇤ .06 .02 0 .01 .04 0 .01 0 .03 0 .03 .17 .04⇤ 0 0 0 0 .01 0 .01 .01 0 .01 0 .02 0 .02⇤ .02 0 .02 0 .01 0 .03⇤ .03 0 0 0 .01 0 .03⇤ .02 0 .05 0 .01 0 0 .02 .04 0 0 .02 0 .01 .01 0 .02 0 .02 0 0 0 0 .02 0 .02 .17 .03⇤ .05 .99 .09 0 .1 0 .03
P9 .03 .02 .08⇤ .09 .1⇤ .01 .06⇤ .01 .01 0 .01 .02 .01 0 .07⇤ .05
all .03 .02 .01⇤ .03 .01⇤ .09⇤ .04⇤ .01⇤ .02⇤ .02⇤ .02⇤ 0 .01 0 .03⇤ .03
P1 .14 -.14 .03
P10 .01 .01 .02 .05 .02 .01 -.01 -.03 -.02 .01 .01 -.01 .00 .01 -.01 .05
P2 -.01 -.01 .02 .03 .01 -.02 -.01
P3 .03 -.01 -.05 -.37 -.06 .03 .00
general software. Interestingly, we did not find any customized GitHub emojis, or , from 164,708 Sci-OSS CR comments. However, we see a few new emojis with sparse presence, such as - (4) (3) (2) (2) (1) (1) (1) (1) (1) (1). Table 6 shows the metadata and features’ correlation with usefulness of CR comments on the entire dataset DB28 and emojicontaining CR comments in DB28 , denoted by DB28 2 . The notable differences in correlation we see for metadata are country, start side, and start line, and for features are polarity, subjectivity, stop word ratio, and emoji sentiment. Among these metadata and features, we see that only the start line has the most contrasting difference. Therefore, emoji-containing CR comments in DB28 are slightly different. Next, we look into the semantic similarity or difference between DB28 and DB28 2 . Table 8 shows stratified 10-fold cross-validation CR commentusefulness prediction performance with emoji-containing CR comments in DB28 , DB28 2 . Though emoji semantics helped in predicting the usefulness of CR comments better in general software [2], Table 8 shows that emoji-semantics-aware models could not predict usefulness better than emoji-semantics-non-aware models. With our additional statistical measure, we find that all approaches reject the Wilcoxon signed-rank null hypothesis. That is, the paired samples of the original and predicted usefulnesses are significantly different. Also, the Cohen’s D effect sizes resemble the results from traditional classification metrics. Ahmed & Eisty also found that emoji semantics performed poorly when DA⌘ 2 , D22 2 , and D>3 2 were unified for usefulness
.15 -.04 .10
.10
.01 -.01 .01 .01 .01 .01 .00 -.02
.02 .00 -.39
.04 .07
Pearson (d) P4 P6 -.03 .02 -.04 .00 -.01 .01 -.04 -.01 -.02 .03 -.01 .00 .01 .01 .00 .00 -.01 -.01 .00 .01 -.02 .00 .00 .01 -.01 -.01 .01 .01 -.04 .04
P7 .05
P8 .03 .06 .02
P9 .01 .03 .05
.00
.04 .06
.02 .02
P7 .31 .07 -.09 .27 .18 -.02 .07
P8 .01 .00 .00 .00 .01 .00 .02 -.01 -.01 -.01 -.01 .00 .00 .00 .01 .01
P9 -.01 .01 .03 .04 .04 .00 .03 .00 .00
.13
.07
.00 .01 .01 .03 .02
all .01 .06 .02 .05 .03 .05 all .01 .01 .00 .01 .01 .04 .02 .00 -.01 -.01 -.01 .00 .00 .00 .01 .01
prediction with semantics and 10-fold cross-validation. Since DB28 comprises 10 prominent SciOSS, we also infer that emojis are used and perceived differently within these 10 SciOSS projects. To verify our inference, we look up the emojis in DB28 2 and their associated automated usefulness labels (Section 3.1.2). We present 10 SciOSS projects of DB28 , the emojis in the CR comments, and their associated usefulness labels in Table 7. Within DB28 -projects P1-10, we see that P1 and P5 do not have emoji-containing CR comments. We also notice that most emojis belong to both useful and not-useful classes. For cross projects, we also find different distributions of emojis. Thus, the models learned different representations of the same emojis when emoji embedding was aggregated with text embedding. Though the emojis in SciOSS have a sparser presence and show variation in usage within and across our selected SciOSS repositories, emoji reactions on code-reviewers’ CR comments also correlate with usefulness. So, the code-author should meaningfully react with emojis, and the code-reviewer should conscientiously perceive emoji reactions to their feedback.
4.5 Practical Implications Our analysis offer the following implication for code-reviewers and code-authors of Sci-OSS. • Our findings confirm prior findings [3] on general software. Therefore, code-reviewers in Sci-OSS should compose positive, respectful, and objective CR comments as well. They also need to specify source-code lines while providing feedback. • Since developers pay more attention to emojis than text and they carry instructions and emotions, emojis should be
Characterizing the Usefulness of Code Review Comments in Scientific Software for Software Quality and Scientific Rigor PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
Table 4: Statistical Properties of Features in DB28 CR comments Here, effect-sizes denoted by ⇤ are statistically significant (p-value<0.05) DB28 -Projects! tone distress empathy is_toxic gratitude cr_senti kw_refact_xerox num_propernouns num_interjections polarity kw_msoft_u num_exclamation is_confirmatory question_ratio kw_satd_potdar implicature kw_msoft_nu code_word_ratio politeness num_Qmark rd_text kw_refact_problem avg_chars formality subjectivity has_out_snippet kw_secdev avg_punct num_sent stop_word_ratio num_tentative informativeness programming_words num_adverb num_adj num_nouns num_determinants num_chars num_words avg_stopwords kw_refact_solution num_verb avg_words
P1 .24 .07 .07 .14 .1 1.29⇤ .36 0 0 1.27⇤ .23 .2 .33 .24 .35 .19 .2 .55 1.07 .87⇤ .8⇤ .34 .83⇤ .88 1.5⇤ .06 .05 .37 .16 .51 .18 .7 .22 .4 .08 .39 .19 .21 .05 .07 .33 .27 .11
P10 .04⇤ .04⇤ .04⇤ .02 .02 .26⇤ .19⇤ .04 .03 .27⇤ .26⇤ .05⇤ .18⇤ .26⇤ .04 0 .18⇤ .16 .15⇤ .26⇤ .2⇤ .04 .19⇤ .09⇤ .34⇤ .04 .16⇤ .07 .28⇤ .32⇤ .18⇤ .04 .03⇤ .28⇤ .21⇤ .15⇤ .29⇤ .21⇤ .27⇤ .28⇤ .26⇤ .2⇤ .16⇤
P2 .04 .03 .03 .04 .06⇤ .22⇤ .15⇤ .01 .03 .3⇤ .21⇤ .05⇤ .21⇤ .15⇤ .05⇤ .02 .16⇤ .13⇤ .23⇤ .11⇤ .21⇤ .13⇤ .19⇤ .13⇤ .31⇤ 0 .03 .08⇤ .12⇤ .24⇤ .06⇤ .14⇤ .03 .12⇤ .06⇤ .02 .12⇤ .01⇤ .06⇤ .15⇤ .05⇤ .01 .03⇤
effect-sizep-value P3 P4 P6 P7 .02 .03 .13⇤ .33 .05 .01 .07 .13 .05 .01 .07 .13 .03 .03 .02 0 .1⇤ .02 .03 .18 .3⇤ .25⇤ .36⇤ .03 .19⇤ .12⇤ .15⇤ .19 .11⇤ .02 .02 0 .11⇤ .01 .01 0 .27⇤ .22⇤ .33⇤ .4 .18⇤ .15⇤ .18⇤ .34 .13⇤ .02 .03 .17 .19⇤ .12⇤ .17⇤ .17 .14⇤ .2⇤ .21⇤ .1 0 .04 .04⇤ .19 .06 .11⇤ .04 .17 .27⇤ .1⇤ .29⇤ .09 .16⇤ .19⇤ .27⇤ .05 .18⇤ .16⇤ .13⇤ .14 .15⇤ .17⇤ .22⇤ .15 .29⇤ .2⇤ .31⇤ .28 .01 .01 .07 .45⇤ .27⇤ .19⇤ .27⇤ .3 .16⇤ .09⇤ .15⇤ .19 .25⇤ .34⇤ .35⇤ .47⇤ .1⇤ .02 .22⇤ .09 .03 .03⇤ .05⇤ .1 .24⇤ .11⇤ .16⇤ .17 .16⇤ .23⇤ .18⇤ .05 .16⇤ .32⇤ .42⇤ .19 .07 .1⇤ .06⇤ .01 .1⇤ .04 .13⇤ .05 .08 .05 .1⇤ .09 .09⇤ .18⇤ .25⇤ .09 .15⇤ .08⇤ .04⇤ .02 .05 .02⇤ .06⇤ .06 .12⇤ .23⇤ .2⇤ .11 .02 .04⇤ .01⇤ .03 .09⇤ .14⇤ .11⇤ 0 .08 .19⇤ .28⇤ .14 .03 .16⇤ .16⇤ .02 .01 .13⇤ .06⇤ .03 .02 .01⇤ .02⇤ .02
P8 .08⇤ .01⇤ .01⇤ .01 .02⇤ .31⇤ .15⇤ .01⇤ 0 .25⇤ .21⇤ .02⇤ .24⇤ .21⇤ .01 .13⇤ .21⇤ .19⇤ .12⇤ .11⇤ .19⇤ 0 .2⇤ .07⇤ .38⇤ .02⇤ .16⇤ .1⇤ .23⇤ .4⇤ .22⇤ .01 0⇤ .32⇤ .16⇤ .09⇤ .31⇤ .16⇤ .25⇤ .35⇤ .22⇤ .21⇤ .16⇤
P9 .03 0⇤ 0⇤ .02 .03 .15⇤ .21⇤ .01 0 .36⇤ .24⇤ 0 .24⇤ .06⇤ .07⇤ .05 .17⇤ .16⇤ .26⇤ .08⇤ .14⇤ .05⇤ .13⇤ .17⇤ .3⇤ .04 .03 .08 .13⇤ .23⇤ .11⇤ .17⇤ .08⇤ .12⇤ .11⇤ .02⇤ .12⇤ .05⇤ .1⇤ .14⇤ .06⇤ .04 .02⇤
all .03⇤ .01⇤ .01⇤ .01 0 .29⇤ .15⇤ .01⇤ 0 .27⇤ .21⇤ .01⇤ .22⇤ .19⇤ .02⇤ .09⇤ .21⇤ .2⇤ .14⇤ .12⇤ .21⇤ .01 .2⇤ .1⇤ .36⇤ .04⇤ .12⇤ .1⇤ .21⇤ .37⇤ .17⇤ .06⇤ .02 .27⇤ .13⇤ .06⇤ .26⇤ .11⇤ .2⇤ .29⇤ .19⇤ .16⇤ .11⇤
P1 -.06 -.02 -.02 .03 .02 .30 .09 -.30 .06 .05 .08 -.06 .09 -.05 .05 .13 -.25 -.21 -.19 .08 .20 .21 -.34 -.01 .01 .09 -.04 -.12 .04 .17 .05 -.10 .02 .09 .05 .05 .01 -.02 .08 -.07 .03
P10 .02 .02 .02 -.01 -.01 .11 .08 -.02 -.01 -.11 .11 .02 .07 -.11 .02 .00 -.08 .07 -.06 -.11 -.08 .02 .08 .04 -.14 -.02 -.07 .03 -.11 -.13 -.08 .02 -.01 -.11 -.09 -.06 -.12 -.09 -.11 -.12 -.11 -.08 -.07
P2 -.02 -.01 -.01 -.02 -.03 .10 .06 .01 .01 -.13 .09 -.02 .09 -.06 .02 .01 -.07 .06 -.10 -.05 -.09 -.05 .08 .06 -.13 .00 -.01 .03 -.05 -.10 -.03 .06 .01 -.05 -.03 .01 -.05 -.01 -.03 -.06 -.02 .00 -.01
Pearson Correlation(X) P3 P4 P6 P7 -.01 .01 -.05 -.13 .02 .00 -.03 -.05 .02 .00 -.03 -.05 .01 .01 -.01 -.05 .01 -.01 -.07 .14 .11 .15 -.01 .09 .05 .06 .08 -.05 -.01 .01 -.05 .00 .00 -.13 -.10 -.14 -.16 .08 .07 .08 .14 -.06 .01 .01 -.07 .09 .05 .07 .07 -.06 -.09 -.09 .04 .00 -.02 .02 .08 -.03 -.05 -.01 -.07 -.13 -.05 -.12 .04 .08 .08 .11 .02 -.09 -.07 -.05 -.06 -.07 -.07 -.09 .06 -.13 -.09 -.13 -.11 .00 .00 -.03 -.18 .13 .08 .11 .12 .07 .04 .06 .08 -.12 -.15 -.14 -.19 .05 -.01 .09 -.04 -.02 -.01 -.02 -.04 .11 .05 .07 .07 -.07 -.10 -.07 -.02 -.07 -.14 -.17 -.08 -.03 -.05 -.02 -.01 .04 .02 .05 .02 .04 .02 .04 .04 -.04 -.08 -.10 -.04 -.07 -.03 -.02 -.01 .02 -.01 .02 .03 -.06 -.10 -.08 -.05 -.01 -.02 .01 .01 -.04 -.06 -.04 .00 -.04 -.08 -.12 -.06 -.01 -.07 -.07 -.01 -.01 -.06 -.03 -.01 -.01 .00 -.01 -.01
P8 .03 .00 .00 .00 .01 .13 .06 .00 .00 -.10 .08 .01 .10 -.08 .00 -.05 -.08 .08 -.05 -.04 -.07 .00 .08 .03 -.15 .01 -.06 .04 -.09 -.16 -.09 .01 .00 -.13 -.06 -.04 -.12 -.06 -.10 -.14 -.09 -.09 -.06
P9 -.01 .00 .00 .01 -.01 .06 .09 .00 .00 -.15 .10 .00 .10 -.02 .03 -.02 -.07 .07 -.11 -.03 -.06 .02 .06 .07 -.13 .02 -.01 .04 -.06 -.09 -.05 .07 .03 -.05 -.04 -.01 -.05 -.02 -.04 -.06 -.03 -.02 -.01
Table 5: Emojis in CR Comments Here, numbers inside the parentheses following the emojis denote the appearance of the emojis Dataset Emojis in the Dataset DA⌘ [27] (1,481) (18), (9), (3), (2), (1), (1), (1), (1), (1), (1), (1), (1) D22 [24] (3,794) (54), (6), (3), (2), (1), (1), (1). D>3 [31] (2,654) (38), (8), (3), (2), (3) DB28 [Ours] (164,708)
(290) (257) (210) (85) (79) (66) (62) (47) (43) (28) (23) (19) (15) (12) (9) (8) (8) (6) (5) (5) (4) (4) (3) (3) (3) (3) (2) (2) (2) (1) (1) (1) (1) (1) (1) (1) (1) (1)
(17) (3)
all .01 .00 .00 .00 .00 .12 .06 .00 .00 -.11 .09 .01 .09 -.08 .01 -.04 -.09 .08 -.06 -.05 -.08 -.01 .08 .04 -.15 .02 -.05 .04 -.09 -.15 -.07 .02 .01 -.11 -.05 -.02 -.11 -.05 -.08 -.12 -.08 -.07 -.04
PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
Sharif Ahmed and Nasir U. Eisty
Table 6: Metadata and Features’ Correlation in DB28 2 Metadata author_association country gender project side start_side line original_line original_position original_start_line position pr_num react_+1 react_-1 react_confused react_eyes react_heart react_hooray react_laugh react_rocket react_total_count start_line
DB28 .01 .06 .02 .05 .03 .05 .01 .01 .00 .01 .01 .04 .02 .00 -.01 -.01 -.01 .00 .00 .00 .01 .01
DB28 2 .05 .19 .09 .04 .03 .13 -.01 .01 .02 .06 .00 .00 .03 -.02 -.05 -.02 .05 .04 .04 -.17
Features tone distress empathy is_toxic gratitude cr_senti kw_refact_xerox num_propernouns num_interjections polarity kw_msoft_u num_exclamation is_confirmatory question_ratio kw_satd_potdar implicature kw_msoft_nu code_word_ratio politeness num_Qmark rd_text kw_refact_problem
DB28 .01 .00 .00 .00 .00 .12 .06 .00 .00 -.11 .09 .01 .09 -.08 .01 -.04 -.09 .08 -.06 -.05 -.08 -.01
DB28 2 .05 .00 .00 .00 .00 .08 .03 -.02 .03 -.17 .05 .01 .06 -.01 -.06 -.02 .02 .02 -.09 .00 .05 .03
programming_words num_adverb num_adj num_nouns num_determinants num_chars num_words emo_senti avg_stopwords kw_refact_solution num_verb avg_words subjectivity has_out_snippet kw_secdev avg_punct num_sent stop_word_ratio num_tentative informativeness avg_chars formality
DB28 .01 -.11 -.05 -.02 -.11 -.05 -.08 -.01 -.12 -.08 -.07 -.04 -.15 .02 -.05 .04 -.09 -.15 -.07 .02 .08 .04
DB28 2 -.02 -.13 -.03 -.03 -.11 -.05 -.08 .034 -.16 -.08 -.08 -.04 -.20 -.03 -.08 .04 -.09 -.19 -.06 .05 .03 .09
Table 7: Emoji Presence in CR comments within DB28 Here, numbers inside the parentheses following the emojis denote the appearance of the emojis
DB28 P2 P3 P4 P6 P8 P9 P10
Emojis in Useful CR comments (39) (30) (13) (13) (6) (3) (3) (2) (2) (1) (1) (1) (1) (1) (1) (1) (6) (4) (2) (2) (2) (1) (1) (1) (1) (1) (1) (1) (1) (1) (39) (33) (22) (17) (9) (5) (5) (3) (2) (2) (1) (1) (1) (1) (1) (1) (1) (113) (107) (73) (52) (34) (28) (26) (22) (14) (10) (7) (7) (6) (5) (4) (3) (2) (2) (2) (1) (1) (1) (1) (1) (1) (58) (6) (5) (5) (4) (3) (3) (2) (2) (1) (1) (1) (1) (1) (1) (1) (1) (22) (17) (8) (5) (4) (3) (2) (2) (2) (2) (1) (1) (1) (1) (1) (1)
Emojis in Not-useful CR comments (26) (5) (3) (2) (2) (2) (1) (3)
(2)
(13)
(13)
(1)
(1)
(2)
(1)
(1)
(1)
(1)
(1)
(6)
(6)
(2)
(1)
(1)
(1)
(1)
(1)
(37) (5) (1) (17)
(36) (26) (16) (13) (11) (10) (3) (3) (3) (2) (2) (1) (1) (1) (1) (3) (3) (3) (1) (1) (1) (1)
(9) (1)
(7) (1)
(9)
(6)
(1)
(1)
(5)
(4)
(2)
(1)
(1)
(1)
Table 8: Emoji-Semantics in Identifying Usefulness of CR comments from Scientific Software 10-fold Stratified Cross-validation Performance of models with emojis and without emojis. P R M A D? 1 DB28 2 (text) DB28 2 (emoji) DB28 2 (text+emoji)
.76 .74 .75
.99 .98 .99
.86 .84 .85
(1)
.22 -.02 .13
.76 .73 .75
.64⇤ .73⇤ .69⇤
Here, P: Precision, R: Recall, A: Accuracy, 1 : 1 -scores, M: Matthews correlation coefficient (MCC), D? : Cohen’s D effect sizeWilcoxon signed-rank p-value , and ⇤ : p-value < 0.05
Characterizing the Usefulness of Code Review Comments in Scientific Software for Software Quality and Scientific Rigor PASC 2026, June 29 – July 1, 2026, Bern, Switzerland
carefully used similar to words [2]. For emoji-containing CR comments in scientific software, we find more association with usefulness, so it reinforces significance of emoji usage. Thus, code-authors should meaningfully react with emojis. Also, code-reviewers should perceive emoji reactions on their feedback conscientiously.
5
THREATS TO VALIDITY
Our experiment setup leverages the existing theories, datasets, experiments, and approaches for identifying and analyzing useful CR comments. However, we discuss the potential threats to the validity of our work.
5.1
External Validity
The generalizability of our findings is a threat to external validity. Our experiment data are from a small set of Sci-OSSs from applied mathematics, programming systems, and machine learning domains. We expect our findings to apply to Sci-OSS broadly; however, there remains a threat to external validity for projects from other domains.
5.2
Internal Validity
First, model computed metrics in our work are trained from different sources; thus, they may carry implicit biases that we are not aware of since the data was collected by other researchers. Second, our approximated gender values are unknown for meaningless or encrypted names, and normalized location values are missing for GitHub profiles sharing improper information. Our findings regarding gender or location could have been different if these data points had been available directly from the GitHub profiles. Third, Developers may communicate through alternative channels (e.g., email, Discord, Slack), which may not be captured in CR comments. However, our work evaluates the usefulness of CR comments based exclusively on their textual content, rather than the complete communication process. Therefore, this does not constitute a significant threat to internal validity of our findings.
5.3
Conclusion Validity
To conclude our findings accurately, we carefully designed our experiments. We compared feature values of our prediction-based useful CR comments with existing datasets. Since the state-of-theart prediction model we used for annotation is not from handcrafted feature-based model, analyzing the features for comparison does not impose any threats. In our statistical analysis, we have considered suitable statistical tests for numeric and categorical metadata/ features so that our findings do not violate the underlying normality and independence assumptions.
6
CONCLUSIONS
Researchers have analyzed, characterized, and identified the usefulness of CR comments in non-scientific closed and open-source software projects. In this paper, we investigate the CR comments in scientific open-source software to describe the distinctive nature of scientific CR comments. We leveraged existing usefulnessannotated CR comment datasets, the best-performing usefulness prediction model, and feature analysis techniques. Our statistical
and XAI feature analyses confirm findings from prior research on general-purpose software. With our additional derived metadata, we analyzed their association with the usefulness of code review feedback. These findings will benefit developers, leaders, scientists, and researchers in both general and Sci-OSS communities. In the future, we plan to conduct an empirical study on the usefulness CR comments focusing scientific open source software code authors and code reviewers.
7
DATA AVAILABILITY
We have shared our code and dataset, including feature values computed in our experiments at https://github.com/sharif509/SciOSS-CRCs-artifact
REFERENCES
[1] Sharif Ahmed and Nasir U Eisty. 2023. Exploring the Advances in Identifying Useful Code Review Comments. In 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–7. [2] Sharif Ahmed and Nasir U Eisty. 2024. Understanding Emojis :) in Useful Code Review Comments. In Proceedings of the Third ACM/IEEE International Workshop on NL-Based Software Engineering (Lisbon, Portugal) (NLBSE ’24). Association for Computing Machinery, New York, NY, USA, 81–84. https://doi.org/10.1145/ 3643787.3648035 [3] Sharif Ahmed and Nasir U Eisty. 2025. Hold On! Is My Feedback Useful? Evaluating the Usefulness of Code Review Comments. Empirical Software Engineering 30, 3 (2025), 70. https://doi.org/10.1007/s10664-025-10617-1 [4] Sharif Ahmed, Addi Malviya Thakur, Gregory R Watson, and Nasir U Eisty. 2025. Uncovering Scientific Software Sustainability through Community Engagement and Software Quality Metrics. arXiv preprint arXiv:2511.07851 (2025). [5] Toufique Ahmed, Amiangshu Bosu, Anindya Iqbal, and Shahram Rahimi. 2017. SentiCR: a customized sentiment analysis tool for code review interactions. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 106–111. [6] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics 5 (2017), 135–146. https://doi.org/10.1162/tacl_a_ 00051 [7] Amiangshu Bosu, Jeffrey C Carver, Christian Bird, Jonathan Orbeck, and Christopher Chockley. 2016. Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft. IEEE Transactions on Software Engineering 43, 1 (2016), 56–75. [8] Amiangshu Bosu, Michaela Greiler, and Christian Bird. 2015. Characteristics of useful code reviews: An empirical study at microsoft. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 146–156. [9] Jason Cohen. 2010. Modern code review. Making Software: What Really Works, and Why We Believe It (2010), 329–336. [10] Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. Academic press. [11] Harald Cramér. 1999. Mathematical methods of statistics. Vol. 9. Princeton university press. [12] Arcos David. 2024. gender-guesser. https://pypi.org/project/gender-guesser. Accessed: 2024-11-08. [13] Nicole Davila and Ingrid Nunes. 2021. A systematic literature review and taxonomy of modern code review. Journal of Systems and Software 177 (2021), 110951. [14] Vasiliki Efstathiou and Diomidis Spinellis. 2018. Code review comments: language matters. In Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results. 69–72. [15] Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel. 2016. emoji2vec: Learning Emoji Representations from their Description. In Proceedings of the Fourth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics, Austin, TX, USA, 48–54. https://doi.org/10.18653/v1/W16-6208 [16] Nasir U Eisty and Jeffrey C Carver. 2022. Developers perception of peer code review in research software development. Empirical Software Engineering 27 (2022), 1–26. [17] Masum Hasan, Anindya Iqbal, Mohammad Rafid Ul Islam, AJM Rahman, and Amiangshu Bosu. 2021. Using a balanced scorecard to identify opportunities to improve code review effectiveness: an industrial experience report. Empirical Software Engineering 26, 6 (2021), 1–34. [18] Oleksii Kononenko, Olga Baysal, and Michael W Godfrey. 2016. Code review quality: How developers see it. In Proceedings of the 38th international conference
PASC 2026, June 29 – July 1, 2026, Bern, Switzerland on software engineering. 1028–1038. [19] Oleksii Kononenko, Olga Baysal, Latifa Guerrouj, Yaxin Cao, and Michael W Godfrey. 2015. Investigating code review quality: Do people and participation matter?. In 2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 111–120. [20] Esmukov Kostya. 2023. geopy. https://pypi.org/project/geopy/. Accessed: 2024-11-08. [21] Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015. Sentiment of emojis. PloS one 10, 12 (2015), e0144296. [22] Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4765–4774. [23] Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60. [24] Benjamin S Meyers, Nuthan Munaiah, Emily Prud’hommeaux, Andrew Meneely, Josephine Wolff, Cecilia Ovesdotter Alm, and Pradeep Murukannaiah. 2018. A dataset for identifying actionable feedback in collaborative software development. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 126–131. [25] Thai Pangsakulyanont, Patanamon Thongtanunam, Daniel Port, and Hajimu Iida. 2014. Assessing MCR discussion usefulness using semantic similarity. In 6th International Workshop on Empirical Software Engineering in Practice. IEEE, 49–54.
Sharif Ahmed and Nasir U. Eisty [26] Karl Pearson. 1900. X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 50, 302 (1900), 157–175. [27] Mohammad Masudur Rahman, Chanchal K Roy, and Raula G Kula. 2017. Predicting usefulness of code review comments using textual features and developer experience. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 215–226. [28] Eric Raymond. 1999. The cathedral and the bazaar. Knowledge, Technology & Policy 12, 3 (1999), 23–49. [29] Lucía Santamaría and Helena Mihaljević. 2018. Comparison and benchmark of name-to-gender inference services. PeerJ Computer Science 4 (2018), e156. [30] Daniel Schneider, Scott Spurlock, and Megan Squire. 2016. Differentiating communication styles of leaders on the linux kernel mailing list. In Proceedings of the 12th International Symposium on Open Collaboration. 1–10. [31] Asif Kamal Turzo and Amiangshu Bosu. 2023. What Makes a Code Review Useful to OpenDev Developers? An Empirical Investigation. Empirical Software Engineering (2023). Just Accepted. [32] Frank Wilcoxon, SK Katti, et al. 1970. Critical values and probability levels for the Wilcoxon rank sum test and the Wilcoxon signed rank test. Selected tables in mathematical statistics 1 (1970), 171–259.
Received TBD; revised TBD; accepted TBD