Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics Sheikh Md. Mushfiqur Rahman
Gregory R. Watson
Nasir U. Eisty
University of Tennessee Knoxville, TN, USA [email protected]
Oak Ridge National Laboratory Oak Ridge, TN, USA [email protected]
University of Tennessee Knoxville, TN, USA [email protected]
arXiv:2605.03243v1 [cs.SE] 5 May 2026
Abstract Context: Scientific open-source software (SciOSS) plays a foundational role in research and engineering, yet its long-term sustainability has often been overlooked and remains a significant concern. Objective: This study investigates the long-term sustainability of SciOSS through code and test quality metrics. Method: We analyze CASS Software Portfolio projects, classifying them by sustainability and comparing their code structure, test coverage, and links between code quality and testing across the dataset. Results: Sustainable projects show higher, more consistent test coverage and clearer code–test correlations, while unsustainable ones show weaker patterns. Overall, test coverage is low in scientific software, and high complexity and coupling reduce testability. Conclusion: In this study, we present a practical, data-driven approach for assessing sustainability in scientific software, offering a foundation for evaluating long-term software health and supporting future efforts in quality assurance and sustainability monitoring.
CCS Concepts • Software and its engineering;
Keywords Scientific Software, Software Metrics, Sustainability, Code Quality, Test Coverage Metrics ACM Reference Format: Sheikh Md. Mushfiqur Rahman, Gregory R. Watson, and Nasir U. Eisty. 2026. Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics. In Proceedings of The Platform for Advanced Scientific Computing (PASC 2026). ACM, New York, NY, USA, 12 pages. https://doi. org/10.1145/nnnnnnn.nnnnnnn
1
Introduction
Scientific software, broadly defined, refers to software developed for scientific purposes, primarily to improve our understanding of real-world phenomena or to enable predictive modeling of such processes [1, 30]. In a research context, such software may be developed with limited knowledge of software requirements (specifications Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. PASC 2026, Bern, Switzerland © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
and features of software), industry-standard software design patterns [23], good coding practices (e.g., using descriptive variable names), version control, software documentation, automated testing, and project management practices (e.g., Agile) [9, 42]. This leads to the creation of source code that is not well-structured, is not easily (re)usable, is difficult to modify and maintain, has scarce internal documentation (code comments) and external documentation (e.g., manuals, guides, and tutorials), and has poorly documented workflows [38]. Scientific software that suffers from these shortcomings is likely to be challenging to sustain and have severe drawbacks for scientific research. For example, it can impede research progress, decrease research efficiency, and hinder scientific progress, as implementing new ideas or correcting mistakes in code that is not well-structured is more difficult and time-consuming [38]. In addition, it increases the likelihood of erroneous results, thereby reducing reliability and hindering reproducibility [43]. These harmful properties can be averted, to some extent, with sustainable scientific software. There are various interpretations of the meaning of sustainable scientific software. Anzt et al. [3] define sustainable scientific software as software that is maintainable, extensible, and flexible (adapts to user requirements); has a defined software architecture; is testable; has comprehensive in-code and external documentation, and is accessible (the software is licensed as open source with a digital object identifier (DOI)). Katz [31] views scientific software sustainability as the process of developing and maintaining scientific software that continues to meet its purpose over time. Existing research has explored the sustainability of SciOSS by examining factors such as early development behavior and project activity [27, 57], authorship and contributor participation [5], social and technical networks [58], and project-level metrics [38, 47]. Building on this foundation, our study adopts a data-driven approach to investigate sustainability in SciOSS through the lens of structural code metrics and test suite coverage. We curated and analyzed a set of C/C++ SciOSS projects stewarded by the Consortium for the Advancement of Scientific Software (CASS) [2], which is an initiative that emphasizes community engagement, reproducibility, and engineering rigor. We labeled each project as either sustainable or unsustainable using two complementary and well-established classification methods: one based on long-term commit activity, and the other based on the presence and continuity of core contributors, known as Truck Factor (TF) developers. By comparing these two groups, we examine how structural complexity, modularity, cohesion, coupling, and test coverage vary between sustainable and unsustainable scientific software. We also
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
explore whether these metrics exhibit meaningful correlations, particularly in sustainable projects, and assess whether stricter definitions of sustainability lead to more reliable distinctions in software quality. Our findings offer insights into the relationship between internal code quality and sustainability, helping to inform better practices for developing and maintaining scientific software. Our study is guided by the following research questions: • RQ1: How do structural code metrics and test coverage differ between sustainable and unsustainable SciOSS? Scientific software is often developed without rigorous engineering practices, leading to poor maintainability and increased risk of errors . While prior studies have primarily focused on social or activity-based indicators (e.g., contributors, commits), there is limited understanding of how internal code properties, such as complexity, modularity, and test coverage, relate to sustainability. This research question addresses that gap by investigating whether measurable structural and testing characteristics can distinguish sustainable projects from those at risk. Establishing such differences is important because it enables the identification of concrete, actionable indicators that developers and stakeholders can monitor to improve long-term software health. • RQ2: Does using a stricter threshold for sustained activity improve the reliability of sustainability classifications based on test quality metrics? As shown in prior work and reinforced in this study, varying criteria (e.g., activity-based vs. contributor-based) can produce contradictory interpretations of quality indicators like test coverage. This research question is important because it examines whether refining the definition of sustainability by setting stricter activity thresholds reduces ambiguity and improves the reliability of empirical findings. Without such validation, conclusions drawn about sustainability may be misleading or dependent on arbitrary labeling choices. By evaluating the impact of stricter thresholds, this RQ contributes to methodological rigor and helps establish more reliable benchmarks for future sustainability research. • RQ3: Is there any strong relationship between code metrics and test coverage in scientific software overall? It remains unclear whether intrinsic code properties, such as complexity, coupling, and cohesion, systematically influence testability. This research question is important because it seeks to uncover fundamental relationships between code structure and testing effectiveness across projects. Identifying such relationships can provide deeper insights into why certain software systems are harder to test and maintain. Moreover, understanding these links enables the use of structural metrics as early indicators of testing challenges, offering a scalable way to assess and improve software quality in large scientific codebases. By answering these questions, we aim to shed light on the structural and testing-related attributes that distinguish sustainable scientific software from those at risk of becoming unsustainable. Our findings contribute to a better understanding of the engineering
Rahman et al.
characteristics that underpin sustainable scientific software and offer actionable insights for developers, maintainers, and stakeholders seeking to improve the longevity and reliability of SciOSS.
2
Related Work
Stănciulescu et al. [47] investigated how code, process, and quality metrics correlate with the sustainability of Apache Software Foundation Incubator (ASFI) projects. By comparing over 200 graduated (sustainable) and retired (unsustainable) projects, the authors find that factors like contributor activity, commit patterns, and code complexity significantly influence project outcomes. Yin et al. [58] presented a method to predict the sustainability of Apache Incubator projects by analyzing their monthly sociotechnical metrics derived from commits and email interactions. Projects are labeled as sustainable if they graduate and unsustainable if they retire. A time-series LSTM model trained on these features achieves 93% accuracy within 8 months of incubation. The authors also apply LIME to interpret the model and validate its predictions through real-world case studies. Xiao et al. [57] proposed a machine learning model to predict the long-term sustainability of open-source GitHub projects early in their lifecycle, using behavioral signals such as founder history, development activity, community engagement, documentation, and popularity from the first few months. Projects are labeled as sustainable if they remain active for two years with a median monthly commit rate above a threshold. They found that stricter thresholds improve the model’s classification accuracy by reducing label noise. Han et al. [27] proposed a forecasting approach that uses LSTM (Long Short-Term Memory) neural networks to predict whether deep learning packages on GitHub will remain active or become dormant. The model is trained on time-series data capturing monthly developer activity and collaboration patterns. In addition to making accurate predictions, they explored how activity trends can identify early signs of decline in otherwise sustainable-looking projects. Valiev et al. [54] investigated what makes OSS projects in the Python Package Index (PyPI) ecosystem sustainable by combining large-scale quantitative analysis with developer interviews. The authors define project sustainability as continued maintenance and label projects as dormant if they average less than one commit per month in the year before their last commit. Using logistic and linear regression models, they evaluate how ecosystem-level factors, such as dependency centrality (PageRank), backporting, and organizational support, affect project survival, revealing that ecosystem position plays a significant role in sustainability. Avelino et al. [5] studied how OSS projects survive or become abandoned after losing their core developers (Truck Factor developers). It introduces the TF Developer Detachment (TFDD) concept and compares surviving vs. non-surviving projects using metrics like number of developers, commits, files, and project age. Statistical tests (Mann–Whitney U, Cliff’s Delta, Benjamini–Hochberg) reveal that surviving projects tend to be more active, older, and attract new key contributors. Complementary to the above prediction-oriented studies, prior work on sustainability has also emphasized multi-signal repository stability models (often referred to as CSI frameworks), which integrate commit activity, contributor retention, responsiveness,
Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics
• The project must be written in C and/or C++, and hosted in a publicly accessible repository to allow cloning, local builds, and transparent reproducibility. • It must use a supported build system such as GNU Make, CMake, or GNU Autotools (e.g., configure, autogen.sh) to ensure reproducible builds, correct dependency resolution, and seamless integration of coverage instrumentation flags across multiple compilation units. • It must build successfully using gcc or clang with coverage instrumentation flags (–coverage -O0 -g). These flags enable line- and branch-level tracking by injecting profiling logic into the compiled object files. Each compilation unit must be instrumented individually to produce .gcno and .gcda files to be processed into coverage summaries. • It must be clang compatible to extract abstract syntax trees (ASTs). • It must contain at least one test executable or driver that runs successfully (i.e., produces “green” test results), which is essential to generate runtime coverage data. Tests must exercise enough code paths to produce meaningful and analyzable .gcda files. • It must be compatible with gcovr [45], the coverage analysis tool we used to generate standardized line, branch, and function coverage reports. gcovr is a lightweight command-line tool that wraps the GNU gcov utility. Our pipeline ran gcovr from the build directory, using project-root anchoring (-r) and supporting both tabular and HTML outputs.
dependency health, and governance signals into composite sustainability indicators [12, 20, 25, 39]. Unlike prior studies that predict sustainability in general-purpose OSS or specific ecosystems (e.g., ASF Incubator, PyPI, GitHub) using process, social, and activity metrics, our study emphasizes the relationship between code-level structural & test metrics with sustainability in SOSS. Using projects from the CORSA Catalog, we combine detailed static code metrics (size, complexity, cohesion, coupling, documentation, function calls) with dynamic test coverage measures (line, branch, function) to investigate how internal code quality relates to long-term sustainability.
3 Methodology 3.1 Project Selection Criteria We curated a set of open-source scientific software projects written in C and C++, drawn from the CASS Software Portfolio [15]. All selected projects are publicly hosted (e.g., on GitHub or institutional repositories), available for cloning and local builds. The CASS Portfolio includes tools widely used across domains such as artificial intelligence, mathematics, physics, performance and correctness analysis, programming systems, and scientific data management and visualization, with a strong emphasis on community relevance, reproducibility, and long-term sustainability. Several projects in the CASS Portfolio proved infeasible to build in a constrained research environment. In practice, we encountered (i) high memory and link-time requirements (often >16–32 GB RAM), (ii) platform-specific or privileged dependencies (e.g., CUDA/ROCm drivers, MPI stacks, legacy toolchains), and (iii) brittle or undocumented build scripts that failed under coverage instrumentation. We standardized builds in Docker with GCC/Clang and gcovr, but excluded repositories that could not configure or compile with –coverage -O0 -g, required specialized hardware, or lacked runnable tests. This feasibility filter reduces the sample size but improves the reproducibility and internal validity of our measurements. To ensure compatibility with our instrumentation and analysis workflow, we restricted our selection to projects that satisfied the following criteria, enabling both static analysis (e.g., AST parsing) and dynamic analysis (e.g., test coverage): Dataset Collection & Instrumentation
Labeling Methodology
Analysis & Key Findings
Scientific Software Dataset
Sustainability Labeling
Metric Comparison & Correlation Analysis
● 14 C/C++ projects from catalog ● Build with GCC/Clang (--Coverage, AST) ● Tool used: ○ gcovr → line/branch/ function coverage ○ Analizo,Lizard, Cloc, Clang AST → Code metrics
● Label: Sustainable/ Unsustainable ● 2 approaches: ○ T-year Sustainability (median number of commits within activity period) ○ TF developer remains/ replanced
● Violin plots to analyse test coverage distributions ● Correlation (Spearman ⲣ): ○ Sustainable projects → More interpretable (NMC↓Coverage↑) ○ Unsustainable projects → Weak/inconsistent correlation Sustainable projects exhibit higher coverage, stronger structure-test relationships, Unsustainable projects shows inconsistent pattern and poor connections.
Figure 1: Overview of The Approach & Key Findings.
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
3.2
Metrics
To characterize the structural complexity, modularity, and documentation of each source file, we collected a diverse set of static and dynamic code metrics. These metrics span multiple dimensions, including size, comments, function calls, complexity, cohesion, coupling, and test coverage, enabling a detailed structural analysis of both C and C++ codebases. These metrics reflect not only code volume and granularity, but also the extent of in-line documentation and function interface design. The full set of collected metrics is summarized in Table 1. 3.2.1 Size and Comment Metrics. We collected a set of size and comment-related metrics, applicable to both C and C++ projects. These metrics provide foundational insights into the structural volume and documentation practices of each source file. Metrics such as Lines of Code (LOC), Number of Methods (NOM), and Method Size (MMLOC, AMLOC) reflect the complexity and granularity of code at the function level. The Number of Attributes (NOA), captures the density of field declarations and global variables, adding another layer to structural analysis. Complementing these are the documentation-related metrics, such as Lines of Comments (LOCCOM) and Comment Density (CD), which quantify the extent of in-line developer commentary. Together, these metrics characterize both the scale of implementation and the degree of self-documentation, which are important factors influencing maintainability and understandability across both C and C++ codebases. Metrics related to static fields or bytecode instructions (e.g., NBI, NSTAF) were excluded as they are not directly applicable or consistently defined across C and C++.
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
Rahman et al.
3.2.2 Function Call Metrics Extraction. To analyze procedural .c files and extract function call-related metrics, we developed a static analysis script built around Clang’s -ast-dump functionality. The script parses both the Clang-generated AST and the source code itself to identify and classify function definitions and invocations within each C source file. Specifically, we extracted Number of Method Calls (NMC), Number of Method Calls External (NMCE), and Number of Method Calls Internal (NMCI). We identified function definitions through source-level pattern matching and detected function calls by analyzing the AST output. We extracted Callee names by scanning lines surrounding each CallExpr node. We considered a function call external and thus counted it towards the NMCE metric if its target function does not match any function defined within the file under analysis. This strategy provides a lightweight approximation of inter-module coupling and behavioral complexity by treating each .c file as a logical unit similar to a class or module.
3.2.6 Test-Quality Metrics. To assess the thoroughness of each project’s test suite, we computed line coverage, branch coverage, and function coverage using the gcovr tool [45]. These metrics quantify how much of the code is exercised by the project’s test suite, serving as a proxy for test effectiveness. We compiled each project using gcc or clang with coverage instrumentation flags (−−coverage -O0 -g), executed the test suite, and then ran gcovr to extract coverage reports, which include line (line_percent), branch (branch_percent), and function (function_percent) coverage. Higher coverage values generally indicate that a larger portion of the program has been exercised by tests, increasing the likelihood of revealing faults (provided appropriate assertions are present). While coverage alone cannot guarantee correctness, it remains a practical and scalable proxy for test adequacy in large-scale studies.
3.2.3 Complexity. Complexity metrics such as Weighted Methods per Class (WMC) and Average Method Complexity (AMC) are based on the cyclomatic complexity measure, which quantifies the number of linearly independent paths through a method’s control flow [35]. We also collected Response for Class (RFC) [11] which calculates the sum between the number of methods in the module and the number of functions called by each module function, and Structural Complexity (SC), which quantifies how difficult a software module may be to maintain by combining two key design aspects: coupling and cohesion. Specifically, it is calculated as the product of CBO (Coupling Between Objects) and LCOM4 (Lack of Cohesion of Methods) [19]. A higher SC value indicates that a module is both tightly connected to other modules and internally less cohesive, which together suggest increased structural complexity and greater maintenance effort.
To collect code metrics, we used a combination of open-source tools and custom analyzers:
3.2.4 Cohesion. Cohesion refers to the degree to which the elements within a software module, such as statements in a method or functions within a file, are functionally related. In object-oriented programming, cohesion typically describes how well the methods within a class work together to fulfill a single responsibility. High cohesion is desirable, as it improves understandability, maintainability, and reusability. Conversely, low cohesion indicates that a module performs a wide range of unrelated tasks, making it harder to comprehend and maintain. To quantify cohesion, several metrics have been proposed. One widely used metric is LCOM4 [29] (Lack of Cohesion of Methods, version 4). It models a module (e.g., a class or a C file) as an undirected graph where nodes represent subroutines (methods or functions), and edges connect nodes that either share access to at least one variable or directly call each other. The LCOM4 value corresponds to the number of connected components in this graph; higher values indicate lower cohesion, indicating that the module contains independent clusters of logic. 3.2.5 Coupling. Coupling metrics characterize the degree of interdependence between different modules. Coupling Between Objects (CBO) calculates the number of calls to other modules. ACC counts how many other modules reference, import, or call functions from the given module.
3.3
Tools and Techniques for Data Collection
• To collect consistent source code metrics across both C and C++ projects, we used Analizo [50]. Originally designed for object-oriented metrics, Analizo generalizes its internal representation by treating each translation unit (e.g., a .c file) as a module, similar to a class in C++. This abstraction allows it to compute a common set of structural metrics, such as method counts, coupling, cohesion, and complexity (metrics such as CBO, NOM, RFC, and LCOM4) for both procedural and object-oriented code. Thus, it is well-suited for analyzing mixed-language scientific software projects. • Lizard [34], a lightweight static analysis tool, was used to compute cyclomatic complexity and function-level metrics. It supports C/C++ without requiring complete headers or preprocessing, making it practical for large scientific codebases. We used it to extract WMC (total cyclomatic complexity per file) and AMC (average complexity per function), based on its per-function analysis of non-comment lines, parameter count, and control flow complexity. • Cloc [18], to count the total number of comment lines in each source file and Comment Density (CD). • Custom Clang-based AST analyzers, to extract additional metrics not directly supported by the above tools, such as internal versus external function calls. • Finally, to compute test-quality metrics, we executed the test suite of each project and used gcovr [45] to collect test coverage report.
4 Results 4.1 RQ1: How do structural code metrics and test coverage differ between sustainable and unsustainable SciOSS? Various indicators help evaluate an OSS project’s sustainability: 1) Artifact-related indicators, such as employing a modular and extensible architecture and maintaining high-quality documentation [47]; 2) Economic indicators, including sufficient funding, low total cost of ownership, and high added value [41, 54, 59]; 3) Supply chain
Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
Table 1: Summary of Source Code and Test Coverage Metrics Metric
Description
Ref.
Size & Comment Metrics Lines of Code (LOC) Number of Methods (NOM) Number of Branches (NOB) Max Method LOC (MMLOC) Average Method LOC (AMLOC) Average Number of Parameters (ANPM)
Non-empty, non-comment lines of code Total number of function definitions per file Total number of branches in a file Lines in the longest function Mean size of all functions in lines Mean number of parameters per function
[36] [36] [36]
Number of Attributes (NOA) Lines of Comments (LOCCOM) Comment Density (CD)
Count of global variables and struct/class fields Total comment lines (computed using cloc) Ratio of comment lines to total lines (code + comments)
[4]
Function Call Metrics Number of Method Calls (NMC) Number of Internal Function Calls (NMCE)
All function invocations (from AST CallExpr nodes) Calls to functions defined in the same file
[52] [52]
External Function Calls (NMCE)
Calls to functions defined in other files
[52]
Complexity Metrics Weighted Methods per Class (WMC) Average Method Complexity (AMC) Response for Class (RFC) Structural Complexity (SC)
Sum of cyclomatic complexity of all functions Mean cyclomatic complexity of functions Number of functions + number of distinct calls they make Product of CBO and LCOM4: SC = CBO × LCOM4
[11] [49] [11] [19]
Cohesion Metric Lack of Cohesion of Methods (LCOM4)
Number of disconnected components in a function-variable/call graph
[51]
Coupling Metrics Coupling Between Objects (CBO) Afferent Coupling Count (ACC)
Number of modules/functions referenced by the file Number of modules that depend on this file
[11] [33]
Test Coverage Metrics Line Coverage (LC) Branch Coverage (BC) Function Coverage (FC)
Percentage of source lines executed during testing Percentage of decision branches exercised by tests Percentage of defined functions called at least once
-
indicators, such as keeping dependencies up-to-date and having a large number of downstream dependents [54, 56]; 4) Development activity indicators, including frequent commits, a well-staffed team, and regular contributions from newcomers [48, 58]. Among these indicators, sustained development activity stands out as a particularly reliable marker of sustainability, since software that lacks ongoing maintenance tends to degrade in usefulness over time—a notion reinforced by Lehman’s Law [32]. While prior work has used development activity as a proxy for both the success [37] and sustainability [54] of OSS projects, Chengalur-Smith et al. [10] emphasize that true sustainability is characterized by consistent, long-term engagement, in contrast to success, which can be assessed at a single point in time. Thus, in this paper, we use sustained activity as the main proxy for studying OSS sustainability, following previous works [5, 13, 24, 54]. In this study, we adopted two complementary approaches to measure sustained activity ensure a more comprehensive labeling
process, each of which has been previously used as a proxy for software sustainability [54, 57]: • A project is classified as exhibiting sustained activity over a duration of 𝑡 years if it satisfies two conditions: (1) it has recorded commits spanning more than 𝑡 years, and (2) its monthly median commit count meets or exceeds a threshold 𝑘. The parameter 𝑘 can be tuned, with larger values reflecting a more rigorous criterion for sustained project activity. Based on the optimal thresholds used in previous works [57], we set the the values 𝑡=2 and 𝑘=6. • A project is labeled sustained if at least one Truck Factor (TF) (the minimal number of developers that the project depends on for its maintenance and evolution [55]) developer remains active or new TF developers emerge after detachment [5]. After categorizing the projects using the two classification methods, we gathered code metrics from the source files of each project in both groups. We examined whether there were any differences in test coverage between the codebases of the two datasets.
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
Rahman et al.
Sustainable projects show higher test coverage under the 𝑡-year criterion, but not under the TF-based classification, suggesting that coverage alone does not guarantee longterm sustainability.
Figure 2: Coverage comparison based on 𝑡-year sustainability.
First, we partitioned the dataset using the t-year sustainability criterion, which labeled 8 projects as sustainable and 6 as unsustainable. In total, the sustainable projects contributed 5,579 files, while the unsustainable projects contributed 1,511 files. Figure 2 summarizes the coverage distributions under this classification, where projects are labeled as sustainable if they maintain active development for a specified number of years. Under this classification, sustainable projects show clearly higher median values for line (0.30), function (0.57), and branch (0.19) coverage than unsustainable projects, which have a median of 0.00. This pattern suggests that projects with long-term activity tend to invest more consistently in testing infrastructure, possibly due to the demands of ongoing feature development, bug fixing, and maintenance. The stark contrast indicates that test coverage can serve as a distinguishing indicator of project longevity and sustained engagement.
Next, we computed the Spearman’s correlation coefficient (𝜌) [40] for 45 (15 × 3) pairwise combinations of structural and test-effort metrics. Spearman’s coefficient is a widely used non-parametric measure of correlation in related studies [8, 53]. We did not use a parametric method such as Pearson correlation, as it assumes normally distributed data and linear relationships—assumptions that may not hold for our metric distributions. Instead, we opted for a non-parametric method that does not require such assumptions [28]. To verify non-normality, we applied the D’Agostino 𝐾 2 test [17], which assesses both kurtosis and skewness to detect deviations from normality. For all metric distributions, the 𝐾 2 test yielded 𝑝values ≤ 𝛼 (with 𝛼 = 0.05), leading us to reject the null hypothesis of normality. Spearman’s coefficient (𝜌) quantifies the degree to which two variables are associated with a monotonic function, i.e., an increasing or decreasing relationship [16]. The coefficient 𝜌 ranges from −1 to +1. A positive 𝜌 indicates that both variables increase together, while a negative 𝜌 suggests that one increases as the other decreases. A 𝜌 value close to zero implies little to no correlation between the two variables. For this research question, we are not interested in the direction of the correlation (positive or negative), and thus we use absolute values |𝜌 |. The colors range from blue (maximum correlation, |𝜌 | = 1.0) to red (minimum correlation, |𝜌 | = 0.0). We interpret |𝜌 | as weak (≤ 0.3), moderate (0.3–0.5), or strong (≥ 0.5), following the widely accepted classification of Cohen [14].
Figure 3: Coverage comparison based on TF classification. Then, we computed the Truck Factor (TF) following Avelino et al. [6]. Using this TF-based classification, 9 projects were labeled as sustainable and 5 as unsustainable. In total, the sustainable projects contributed 3,305 source files, whereas the unsustainable projects contributed 3,785 files. Figure 3 summarizes the coverage distributions under this classification, where sustainability is inferred from the presence and continuity of multiple key contributors rather than prolonged activity alone. Interestingly, in this grouping, unsustainable projects show higher median coverage values—0.40 for line, 0.70 for function, and 0.20 for branch—compared to sustainable ones, which have noticeably lower values. This inversion may reflect a scenario where unsustainable projects were initially built and tested by a small, highly concentrated group of contributors but lacked broader community support for continued evolution. It underscores that while test coverage is important, it is not sufficient on its own to ensure sustainability if the developer base is fragile or narrow in scope.
Figure 4: Comparison of heatmaps for sustainable and unsustainable projects using 𝑡-year sustainability classification. The heatmap in Figure 4 illustrates the Spearman correlation (𝜌) between static code metrics and test coverage metrics (line, branch, and function) across sustainable and unsustainable software projects. In sustainable projects, stronger and more consistent correlations are observed. Notably, NMCE and NMC show moderate to strong negative correlations with all types of coverage, while LCOM4 and WMC also correlate negatively, suggesting that high complexity and low cohesion are associated with reduced test
Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics
coverage. In contrast, CD and AMLOC show weak to moderate positive correlations, implying better documentation and modularity may or may not improve testability. On the other hand, unsustainable projects show generally weaker and more erratic correlations, with most values near zero. Although CD retains a positive trend, the absence of strong relationships suggests a lack of consistent structural-testability patterns in unsustainable projects’ codebase.
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
compared to other correction methods. Additionally, we compute effect sizes using Cliff’s delta [26], interpreted according to the thresholds proposed by Romano et al. [44].
Figure 6: Cliff’s Delta on 𝑡-year sustainability classification. Figure 5: Comparison of heatmaps for sustainable and unsustainable projects using TF classification. Figure 5 presents the same comparison, but for classification based on TF. In sustainable projects, metrics like NMC and ANPM show strong negative correlations with coverage, suggesting that increased structural complexity is associated with poorer testability. Conversely, CD consistently exhibits a positive correlation, indicating that better-documented code tends to have higher coverage. In contrast, unsustainable projects display weaker or even reversed correlations—for instance, NMC and AMC (Average Method Complexity) correlate positively with coverage, which may reflect inconsistent or inflated metric distributions in projects lacking long-term maintenance. The metric RFC, positively correlated with coverage in sustainable projects, turns negative in unsustainable ones, underscoring a possible breakdown in cohesion between code structure and testability. Overall, sustainable projects exhibit more coherent and interpretable correlations, while unsustainable ones demonstrate erratic or contradictory patterns, reinforcing the idea that well-maintained codebases tend to follow better engineering and testing practices.
The Cliff’s Delta plot comparing software metrics between sustainable and unsustainable projects (based on t-year sustained activity) in Figure 6 shows that sustainable projects tend to have higher modularity and test coverage (NMC, NMCE, NOA, FC) and slightly more function and branch coverage (LC, BC), as indicated by positive effect sizes. In contrast, unsustainable projects exhibit significantly higher values in documentation-related metrics (LOCCOM, CD), coupling and complexity measures (ACC, AMC, CBO, SC), and overall size (LOC, RFC, WMC). Metrics like LCOM4, NOB, and NMCI show negligible differences, suggesting they are less predictive of sustainability.
In both classification schemes, sustainable projects show stronger and more consistent correlations between code structure and test coverage, particularly with complexity and documentation metrics. Unsustainable projects, by contrast, exhibit weaker or inconsistent relationships, suggesting less disciplined testing practices and structural design. Figure 7: Cliff’s Delta on TF sustainability classification. We further investigate whether different metrics can help distinguish between sustainable and unsustainable projects, aiming to understand if such metrics offer insights into project longevity. To assess these differences, we perform two-sided Mann–Whitney U tests and visually examine the distributions. To account for the multiple comparisons across various metrics, we apply the BenjaminiHochberg procedure [7], which offers greater statistical power
Figure 7 presents the Cliff’s Delta effect sizes for various code and testability metrics when comparing sustainable and unsustainable projects. Positive values indicate higher metric values in sustainable projects, while negative values indicate the opposite. The results show that sustainable projects exhibit substantially higher structural complexity (SC), coupling (CBO, ACC), class-level response
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
(RFC), external function interaction (NMCE), and comment-related metrics (LOCCOM, CD), with medium to large effect sizes. This suggests that sustainable projects tend to be more complex, interconnected, and better documented. In contrast, test coverage metrics, such as function (FC), line (LC), and branch coverage (BC), are slightly higher in unsustainable projects, though the effect sizes are small. This may reflect simpler codebases or limited feature evolution in unsustainable projects. Overall, sustainable projects demonstrate more mature and intricate code structures, while unsustainable ones show marginally better test coverage. Result Sustainable projects tend to exhibit higher structural complexity, better documentation, greater modularity, and more consistent correlations between code metrics and test coverage. While projects classified as sustainable under the t-year criterion show higher test coverage, those deemed unsustainable by the TF-based method display slightly better coverage, likely due to concentrated early development. Overall, mature architecture, modular design, and strong documentation, not coverage alone, serve as more reliable indicators of long-term sustainability.
4.2
RQ2: Does using a stricter threshold for sustained activity improve the reliability of sustainability classifications based on test quality metrics?
Despite observing meaningful correlations between structural metrics and test coverage in sustainable projects (as defined in RQ1), the classification method significantly influenced the interpretation of test coverage trends. According to the t-year criterion, which requires long-term development activity, sustainable projects consistently exhibited higher test coverage across all metrics (line, function, and branch), suggesting that enduring maintenance often coincides with better testing practices. In contrast, the TF-based classification, which focuses on developer continuity, yielded the opposite trend: unsustainable projects showed higher test coverage. This apparent contradiction implies that test coverage alone may not be a definitive signal of sustainability. To further investigate this assumption, we apply a stricter threshold for sustainability classification, following the methodology proposed by Xiao et al.[57]. In their study, the authors develop a machine learning model to predict the long-term sustainability of open-source GitHub projects based on early behavioral signals. They define a project as sustainable if it remains active for at least two years (𝑡 = 2) and maintains a median number of commits per month greater than or equal to a threshold 𝑘, where 𝑘 controls the strictness of the sustainability label. In Section 4.3 (Sensitivity Analysis), they examine the impact of increasing 𝑘 (e.g., from 1 to 2 to 6) on classification performance and find that stricter thresholds improve model accuracy, as reflected in higher AUC and F1 scores. These findings support the idea that a more rigorous sustainability definition reduces label noise and leads to more reliable distinctions between sustainable and unsustainable projects. Motivated
Rahman et al.
by this, we conducted additional experiments using increasingly strict 𝑘-values to refine our sustainability labels and test whether the observed inconsistencies in coverage trends persist under more robust classification criteria. The series of violin plots from Figure 8 illustrates how test coverage varies between sustainable and unsustainable projects under increasing thresholds of commit frequency. At lower thresholds (e.g., 2 and 4 commits per month), unsustainable projects surprisingly exhibit higher median coverage values, suggesting that some dormant or less active projects were once well-tested. However, as the threshold becomes more stringent (e.g., 8, 16, and 32), this pattern reverses: sustainable projects show consistently higher median coverage across all three dimensions—line, function, and branch—while unsustainable projects drop to near-zero coverage. This shift indicates that truly active and frequently maintained projects are more likely to invest in and maintain systematic testing infrastructure. Thus, under stricter definitions of sustained activity as a proxy of sustainability, test coverage becomes a more reliable indicator of sustained project health and engineering discipline. Figure 9 presents Cliff’s Delta values across different sustainability thresholds (𝐶 Th ), comparing sustainable and unsustainable projects. Across all thresholds, sustainable projects consistently show higher values in metrics related to testing and documentation, such as FC, LC, BC, CD, and LOCCOM, indicating a stronger emphasis on code quality and maintainability. In contrast, structural and complexity-related metrics like SC, WMC, CBO, and RFC tend to be higher in unsustainable projects, suggesting greater code complexity and interdependence. Notably, the NMC is substantially higher in sustainable projects, especially at higher thresholds, reflecting their modular and reusable design practices. As the threshold decreases, the effect sizes of structural metrics diminish, while the positive association between coverage, documentation, and sustainability remains robust. Result Initial inconsistencies in TF-based results, where some unsustainable projects showed higher coverage, motivated a re-evaluation using stricter thresholds. As the commitfrequency threshold (𝑘) increased, sustainable projects showed higher median coverage, while unsustainable projects dropped toward zero. This confirms that stricter activity-based definitions better capture sustainable behavior. Effect-size analyses further show that sustainable projects favor modularity, documentation, and higher coverage, whereas unsustainable ones exhibit greater complexity and coupling. Overall, refined criteria strengthen the link between code quality and sustainability.
4.3
RQ3: Is there any strong relationship between code metrics and test coverage in scientific software overall?
The violin plots in Figure 10 illustrate the distribution of test coverage metrics—line, function, and branch coverage—across all projects. The median values reveal generally low coverage, with line coverage at 21%, function coverage at 50%, and branch coverage at
Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics
PASC 2026, June 29–July 1, 2026, Bern, Switzerland Metric Sensitivity to Threshold (Cth) Large Med
Metrics
LOC LC NOB BC NOM FC ACC CBO LCOM4 NOA RFC SC LOCCOM NMC NMCI NMCE WMC AMC CD
Cliff's Delta
Small 0 Small
Med Large 5
10
15
Cth
20
25
30
Figure 9: Comparing Cliff’s delta by changing threshold.
Figure 10: Test Coverage of the projects.
Figure 11: Heatmap for the full dataset. Figure 8: Coverage plots across increasing commit frequency thresholds.
just 14%. While function coverage appears relatively higher, the distributions for both line and branch coverage are heavily skewed toward zero, indicating that many projects have minimal or no test coverage. The function coverage distribution is notably bimodal, suggesting a divide between well-tested and poorly-tested projects. Overall, these results highlight inconsistent testing practices and suggest that significant improvements are needed, particularly in achieving adequate line and branch coverage. The heatmap in Figure 11 presents the Spearman correlation coefficients between various code metrics and three coverage metrics: line, branch, and function coverage percentages. Overall, most code metrics exhibit weak negative correlations with test coverage, suggesting that as certain structural or complexity-related
attributes of the code increase, coverage tends to decrease. Notably, the strongest negative correlations are observed with NMCE, which shows moderate negative associations with all three coverage types (e.g., 𝜌 = -0.35 with line coverage), indicating that code with more external dependencies may be harder to test thoroughly. Similarly, NMC also shows moderately negative correlations across the board. Metrics such as LCOM4, WMC, and RFC, which reflect cohesion and complexity, also have modest negative correlations. In contrast, metrics like AMLOC and CD show slight positive correlations, possibly indicating that more verbose and well-documented code may facilitate better testability. However, these correlations are weak and should be interpreted cautiously. The overall trend supports the notion that higher code complexity and coupling are generally associated with lower test coverage across scientific software projects.
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
Result Overall, test coverage in scientific software is low and uneven, with many projects lacking adequate line and branch coverage. Correlation analysis shows that higher code complexity and coupling (e.g., NMCE, NMC) are moderately associated with lower test coverage, while documentation and verbosity show slight positive links. These findings suggest that structural complexity hampers testability across the board.
5
Discussion
Scientific software sometimes exhibits intricate, non-deterministic behavior with numerous execution paths and extensive input demands, making it challenging to manually define key input boundaries or effectively partition the input space for testing purposes [22]. Developers of such software often have a limited understanding of standard software engineering testing practices and may prioritize producing results over ensuring software quality, driven by budget constraints or the exploratory nature of research [30]. Moreover, many lack formal training in software engineering, treating testing as a secondary activity [46]. Our experimental observations are consistent with these findings: the scientific software under study demonstrates poor test coverage in its existing test suite, underscoring the persistent difficulties in achieving robust testing for research-oriented codebases. Beyond testing challenges, structural and cultural issues further hinder sustainability. Software citation is not yet a standard practice in research workflows; software is often not published in a citable form, and metadata quality and persistence remain unreliable. Support from publishers and funders is limited, software is not consistently recognized as a valued research output, credit attribution is ambiguous, and dependencies are frequently overlooked. These technical and social challenges reinforce one another, impeding proper maintenance and testability [21]. Overall, our analysis reveals that while sustainable projects tend to achieve higher test coverage under time-based sustainability measures (t-year criterion), this trend does not hold under the TFbased classification, indicating that high coverage alone is not a sufficient marker of long-term project health. Across both classification methods, sustainable projects consistently demonstrate stronger correlations between structural quality metrics, such as complexity, modularity, and documentation, and test coverage. This observation suggests that these projects adopt more disciplined engineering and testing practices. Conversely, unsustainable projects show weaker or inconsistent relationships, implying less structured development and testing efforts. When stricter activity-based sustainability thresholds are applied, sustainable projects maintain higher coverage and structural integrity, while unsustainable ones show a significant decline, confirming that ongoing development activity better reflects sustainability. Overall, the study emphasizes that true software sustainability is rooted not merely in test coverage but in robust architecture, strong documentation, and consistent maintenance practices that promote long-term viability and adaptability.
Rahman et al.
In addition, our investigation of projects from the CASS Portfolio revealed that a significant number lacked proper containerization and updated build documentation. This finding corroborates prior research showing that research software often suffers from poor reproducibility, portability, and deployment support, which are factors that critically undermine long-term sustainability [38].
5.1
RQ1: Practical Implications
The findings from RQ1 suggest that sustainable projects exhibit stronger relationships between code structure and testing practices, along with more consistent quality patterns. Practically, this implies that developers should not treat testing and code design as separate concerns. Instead, they should adopt integrated engineering practices where modularity, low coupling, and clear structure are prioritized alongside test development. For project maintainers and funding agencies, these results highlight the value of monitoring internal code metrics (e.g., complexity, cohesion, coverage trends) as early warning signals of declining sustainability. Tooling support can also be developed to automatically flag projects with poor structural-test alignment, enabling proactive intervention before sustainability issues become critical.
5.2
RQ2: Practical Implications
The results of RQ2 demonstrate that stricter activity-based definitions of sustainability lead to more consistent and reliable interpretations of quality metrics. This has important implications for both researchers and practitioners. For researchers, it emphasizes the need to carefully select and justify sustainability definitions to avoid misleading conclusions. For practitioners and policymakers (e.g., funding bodies, open-source governance organizations), it suggests that simple activity thresholds (such as commit frequency) can serve as practical proxies for identifying genuinely sustainable projects. Additionally, project evaluation frameworks should incorporate stricter criteria to better distinguish between actively maintained and superficially active or legacy systems. This can improve decision-making in resource allocation, project adoption, and long-term support strategies.
5.3
RQ3: Practical Implications
The findings from RQ3 indicate that higher code complexity and coupling are generally associated with lower test coverage, while better documentation and modularity may support improved testability. This has direct implications for software engineering practices in scientific domains. Developers should actively manage complexity and reduce interdependencies to make their code more testable. Practices such as modular design, refactoring, and improved documentation can lower the barriers to writing effective tests. For tool builders, these insights open opportunities to create automated tools that predict testability issues based on static code metrics, enabling early intervention. For educators and research institutions, the results reinforce the importance of training scientists in software engineering best practices, particularly in designing testable and maintainable code.
Exploring Sustainability in Scientific Software through Code Quality & Test Coverage Metrics
6
Threats to Validity
Construct Threats. The construct threat pertains to how sustainability is defined for the SciOSS. In this study, we derived the sustainability labels from prior research and validated them against the literature [5, 27, 54, 57]. Sustained development activity and developer continuity are used as operational proxies for sustainability. Both are supported by prior work as reliable indicators of long-term maintenance and engagement. By employing two complementary classification methods validated in prior studies, we ensure a multidimensional perspective and reduce construct bias, thereby enhancing the validity of the sustainability construct. External Threats. External threats concern the generalizability of the findings. The dataset is drawn from the CORSA catalog, which focuses on high-quality scientific software projects written in C/C++. While this ensures strong relevance to scientific computing, it limits applicability to other ecosystems, such as Python-based data analysis or Java enterprise software. Moreover, our dataset includes only 14 projects—the subset that could be successfully built, instrumented for coverage, and analyzed with a uniform metric-extraction pipeline. This necessarily excludes projects with complex MPI/GPU configurations, specialized hardware requirements, missing or non-runnable tests, or build systems that resist reproducible compilation. As a result, the dataset overrepresents well-engineered, actively maintained projects and underrepresents highly experimental, abandoned, or hard-to-build codebases, introducing survivorship and instrumentation bias common in empirical studies of scientific software. While these constraints limit broad generalizability, they ensure consistent, reproducible metric collection, which is essential for structural and test-quality analysis. Future work will expand CORSA and explore techniques for instrumenting more complex or hardware-dependent projects. Internal Threats. Internal threats relate to the robustness of data collection and statistical inference. We used non-parametric statistical tests (Mann–Whitney U, Spearman’s rank, and Cliff’s delta) due to the non-normality of metric distributions, verified with the D’Agostino 𝑘 2 test. The Benjamini–Hochberg correction controls for false discovery in multiple comparisons. However, the small sample size (14 projects) limits statistical power, making it harder to detect small effects and increasing uncertainty in marginal results. Despite rigorous testing, the limited dataset constrains internal reliability. Conclusion Threats. To mitigate measurement and instrumentation bias, the study uses a standardized, tool-agnostic metric extraction pipeline centered on Analizo [50]. This framework normalizes both procedural and object-oriented C/C++ projects by modeling each translation unit as a class-like module, ensuring consistent computation of structural metrics (e.g., CBO, LCOM4, WMC, NOM). The use of complementary tools, such as Lizard, Cloc, and custom Clang AST analyzers, further strengthens metric accuracy and coverage across complexity, documentation, and function calls. These design choices enhance measurement consistency and, consequently, the reliability of the study’s conclusions. We also acknowledge limitations in the structural metrics derived for this study. OO-based measures such as LCOM4, CBO, and RFC were developed for class-oriented systems and only approximate cohesion and coupling in procedural C, where they largely capture
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
patterns of global-state sharing or inter-function interaction rather than classical OO semantics. Likewise, our call metrics (NMC, NMCI, NMCE) rely on per–translation unit Clang AST analysis, which under-approximates call relationships involving function pointers, macro expansions, templated or inlined code, and cross-unit interactions—well-known challenges in static call-graph construction for C/C++. Accordingly, we interpret both cohesion/coupling and call metrics as conservative, relative indicators of structural organization rather than complete or absolute measures of modular quality.
7
Conclusion
Sustaining SciOSS is critical for ensuring the reproducibility, reliability, and long-term impact of computational research. This study investigated the relationship between internal software quality and the sustainability of SOSS projects. By analyzing C/C++ projects from the CORSA Catalog, we classified projects using two validated methods and compared their structural and test quality characteristics. Our results show that sustainable projects tend to be better tested, more modular, and less complex than unsustainable ones. Additionally, stricter sustainability thresholds yield more consistent trends in quality metrics, suggesting that refined definitions can enhance classification reliability. We also find that several code metrics are moderately correlated with test coverage, indicating that structural properties may serve as proxies for testing effort and software maintainability. These findings highlight the importance of incorporating internal quality assessments into sustainability evaluations and can inform future efforts to support the development of robust, maintainable, and enduring scientific software.
8
Acknowledgment & Data
We acknowledge the support of the Consortium for Open-Source Research Software Advancement (CORSA), a project supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Next-Generation Scientific Software Technologies program, under contract number DE-AC05-00OR22725. We share all code and data used in this study at https://github.com/Mushfiq344-utk/SciOSS_Sustainability.
References [1] [n. d.]. CVE - CVE-2021-44228 — cve.mitre.org. https://cve.mitre.org/cgi-bin/ cvename.cgi?name=CVE-2021-44228. [Accessed 04-08-2025]. [2] 2025. Consortium for the Advancement of Scientific Software. https://cass. community/. Accessed: 2025-12-12. [3] Hartwig Anzt, Felix Bach, Stephan Druskat, Frank Löffler, Axel Loewe, Bernhard Y Renard, Gunnar Seemann, Alexander Struck, Elke Achhammer, Piush Aggarwal, et al. 2021. An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action. F1000Research 9 (2021). [4] Oliver Arafat and Dirk Riehle. 2009. The comment density of open source software code. In 2009 31st International Conference on Software Engineering-Companion Volume. IEEE, 195–198. [5] Guilherme Avelino, Eleni Constantinou, Marco Tulio Valente, and Alexander Serebrenik. 2019. On the abandonment and survival of open source projects: An empirical investigation. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–12. [6] Guilherme Avelino, Leonardo Passos, Andre Hora, and Marco Tulio Valente. 2016. A novel approach for estimating truck factors. In 2016 IEEE 24th International Conference on Program Comprehension (ICPC). IEEE, 1–10. [7] Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) 57, 1 (1995), 289–300.
PASC 2026, June 29–July 1, 2026, Bern, Switzerland
[8] Magiel Bruntink and Arie van Deursen. 2006. An empirical study into class testability. Journal of systems and software 79, 9 (2006), 1219–1232. [9] Jeffrey C Carver, Nic Weber, Karthik Ram, Sandra Gesing, and Daniel S Katz. 2022. A survey of the state of the practice for research software in the United States. Peerj computer science 8 (2022), e963. [10] InduShobha Chengalur-Smith, Anna Sidorova, and Sherae L Daniel. 2010. Sustainability of free/libre open source projects: A longitudinal study. Journal of the Association for Information Systems 11, 11 (2010), 5. [11] Shyam R Chidamber and Chris F Kemerer. 1994. A metrics suite for object oriented design. IEEE Transactions on software engineering 20, 6 (1994), 476–493. [12] Jailton Coelho and Marco Tulio Valente. 2017. Why modern open source projects fail. In Proceedings of the 2017 11th Joint meeting on foundations of software engineering. 186–196. [13] Jailton Coelho, Marco Tulio Valente, Luciana L Silva, and Emad Shihab. 2018. Identifying unmaintained projects in github. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. [14] Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. routledge. [15] Constorium for the Advancement of Scientific Software (CASS). 2024. CASS Software Portfolio. https://cass.community/software/. Accessed: July 2025. [16] Gregory W Corder and Dale I Foreman. 2009. Nonparametric statistics for non-statisticians. (No Title) (2009). [17] Ralph B D’agostino, Albert Belanger, and Ralph B D’Agostino Jr. 1990. A suggestion for using powerful and informative tests of normality. The American Statistician 44, 4 (1990), 316–321. [18] Albert Danial. 2021. cloc: v1.92. doi:10.5281/zenodo.5760077 [19] David P Darcy, Chris F Kemerer, Sandra A Slaughter, and James E Tomayko. 2005. The structural complexity of software an experimental test. IEEE Transactions on software engineering 31, 11 (2005), 982–995. [20] Edson Dias, Paulo Meirelles, Fernando Castor, Igor Steinmacher, Igor Wiese, and Gustavo Pinto. 2021. What makes a great maintainer of open source projects?. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 982–994. [21] Stephan Druskat, Daniel S Katz, and Ilian T Todorov. 2021. Research Software Sustainability and Citation. In 2021 IEEE/ACM International Workshop on Body of Knowledge for Software Sustainability (BoKSS). IEEE, 1–2. [22] Nasir U Eisty, Danny Perez, Jeffrey C Carver, J David Moulton, and Hai Ah Nam. 2020. Testing research software: A case study. In International Conference on Computational Science. Springer, 457–463. [23] Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1994. Design Patterns: Micro-Architectures for Reusable Object-Oriented Design. Reading: Addison-Wesley (1994). [24] Amir Hossein Ghapanchi. 2015. Predicting software future sustainability: A longitudinal perspective. Information Systems 49 (2015), 40–51. [25] Sean Goggins, Kevin Lumbard, and Matt Germonprez. 2021. Open source community health: Analytical metrics and their corresponding narratives. In 2021 IEEE/ACM 4th International Workshop on Software Health in Projects, Ecosystems and Communities (SoHeal). IEEE, 25–33. [26] Robert J Grissom and John J Kim. 2005. Effect sizes for research: A broad practical approach. Lawrence Erlbaum Associates Publishers. [27] Junxiao Han, Yunkun Wang, Zhongxin Liu, Lingfeng Bao, Jiakun Liu, David Lo, and Shuiguang Deng. 2024. Sustainability forecasting for deep learning packages. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 981–992. [28] Jan Hauke and Tomasz Kossowski. 2011. Comparison of values of Pearson’s and Spearman’s correlation coefficients on the same sets of data. Quaestiones geographicae 30, 2 (2011), 87–93. [29] Martin Hitz and Behzad Montazeri. 1995. Measuring coupling and cohesion in object-oriented systems. na. [30] Upulee Kanewala and James M Bieman. 2014. Testing scientific software: A systematic literature review. Information and software technology 56, 10 (2014), 1219–1232. [31] DS Katz. 2022. Research Software: Challenges & Actions. The Future of Research Software: International Funders Workshop, Amsterdam, the Netherlands. [32] Meir M Lehman. 2005. Programs, life cycles, and laws of software evolution. Proc. IEEE 68, 9 (2005), 1060–1076. [33] Robert Martin. 1994. OO design quality metrics. An analysis of dependencies 12, 1 (1994), 151–170. [34] Stewart Martin-Haugh, Stefan Kluth, Rolf Seuster, Scott Snyder, Emil Obreshkov, Shaun Roe, Peter Sherwood, and Graeme A Stewart. 2017. C++ software quality in the ATLAS experiment: tools and experience. In Journal of Physics: Conference Series, Vol. 898. IOP Publishing, 072011. [35] Thomas J McCabe. 1976. A complexity measure. IEEE Transactions on software Engineering 4 (1976), 308–320. [36] Paulo Roberto Miranda Meirelles. 2013. Monitoramento de métricas de códigofonte em projetos de software livre. (2013). [37] Vishal Midha and Prashant Palvia. 2012. Factors affecting the success of Open Source Software. Journal of Systems and Software 85, 4 (2012), 895–905.
Rahman et al.
[38] Emmanuel Nyenah, Petra Döll, Daniel S Katz, and Robert Reinecke. 2024. Software sustainability of global impact models. Geoscientific Model Development Discussions 2024 (2024), 1–29. [39] Marc Oriol, Carlos Mueller, Jordi Marco, Pablo Fernandez, Xavier Franch, and Antonio Ruiz-Cortes. 2023. Comprehensive assessment of open source software ecosystem health. Internet of Things 22 (2023), 100808. [40] COVO Pearson’s. 2011. Comparison of values of Pearson’s and Spearman’s correlation coefficients. Comparison Of Values Of Pearson’s And Spearman’s Correlation Coefficients (2011). [41] Huilian Sophie Qiu, Alexander Nolte, Anita Brown, Alexander Serebrenik, and Bogdan Vasilescu. 2019. Going farther together: The impact of social capital on sustained participation in open source. In 2019 ieee/acm 41st international conference on software engineering (icse). IEEE, 688–699. [42] Robert Reinecke, Tim Trautmann, Thorsten Wagener, and Katja Schüler. 2022. The critical need to foster computational reproducibility. Environmental Research Letters 17, 4 (2022), 041005. [43] Robert Reinecke, Tim Trautmann, Thorsten Wagener, and Katja Schüler. 2022. The critical need to foster computational reproducibility. Environ. Res. Lett. 17, 4 (April 2022), 041005. [44] Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, and Jeff Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys. In annual meeting of the Florida Association of Institutional Research, Vol. 177. [45] Felix Schwarz and contributors. 2024. gcovr: Code coverage reporting for C/C++ projects. https://gcovr.com. Version 6.0, Accessed: July 2025. [46] Judith Segal. 2007. Some problems of professional end user developers. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2007). IEEE, 111–118. [47] S, tefan Stănciulescu, Likang Yin, and Vladimir Filkov. 2022. Code, quality, and process metrics in graduated and retired asfi projects. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 495–506. [48] Xin Tan, Minghui Zhou, and Zeyu Sun. 2020. A first look at good first issues on GitHub. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 398–409. [49] Mei-Huei Tang, Ming-Hung Kao, and Mei-Hwa Chen. 1999. An empirical study on object-oriented metrics. In Proceedings sixth international software metrics symposium (Cat. No. PR00403). IEEE, 242–249. [50] Antonio Terceiro, Joenio Costa, João Miranda, Paulo Meirelles, Luiz Romário Rios, Lucianna Almeida, Christina Chavez, and Fabio Kon. 2010. Analizo: an extensible multi-language source code analysis and visualization toolkit. In Brazilian conference on software: theory and practice (Tools Session). [51] Antonio Terceiro, Manoel Mendonça, Christina Chavez, and Daniela S Cruzes. 2012. Understanding structural complexity evolution: A quantitative analysis. In 2012 16th European Conference on Software Maintenance and Reengineering. IEEE, 85–94. [52] Valerio Terragni, Pasquale Salza, and Mauro Pezzè. 2020. Measuring software testability modulo test quality. In Proceedings of the 28th international conference on program comprehension. 241–251. [53] Fadel Toure, Mourad Badri, and Luc Lamontagne. 2018. Predicting different levels of the unit testing effort of classes using source code metrics: a multiple case study on open-source software. Innovations in Systems and Software Engineering 14, 1 (2018), 15–46. [54] Marat Valiev, Bogdan Vasilescu, and James Herbsleb. 2018. Ecosystem-level determinants of sustained activity in open-source projects: A case study of the PyPI ecosystem. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 644–655. [55] Laurie Williams and Robert R Kessler. 2003. Pair programming illuminated. Addison-Wesley Professional. [56] Titus Winters, Tom Manshreck, and Hyrum Wright. 2020. Software engineering at google: Lessons learned from programming over time. " O’Reilly Media, Inc.". [57] Wenxin Xiao, Hao He, Weiwei Xu, Yuxia Zhang, and Minghui Zhou. 2023. How early participation determines long-term sustained activity in github projects?. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 29–41. [58] Likang Yin, Zhuangzhi Chen, Qi Xuan, and Vladimir Filkov. 2021. Sustainability forecasting for apache incubator projects. In Proceedings of the 29th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 1056–1067. [59] Yuxia Zhang, Minghui Zhou, Audris Mockus, and Zhi Jin. 2019. Companies’ participation in oss development–an empirical study of openstack. IEEE Transactions on Software Engineering 47, 10 (2019), 2242–2259.