Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study JOSHUA OWOTOGBE, Jheronimus Academy of Data Science, Netherlands and Tilburg University, Netherlands INDIKA KUMARA, Jheronimus Academy of Data Science, Netherlands and Tilburg University, Netherlands WILLEM-JAN VAN DEN HEUVEL, Jheronimus Academy of Data Science, Netherlands and Tilburg University, Netherlands

arXiv:2608.28396v1 [cs.SE] 28 Aug 2026

DAMIAN TAMBURRI, Jheronimus Academy of Data Science, Netherlands and University of Sannio, Italy Robustness evaluation is essential for deploying machine-learning (ML) systems in real-world settings, where models may face adversarial perturbations, distribution shifts, and other operational stressors. Many open-source tools, including Adversarial Robustness Toolbox, Foolbox, and Robustness Gym, support robustness testing and evaluation. However, little is known about how these tools are maintained, publicly engaged with, and sustained over time, even though practitioners may rely on them to select evaluation dependencies, reproduce robustness assessments, and provide evidence for AI assurance. We present an empirical study of the opensource robustness tooling ecosystem. Starting from a curated seed set derived from prior work, we systematically searched GitHub and identified 28 robustness-tool repositories. We analyzed repository artifacts to characterize observable community engagement, maintenance activity, and project longevity using established software-engineering metrics. Our results show that engagement and maintenance are unevenly distributed, with sustained activity concentrated in a small subset of repositories. At the data collection date of January 21, 2026, five repositories were classified as active, 22 as inactive, and one as archived. These findings highlight the need to treat robustness tools as evolving software systems. CCS Concepts: • Software and its engineering → Software libraries and repositories. Additional Key Words and Phrases: Robustness, Reliability, Adversarial Testing, Open-Source Software, Software Sustainability, Tool Maintenance, GitHub

1

Introduction

Machine-learning (ML) components are increasingly integrated into software systems that support operational decisionmaking in healthcare, finance, autonomous systems, and large-scale online services, including safety-critical contexts [5, 57, 68]. These systems often operate under conditions that differ from controlled training and testing settings, including noisy, incomplete, adversarial, or distribution-shifted inputs [62, 79, 87]. ML models can exhibit brittle behavior under such conditions [15, 37, 74], and failures in high-stakes settings can result in degraded performance, unexpected system behavior, and significant harm [57, 65]. Ensuring robustness has therefore become a prerequisite for dependable deployment of ML-enabled systems [16, 68, 88]. Open-source robustness tools help researchers and practitioners test, evaluate, and improve model behavior under stress [53, 66]. These tools operationalize robustness through adversarial testing, perturbation analysis, stress testing, and out-of-distribution evaluation [15, 24, 37]. Authors’ Contact Information: Joshua Owotogbe, j.s.owotogbe@tilburguniversity.edu, Jheronimus Academy of Data Science, ’s-Hertogenbosch, North Brabant, Netherlands and Tilburg University, Tilburg, North Brabant, Netherlands; Indika Kumara, i.p.k.weerasinghadewage@tilburguniversity.edu, Jheronimus Academy of Data Science, ’s-Hertogenbosch, North Brabant, Netherlands and Tilburg University, Tilburg, North Brabant, Netherlands; WillemJan van den Heuvel, w.j.a.m.vdnHeuvel@tilburguniversity.edu, Jheronimus Academy of Data Science, ’s-Hertogenbosch, North Brabant, Netherlands and Tilburg University, Tilburg, North Brabant, Netherlands; Damian Tamburri, datamburri@unisannio.it, Jheronimus Academy of Data Science, ’s-Hertogenbosch, North Brabant, Netherlands and University of Sannio, Benevento, Italy.

1

© 2026 The Authors. This is the authors’ preprint version of the work.

2

Owotogbe et al. However, robustness tools are also software components that ML evaluation pipelines may depend on. Their

usefulness therefore depends not only on the attacks, defenses, or evaluation procedures they implement, but also on whether they remain compatible with evolving ML frameworks, model interfaces, libraries, datasets, and evaluation practices [16, 28, 57, 61, 69]. A tool that is no longer maintained may become difficult to run, reproduce, or trust, even if it remains visible, cited, or historically important [4, 13, 20, 82]. This paper studies the sustainability of open-source ML robustness tools. We mined 28 publicly available GitHub repositories to examine their functional scope, community engagement, maintenance activity, and project longevity. Our findings show that engagement and recent development activity are concentrated in a small subset of repositories. At the data collection date of January 21, 2026, five repositories were classified as active, 22 as inactive, and one as archived under a 180-day commit-recency rule. In the primary Kaplan-Meier analysis of the 27 non-archived repositories, the estimated median time to 180-day inactivity was 40.63 months, or approximately 3.39 years. These findings show that historical visibility and project age should not be used alone to infer continued maintenance. This concern is practical. ML systems are frequently retrained, dependencies evolve, and evaluation pipelines require adaptation [5, 28, 69]. The adversarial landscape also changes as new attack methods, threat models, and defense evaluations are introduced [15, 58]. In safety-critical domains, outdated robustness evaluations may provide a false sense of security if they do not reflect current attack methods or recent breaking changes in underlying ML frameworks [17, 65, 86]. For example, CleverHans (a library for adversarial testing of ML systems) [19] accumulated substantial historical GitHub engagement in our dataset, including over 6,400 stars and 185 watchers, but it was classified as inactive at the data collection date because its default branch contained no commits during the final 180 days of observation. This contrast illustrates that repository visibility does not necessarily imply current development activity or compatibility with current ML frameworks. Prior work has advanced the definition, classification, and benchmarking of robustness through formal and algorithmic methods, and comparative surveys have described robustness tools at a feature level [24, 38, 52]. However, they predominantly treat robustness as a property to be analyzed or certified, rather than as a software capability embodied in tools that must evolve alongside models, libraries, and deployment environments. The most closely related repositorymining study is by Mim et al. [51], who examined open-source fairness tools to characterize community participation, maintenance practices, and project sustainability on GitHub. We adapt several elements of their empirical design, including repository discovery, the 12 maintenance features, the 24-month analysis window, and the Mann-Whitney U comparison. We extend this empirical perspective to ML robustness tools, whose maintenance depends on both evolving ML frameworks and evolving adversarial evaluation practices [58]. To the best of our knowledge, no prior study has systematically analyzed open-source ML robustness tools with respect to their maintenance activity, community engagement, and long-term sustainability within the GitHub ecosystem. This gap has implications for dependency selection, compatibility testing, release management, provenance tracking, and maintenance planning in ML evaluation pipelines. It also matters for AI assurance documentation. Artificial Intelligence Bills of Materials (AIBOMs) have been proposed as structured inventories of the models, datasets, software components, and tools used in AI systems, supporting provenance tracking and regulatory oversight [54, 63]. Because robustness tools may provide evidence that an ML component has undergone robustness evaluation, AIBOM records should capture not only the tool name, but also the repository, commit or release version, maintenance state, data collection date, supported framework versions, and dependency environment. The goal of this work is to understand and guide the sustainability of open-source ML robustness tool projects. We address the following research questions:

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

3

RQ1: Which ML robustness tools have been released as open-source projects? RQ1 identifies publicly available open-source ML robustness tools and characterizes the robustness tooling ecosystem. We identify 28 tools released between 2016 and 2025, predominantly academic and Python-based. Most tools focus on deep learning models and evasion attacks, while support for natural language processing, large language models, federated learning, graph learning, and multimodal vision-language models is more limited. RQ2: To what extent do open-source robustness tool projects engage with their communities? RQ2 examines publicly observable GitHub engagement through stars, watchers, forks, and pull requests. These measures capture visible interest and collaboration on GitHub, but do not directly measure package use, downstream dependence, private deployment, or industrial adoption [13, 82]. We find that engagement is highly concentrated: a small subset of repositories accounts for most stars, watchers, forks, and pull-request activity, while many repositories show comparatively limited public participation. RQ3: How actively are open-source robustness tool projects maintained? RQ3 examines ongoing maintenance activity using development-oriented indicators, including commit activity, contributor participation, issue handling, and pull-request activity [21, 51]. We find that five of the 28 repositories (17.9%) were active, 22 (78.6%) were inactive, and one (3.6%) was archived on the data collection date. Exploratory comparisons between active and inactive repositories showed differences in nine of the 12 maintenance features before correcting for multiple tests, with six features remaining significant after Holm correction. Maintenance-related records frequently concerned API changes, changelog updates, deprecations, refactorings, integrations, and backward compatibility. RQ4: How long do open-source robustness tool projects remain under active development? RQ4 evaluates project longevity as a temporal indicator of sustainability by measuring development timelines and computing the survivor function (S(t)) to quantify the probability of a tool remaining active over time [4]. The primary Kaplan Meier analysis was conducted on the 27 non-archived repositories, with 22 observed inactivity events and five right-censored active repositories. The estimated median time to 180-day inactivity was 40.63 months, or approximately 3.39 years. The estimated probability of remaining active was 92.44% after one year and 30.42% after five years. The contributions of this paper are fourfold: • A curated catalog and publicly available dataset of 28 open-source ML robustness tools developed by academic, industry, and individual contributors, including their functional scope, repository characteristics, and maintenance status. • An empirical analysis of publicly observable GitHub engagement, maintenance activity, and project survival, including repository-level comparisons and a Kaplan-Meier analysis of commit-based project longevity. • A replication package, publicly available in the online appendix (Section 4.4), containing the data-collection and analysis scripts, the curated list of robustness-tool repositories, and the datasets used to address each research question. Paper Organization. Section 2 introduces the concepts of ML robustness, robustness tools, and open-source software sustainability. Section 3 reviews prior research on open-source repository mining and robustness tooling. Section 4 describes the methodology used to identify ML robustness tools and analyze repository-level artifacts. Section 5 presents the empirical findings on tool characteristics, community engagement, maintenance activity, and project lifespan. Section 6 discusses the implications of these findings. Section 7 outlines threats to validity. Finally, Section 8 concludes the paper and discusses future research directions.

4

Owotogbe et al.

2

Background

This section introduces the concepts and context necessary to understand our study. We describe machine learning robustness and the categories of tools used to evaluate it, the role these tools play in the ML development lifecycle, and the notion of open-source software sustainability as it applies to tool projects hosted on GitHub. 2.1

Machine Learning Robustness

Machine learning models are trained on data drawn from limited development distributions, but are often deployed in environments where inputs may differ from the data observed during training and testing. Robustness refers to the capacity of a trained model to maintain reliable predictive performance under specified changes in its input data [15, 37]. These changes may arise from adversarial or non-adversarial conditions. Adversarial robustness concerns deliberate input modifications designed to mislead a model, such as adversarial examples crafted to cause misclassification or exploit model vulnerabilities [12, 15, 74]. Non-adversarial robustness concerns naturally occurring or synthetic input changes that are not introduced with malicious intent, such as sensor noise, image compression, lighting changes, weather effects, temporal drift, or other deployment-time variations [12, 37, 62]. Distribution shift can occur in both settings. In adversarial settings, the shift may be induced intentionally by an attacker. In non-adversarial settings, it may arise from changes in the deployment environment, data collection process, user behavior, or other naturally occurring factors [12]. In both cases, the central concern is whether the model’s predictive performance remains within an acceptable tolerance under the specified input changes. Failures under these conditions carry significant consequences in safety-critical application domains, including healthcare, autonomous systems, and financial services, where model outputs inform high-stakes decisions [57, 65]. In such settings, robustness failures can result in degraded performance, unexpected system behavior, and significant harm [68, 88]. Ensuring robustness has therefore become a recognized prerequisite for the dependable deployment of ML-enabled systems [16, 61]. 2.2

ML Robustness Tools in the ML Development Lifecycle

Robustness tools translate robustness concerns into executable workflows for testing, evaluation, and mitigation. These tools support tasks such as generating adversarial or perturbed inputs, applying defenses, and comparing model behavior across robustness benchmarks [24, 53, 66]. Many are implemented in Python and hosted on public version-control platforms such as GitHub, reflecting broader conventions in the ML software ecosystem [2, 58]. Robustness tools implement one or more of three core functions. Attack generation produces adversarial or perturbed inputs to probe model vulnerability across threat scenarios, including evasion attacks, poisoning attacks, and model extraction attacks [24, 53]. Defense application applies input preprocessing transformations, adversarial training procedures, or model-hardening techniques to improve model resilience against identified threats [53]. Robustness evaluation and benchmarking measures robustness indicators and enables comparison of model behavior across datasets and evaluation protocols [2, 24]. Most robustness tools target the model training and evaluation phases of the ML lifecycle, with a subset also supporting data preprocessing pipelines [5, 57]. Because these tools depend on the APIs and data abstractions of underlying ML frameworks such as PyTorch and TensorFlow, their continued operation requires tracking framework version changes and maintaining compatibility with evolving dependencies [28, 69]. A further maintenance obligation arises from the adversarial landscape itself: new attack methods are regularly introduced, and existing defenses are frequently shown to be circumventable. Robustness tools may therefore need

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

5

to incorporate updated techniques to remain practically relevant [15, 58]. This combination of framework evolution and changing adversarial evaluation practices creates recurring maintenance demands for robustness tool developers. Similar dependency and compatibility concerns occur in other software domains [51] 2.3

Open-Source Software Sustainability

Open-source software sustainability refers to the long-term capacity of a project to attract and retain contributors, respond to reported issues, incorporate code changes, and remain compatible with its software dependencies [21, 32]. A sustainable project evolves alongside the technical environment in which it operates, whereas a project with declining maintenance may become progressively misaligned with user needs and upstream libraries [20, 82]. In repository mining studies, sustainability is therefore assessed through observable artifacts available on public hosting platforms such as GitHub. Prior work commonly uses two categories of repository-level indicators to study the sustainability of open-source projects [41, 51]. The first category captures community engagement. GitHub stars indicate public interest in or awareness of a repository, watchers indicate interest in receiving project updates, forks indicate that users may want to inspect, modify, or extend the project, and pull requests indicate proposed code or documentation contributions [11, 26, 91]. The second category captures maintenance activity. Commit frequency, contributor count, issue-resolution time, and pull-request merge activity describe how frequently a project changes, how many people participate in its development, and how responsive the project is to reported issues or proposed changes [8, 13]. Based on these repository-level indicators, open-source repository studies commonly distinguish active, inactive, and archived projects [21, 51]. An active project exhibits development activity within a specified recent analysis window. An inactive project lacks activity under the operational criterion adopted by a study, although inactivity does not necessarily imply that the software is unusable or permanently abandoned. An archived project has been placed in read-only status by its owner on GitHub. In this study, we operationalize these categories using the GitHub archived flag and a reproducible six-month commit-recency rule, implemented as 180 days before the January 21, 2026, data collection date, as described in Section 4. These distinctions have practical implications for teams that depend on open-source robustness tools. A repository with no recent default-branch commits may still provide stable and reproducible functionality, but reduced observable development can increase uncertainty about future dependency updates, framework compatibility, vulnerability remediation, and support for emerging attack or evaluation methods. Archived repositories provide an additional repository-level indication, as their owners have explicitly set them to read-only status. Accordingly, maintenance state should be considered alongside tool coverage, version compatibility, and reproducibility when selecting a robustness tool for an ML workflow. Research-driven open-source projects may face sustainability challenges when development is closely tied to a publication or funding cycle and declines after the associated research effort concludes [21, 51]. Prior work has shown that project age alone is not a reliable indicator of continued viability: long-lived projects can become inactive while recently introduced projects may demonstrate sustained activity, making recency of contribution and breadth of the contributor base important indicators of long-term sustainability [4, 13]. These findings motivate our analysis of project lifespan and survival in Section 5. 3

Related Work

This work examines robustness tools developed and maintained as open-source projects on GitHub. Accordingly, we situate our study within two strands of prior research: studies that mine and analyze open-source repositories to

6

Owotogbe et al.

characterize software development practices and project sustainability, and work on robustness evaluation and tooling for machine learning systems. Prior work uses GitHub repositories as empirical data to study software development practices, maintenance activities, and project evolution [23, 35, 39]. Existing studies have analyzed artifacts such as commits, issues, and pull requests to characterize development dynamics and project activities over time [55, 64, 83, 89, 90]. Repository data have also been employed to construct and validate models that assess maintenance status, sustainability, and project health [7, 20, 30, 36, 70, 82]. Other work has focused on repositories within specific ecosystems or programming environments, including Java and .NET, to identify domain-specific development and maintenance practices [28, 82]. Some studies have complemented repository mining with developer surveys to gain deeper insight into development effort and contributor behavior [13, 42]. More recently, repository mining has been extended to study sustainability in MLadjacent tool ecosystems, including open-source fairness tools and chaos engineering tools [51, 56], establish that domain context significantly shapes maintenance dynamics and community engagement patterns. Open-source robustness tooling represents a distinct and as yet unstudied category within this emerging line of work, one characterized by the additional pressure of co-evolving with both ML framework releases and an advancing adversarial attack landscape. Several prior efforts have proposed tools to support robustness analysis and adversarial evaluation of machine learning models. Nicolae et al. [53] introduced an open-source Python library that implements adversarial attacks and defenses, detection techniques, and robustness metrics for evaluating and improving the security of ML models under adversarial settings. Croce et al. [24] proposed RobustBench, a benchmark and public leaderboard for assessing adversarial robustness in image classification. Sun et al. [72] presented CANARY, an evaluation platform that comprehensively measures adversarial robustness using a multi-dimensional scoring framework. Arcaini et al. [6] introduced ROBY, a tool for robustness analysis of neural network classifiers that extends a previously defined robustness notion based on plausible input alterations to support multiple data types and execution environments. Other work has examined robustness tooling and techniques through surveys and comparative evaluations. Abdalla et al. [1] surveyed definitions and techniques related to program robustness, including testing and measurement tools. Liu and Jin [46] provided a comprehensive survey of robust deep learning in computer vision, covering adversarial attacks, defenses, evaluation metrics, and architectural approaches. Pelekis et al. [58] present a survey of adversarial machine learning that reviews attack and defense methods, robustness benchmarks, and open-source tools, and examines robustness and privacy considerations in large language model-based systems. Agarwal and Nene [2] conducted a comparative review of eleven ML robustness toolboxes, using repository metadata and documentation to evaluate community support, popularity, and versatility for feature-oriented tool selection. Taken together, these comparative analyses treat robustness tools as static artifacts to be evaluated on feature coverage at a point in time; they do not examine whether tools remain actively maintained, whether their contributor communities sustain engagement over time, or whether tools continue to function correctly as the underlying ML frameworks and adversarial techniques evolve. Our study adapts established repository-mining procedures to a previously unexamined software ecosystem: opensource ML robustness tools. The most closely related work is the repository-mining study by Mim et al. [51], which examined open-source fairness tools to characterize community participation, maintenance practices, and project sustainability. However, fairness-oriented tools address technical objectives distinct from those of robustness tools, concentrating on bias identification and mitigation rather than adversarial threats, distributional stress, and robustness evaluation. Robustness tools must remain compatible with evolving ML frameworks while also incorporating changes in attack, defense, and evaluation practices [58]. These characteristics motivate examining robustness tools as a distinct software ecosystem. To the best of our knowledge, no prior study has systematically analyzed open-source ML robustness

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

7

tools with respect to their maintenance activity, community engagement, and long-term sustainability within the GitHub ecosystem. This gap has practical implications because practitioners lack empirical evidence about which robustness tools remain actively maintained and what repository-level signals indicate sustainability risk. 4

Methodology

This study follows an observational repository-mining design, using publicly observable GitHub artifacts as the primary data source [23, 39]. We adapt established repository-mining procedures from prior studies of open-source ML tool ecosystems [51, 56] and apply them to ML robustness tools. Figure 1 summarizes the main steps of the study: repository discovery, repository mining, and manual revision and classification. All repository metadata, issue, pull-request, commit, and status analyses were bounded by a common data collection date of January 21, 2026. Following prior repository-mining work that restricts longitudinal observations to complete temporal units, we exclude partial 2026 data from annual trend analyses and restrict these analyses to the complete calendar years 2019-2025 [48].

Robustness Tool Search

Repository Mining

Manual Revision and Classification

Seed Toolset

Repository Search

False Positive Removal

GitHub API Search

Data Extraction GitHub REST API

Classification RQ1-RQ4 Analysis

Fig. 1. Overview of the research methodology, consisting of three phases: robustness tool search, repository mining, and manual revision and classification.

4.1

Robustness Tool Search

A common strategy for repository-mining studies is to begin with a representative set of established tools [51, 56, 81]. We performed a systematic search for open-source ML robustness tools in two steps. Seed Toolset. We began with 11 robustness tools identified in a prior comparative analysis of ML robustness toolboxes [2], which we refer to as the Seed Toolset. These tools, including the Adversarial Robustness Toolbox, Foolbox, and CleverHans, serve as the baseline for our analysis of established robustness tooling in academic and practitioner literature. Complementary GitHub Search. During the study, we observed that several robustness tools actively used on GitHub were absent from prior comparative analyses. Related repository-mining studies have reported similar gaps [51, 56, 81]. We therefore conducted a complementary GitHub search to identify tools beyond the Seed Toolset. We constructed a vocabulary of 43 search terms from the README files, About sections, package names, module names, and documented import statements of the 11 Seed Toolset repositories. The vocabulary covered four categories. First, it included general robustness concepts, such as AI robustness, machine learning robustness, model robustness, robustness evaluation, and machine learning security. Second, it covered adversarial-ML terminology, including adversarial attacks, adversarial robustness, LLM robustness attack, and graph adversarial attack. Third, it included framework- and organization-specific combinations, such as TensorFlow adversarial robustness, PyTorch adversarial, IBM adversarial, and Microsoft adversarial.

8

Owotogbe et al.

Finally, it contained tool names, package identifiers, and Python import expressions, including foolbox, advertorch, textattack, deeprobust, import art, and from art.attacks.evasion. We removed exact duplicates but retained singular and plural variants, as well as tool-specific expressions, when they could produce distinct GitHub results. Because Python predominates in ML robustness tooling, we limited the search to Python-based repositories. The complete list of 43 search terms is provided in the replication package. For each search term, we performed both repository-level and code-level searches using the GitHub REST API. Repository searches were restricted to Python repositories and sorted by star count in descending order. We retained up to the 10 most-starred repositories returned for each term. For code searches, we retained up to the first 10 results under GitHub’s default relevance ranking. During manual screening, when a repository did not provide a description, we used the first paragraph of its README as a fallback. The 10-result threshold bounded the manual screening effort and was consistent with the general search scope adopted by Mim et al. [51]. 4.2

Repository Mining

The second part of the data collection process is to identify and extract data from repositories providing evidence of open-source ML robustness tool development. Repository Search. The repository- and code-search outputs were combined using the GitHub repository identifier. Repositories returned by multiple keywords or by both search endpoints were deduplicated before screening. Repositories already included in the 11-tool Seed Toolset were also removed from the complementary search results so that the candidate count represented only potentially new tools. This process yielded 334 unique candidate repositories for manual screening. A candidate repository was retained when it satisfied all of the following criteria: (1) it was publicly accessible on GitHub; (2) it provided reusable software rather than only experimental scripts or supplementary material; (3) its stated purpose included adversarial testing, perturbation analysis, robustness evaluation, stress testing, or another direct form of ML robustness assessment; (4) its documentation was sufficient to identify the supported functionality; and (5) it was distributed under an Open Source Initiative-approved license. Repositories that did not meet all criteria were excluded prior to metric extraction. Data Extraction. For each of the candidate repositories, we used the GitHub REST API1 to automatically collect the following data: repository name, description, creation date, last update, and URL; owner metadata including username and affiliation; project attributes including primary language, license, and topics; and activity metrics including commits, contributors, stars, forks, watchers, issues, pull requests, and timestamps of first and last commits. Table 1 maps the collected metrics to the research questions they address. Table 1. Mapping of collected metrics to research questions. Metrics Release year, target system, attack/defense categories, associated publications Stars, watchers, forks, pull requests, project origin Commits, contributors, issues, PRs, commit gaps, owner activity First commit date, last commit date, elapsed time since last commit

1 https://docs.github.com/en/rest?apiVersion=2022-11-28

RQ RQ1 RQ2 RQ3 RQ4

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study 4.3

9

Manual Revision and Classification

We manually reviewed the mined repositories to remove false positives and classify the retained tools for the analyses in RQ1-RQ4. False Positive Removal. After obtaining the list of 334 candidate repositories, we manually inspected each one. The first author manually inspected the repository description, README, license, topics, file structure, and linked publication (when available) for each of the 334 candidates. A repository was excluded when it represented any of the following: a tutorial or course assignment; a paper-specific reproduction package without a reusable software interface; a collection of isolated attack or defense scripts; model checkpoints or datasets without a tool implementation; a fork, or duplicate of another candidate; software unrelated to ML robustness despite matching a search term; or a repository without an eligible open-source license. Repositories with ambiguous scope were retained temporarily and reviewed by the second author against the same inclusion and exclusion criteria. The authors resolved uncertain cases through discussion using repository documentation and linked publications as supporting evidence. Of the 334 unique candidates, 317 were excluded, and 17 were retained as previously unidentified robustness tools, which we call the Extended Toolset. Combining these 17 tools with the 11 Seed Toolset repositories yielded the final dataset of 28 open-source ML robustness tools. Tool Classification (RQ1). For each of the 28 tools, we recorded release year, target-system type, supported attack and defense categories, and associated publications based on repository documentation and related papers. We treated project origin and research-publication availability as separate characteristics. Project origin was classified as academic, industry, or individual using repository ownership, contributor affiliations, project documentation, and associated publications. Publication availability was determined independently by examining repository README files, citation sections, project documentation, and scholarly search results. A tool was classified as having an associated research paper only when an identifiable conference paper, journal article, workshop paper, technical report, or arXiv paper introduced, described, or evaluated the tool. This procedure follows the approach used by Mim et al. [51]. These functional attributes were also used to contextualize the engagement and maintenance findings in RQ2 and RQ3. In particular, we distinguished tools by target-system type, supported attack and defense categories, and breadth of lifecycle coverage. We did not treat these attributes as predictors in the statistical analysis because several functional categories contained too few repositories for reliable subgroup testing. Instead, we used them descriptively to identify cases in which differences in repository activity could plausibly reflect differences in tool scope or intended use. The first author performed the initial classification of each tool’s origin as academic, industry, or individual using repository ownership, contributor affiliations, project documentation, and associated publications. The second author then reviewed each classification against the same evidence and marked cases that required reconsideration. The authors resolved these cases through discussion until consensus was reached. Community Engagement Analysis (RQ2). To address RQ2, we examined observable GitHub engagement across the 28 robustness-tool repositories using socialinterest indicators and participation indicators. Stars and watchers were used to capture public attention to a repository, forks were used to capture interest in inspecting or extending a repository, and pull requests were used to capture proposed contributions to the project. These measures are commonly used in repository-mining studies as proxies for public interest and visible collaboration [13, 51, 82]. We do not treat them as direct evidence of practical adoption or industrial use, because repository-level engagement may differ from package downloads, downstream dependencies,

10

Owotogbe et al.

citations, production deployment, or use within private organizations. Pull requests were used as the primary indicator of visible contribution because they more directly represent proposed changes than stars or watchers [8]. Stars and watchers were analyzed as indicators of public attention [11, 26], while forks were treated as indicators of interest in modifying or extending a repository [91]. Because older repositories have had more time to accumulate stars, watchers, and forks, we interpreted cumulative engagement counts as descriptive repository totals rather than age-adjusted measures of adoption. We therefore reported cumulative totals separately from the temporal analyses and avoided using them alone to compare repositories released in substantially different years. To provide temporal context, we reconstructed annual star acquisition from the timestamps of individual GitHub starring events, assigning each star to the calendar year in which it occurred. We also analyzed annual pull-request activity from 2019 through 2025. Each pull request was assigned to the year in which it was created and was then classified according to its GitHub state at the data collection date: open, closed without merge, or merged. As a result, the annual pull-request trends show the final observed states of pull requests created in each year, not the number of pull requests closed or merged during that same year. This distinction is important because a pull request may be created in one year and closed or merged in a later year. For the pull-request trend visualization, we retained only repository-year observations with at least 10 pull requests across the open, closed, and merged categories. This threshold reduced visual noise from repository-years with very sparse pull-request activity and retained observations from 18 of the 28 repositories. Repository-year observations below the threshold were excluded only from the visualization; they were not treated as evidence of no community engagement. Finally, to contextualize cumulative engagement by repository age, we grouped repositories descriptively by maturity at the data collection date. Project age was measured from the first commit to January 21, 2026. Repositories were classified as emerging when they were less than two years old, established when they were between two and five years old, and long-running when they were more than five years old. These maturity groups were created for descriptive interpretation in this study and were not used as inferential categories, because some groups contained too few repositories for reliable subgroup testing.

Tool Maintenance Analysis (RQ3). RQ3 examines repository maintenance activity and compares repositories by their maintenance status. To keep the maintenance and lifespan analyses consistent, RQ3 and RQ4 use the same deterministic repository-status definition. A repository marked as archived through GitHub’s archival mechanism was classified as archived. For non-archived repositories, we used a six-month commit-recency rule. A repository was classified as active if its default branch contained at least one commit during the 180 days preceding the data collection date of January 21, 2026. A non-archived repository without a default-branch commit during this period was classified as inactive. This status definition captures recent observable development activity, not definitive project abandonment. Maintainer statements and repository activity during the preceding two years were inspected as supporting context, but they did not override the commit-based rule. A repository classified as inactive may still provide stable and usable software, particularly when its intended functionality is mature and requires few code changes. We therefore use the term inactive to indicate the absence of recent default-branch commits, and reserve the term abandoned for repositories with explicit maintainer statements or archival evidence. Applying this procedure yielded five active repositories, 22 inactive repositories, and one archived repository. The status assignment was generated from the GitHub archived flag and the most recent default-branch commit date relative to the data collection date. The first author executed the classification procedure, and the second author verified the

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

11

repository identifiers, archived flags, commit dates, and resulting labels. Any discrepancies in the extracted evidence were resolved by rechecking the repository and rerunning the corresponding collection step. We adopted the 12 repository-level maintenance features used in prior repository-maintenance studies [21, 51]. The features capture four complementary dimensions of maintenance activity: repository participation (forks and contributors); issue and pull-request handling (total issues, closed issues, open pull requests, closed pull requests, and merged pull requests); development continuity (total commits, maximum days without a commit, and commits by the most active developer); and owner-level activity (owner projects and owner commits). We retained the complete feature set from prior work to support comparability and to avoid selecting features after observing the outcomes [51]. We examined each feature independently using descriptive statistics and two-sided Mann-Whitney U tests. We also report repository-level distributions and group medians because the small and unequal groups make the results sensitive to individual repositories. Feature extraction covered the 24 months preceding January 21, 2026 and used fixed three-month temporal windows [21, 51]. We compared the distributions of the 12 repository-level maintenance features between the five active and 22 inactive repositories using two-sided Mann-Whitney U tests. This non-parametric test was selected because the repository metrics were highly skewed and the two groups were unequal in size. The archived repository was excluded from the comparison. Following Mim et al. [51], we used (𝑝 ≤ 0.05) as the significance threshold. Because the analysis is observational and the active group contains only five repositories, we interpret statistically significant results as differences in observed repository activity, not as evidence that a feature caused a repository to remain active or become inactive. For repositories with fewer than two commits during the 24-month observation window, the maximum commitgap feature could not be estimated directly. In the extracted dataset, such repositories received a value of zero. This value therefore indicates insufficient commit observations rather than uninterrupted development and is interpreted cautiously. Separately, we also examined maintenance-related issues and pull requests using nine keyword expressions adapted from prior maintainability analysis of open-source tool repositories [51? ]. The expressions were selected to capture recurring compatibility, interface, and software-evolution concerns: API, endpoint, changelog, deprecation, refactor, integration, backward compatibility, library update, and API update. Seven of the nine expressions returned at least one matched issue or pull request in the final dataset; library update and API update returned no matched records. For each expression, we recorded the numbers of matched issues and pull requests. For matched records that were closed, resolution time was calculated as the number of days between creation and closure, while open records were excluded from the resolution-time calculation. The keyword categories were not mutually exclusive: a single issue or pull request could match more than one expression and was counted in each matching category. Consequently, category frequencies should not be summed to obtain the number of unique maintenance records. We report the number of matched issues and pull requests, the number of closed records used in the resolution-time calculation, and the mean and median resolution times for each expression. To assess the precision of the keyword-matching procedure, we drew a stratified random sample of 120 matched records across keyword and record-type strata. Each sampled issue or pull request was manually coded as relevant, not relevant, or unclear based on whether the title and linked record substantively concerned software maintenance or evolution rather than merely containing the matched keyword.

12

Owotogbe et al. Table 2. Variables used in the primary Kaplan-Meier survival analysis. Variable

Operational definition

First commit Final commit collection date Active Inactive Archived repository Event indicator Event duration Censored duration

Earliest commit on the repository’s default branch. Most recent commit on or before January 21, 2026. January 21, 2026. Non-archived repository with a commit during the preceding 180 days. Non-archived repository with no commit during the preceding 180 days. Excluded from the primary Kaplan-Meier analysis. 1 for non-archived inactive repositories; 0 for active repositories. Days from the first commit to 180 days after the final commit. Days from the first commit to the collection date.

Project Lifespan Analysis (RQ4). To address RQ4, we retrieved the first commit and the most recent commit on or before January 21, 2026, from the default branch of each repository. The first commit defined the beginning of the observed project lifespan. We selected the 180-day threshold to operationalize the six-month activity window used in prior repository-maintenance studies [21, 51]. Expressing the window as a fixed number of days makes the classification reproducible for all repositories relative to the common collection date. We used the same status definition as in RQ3. A non-archived repository was classified as active when its default branch contained at least one commit during the 180 days preceding January 21, 2026. Active repositories were treated as right-censored because their eventual inactivity times were not observed. For inactive repositories, the event date was defined as 180 days after the final observed default-branch commit. The archived repository, PromptBench, was excluded from the primary Kaplan-Meier survival analysis because archival is an explicit repository state rather than the same commit-recency event used for non-archived repositories. The primary Kaplan-Meier analysis therefore included 27 non-archived repositories: 22 observed inactivity events and five right-censored active repositories. For inactive repositories, survival duration was calculated as the number of days between the first commit and the 180-day inactivity event. For active repositories, survival duration was calculated as the number of days between the first commit and the collection date. Durations were converted from days to years using 365.25 days per year. We estimated the survivor function using the Kaplan-Meier estimator implemented with the KaplanMeierFitter class in the Python lifelines package. Durations were converted from days to years using 365.25 days per year. The primary analysis included 22 observed inactivity events and five right-censored active repositories. We report the estimated survival probabilities at yearly project ages, the median survival time, 95% confidence intervals, and the number of repositories at risk over time. 4.4

Replication Package

To support validation and reproducibility, we provide a replication package online 2 . The package includes the curated repository dataset, seed-tool information, GitHub search queries, data-collection scripts, processed analysis tables, statistical analysis scripts, generated figures, and supporting documentation. It also includes intermediate files used to classify repository origin, functional scope, maintenance status, and maintenance-related keyword records. 5

Findings

In this section, we present our findings for each research question. 2 https://figshare.com/s/522d8686deb621b4f583

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study 6

Deep Learning Model (46.4%)

5

Machine Learning Model (17.9%) Target System Type

Number of tools

13

4 3 2

Natural Language Processing Model (14.3%) Large Language Model (10.7%) Vision-Language Model (3.6%) Federated Learning Model (3.6%)

1

Graph Machine Learning Model (3.6%) 25 20

24

23

20

Year (a) Robustness tools released per year

20

22 20

21 20

20 20

19 20

18 20

17 20

20

16

0 0

10

20 30 40 50 Percentage (%) (b) Robustness tools by target system type

Fig. 2. Overview of robustness tools across time and target system types.

5.1

ML Robustness Tool Dataset (RQ1)

We identified 28 open-source ML robustness tools. Table 3 reports key GitHub repository characteristics for each tool, including GitHub metadata, project origin (academic, industry, or individual), targeted ML lifecycle phases, and number of attacks and defenses supported. The tools vary widely in scale, maturity, and adoption: a small number exhibit strong community interest and sustained development, while many others show limited contributor involvement and modest commit histories. Although most tools originate from academia, industry-backed projects more often exhibit recent development activity. Tools originating from the Seed Toolset are denoted with an asterisk (*); all remaining tools were identified through our automated search and belong to the Extended Toolset. We found that 78.57% of the tools originated from academia, 17.85% from industry, and 3.57% from individual contributors. Most tools focus on model training and evaluation, provide limited support for data preprocessing, and are exclusively concerned with adversarial robustness (e.g., evasion and poisoning attacks). We also examined whether each tool had an identifiable associated research publication. Of the 28 tools, 20 (71.4%) had an associated research paper, whereas no associated paper was found for eight tools (28.6%). Figure 2(a) shows the number of robustness tools released between 2016 and 2025. Releases increased from 2016 and reached their highest point in 2019, followed by lower but continued releases through 2025. Figure 2(b) reports tool distribution by target software system. Most tools focus on deep learning models (46.4%), followed by traditional machine learning models (17.9%). Support for natural language processing models (14.3%) and large language models (10.7%) is more limited, while federated learning, graph learning, and multimodal vision-language models are represented by one tool each (3.6%). Figure 3 summarizes this distribution. These results form the foundation for our analyses addressing the remaining research questions. Figures 4(a) and 4(b) show that robustness tools overwhelmingly focus on evasion attacks (89.1%), with limited support for poisoning, backdoor, and model extraction attacks. Similarly, training-based defenses (41.1%) and preprocessing defenses (33.0%) dominate, while detection, post-processing, and transformation defenses are sparsely supported.

14

Owotogbe et al.

Table 3. Robustness tools under study: S (stars), W (watchers), F (forks), C (total commits), CN (contributors), FC (first commit year), LC (last commit year), PO (project origin), MP (supported ML phases), NA (number of attacks), and ND (number of defenses). Tool Name Adversarial Robustness Toolbox (ART)* [80] Counterfit* [50] OpenAttack* [78]

S W 5784 95

F 1283

C CN 13057 109

FC 2017

LC 2025

PO Industry

904 764

29 16

148 130

64 686

7 12

2021 2020

2025 2022

Industry Academic

TextAttack* [60]

3348

33

435

2772

56

2019

2025

Academic

DeepRobust [29]

1076

12

191

858

17

2019

2025

Academic

DEEPSEC* [45]

226

13

72

5

1

2019

2019

Academic

AdverTorch* [10]

1363

24

201

309

16

2018

2022

Industry

AdvBox* [34]

1409

50

268

378

12

2018

2022

Industry

Foolbox* [9] CleverHans* [18]

2936 6402

41 185

435 1400

1711 3203

32 108

2017 2016

2024 2023

Academic Academic

Robustness Gym* [33] RobustVLM* [67]

445 151

15 3

36 7

48 24

6 2

2020 2024

2021 2025

Academic Academic

ByzFL [47]

31

2

4

287

3

2023

2025

Academic

ARES 2.0 [76]

523

12

94

190

6

2019

2023

Academic

Graph Robustness Benchmark (GRB) [77] SecML Malware [27] Torchattacks [43] AutoAttack [25] PromptBench [49] Canary SEFI [71]

98

6

18

161

3

2021

2023

Academic

242 2135 733 2771 118

7 17 8 18 26

56 367 118 219 12

217 644 176 259 493

6 16 15 14 9

2019 2019 2020 2023 2022

2025 2023 2023 2024 2024

Academic Academic Academic Industry Academic

TextFlint [75] ASTRA-RL [73]

652 25

17 6

95 1

257 144

17 7

2021 2025

2022 2025

Academic Academic

Paddle-AdversarialToolbox (PAT) [3] AI Robustness Testing Kit (AiR-TK) [40] AI Robustness Platform [85] SecML-Torch (SecMLT)[59] URET [31] LLMart [22]

17

1

1

12

1

2021

2021

Individual

5

0

0

92

4

2024

2024

Academic

4

1

0

90

1

2025

2025

Academic

102

3

15

503

10

2023

2025

Academic

32 44

4 1

10 8

22 22

2 4

2022 2024

2024 2025

Industry Industry

MP Training, Evaluation Evaluation Training, Evaluation Data Preprocessing, Training, Evaluation Training, Evaluation Training, Evaluation Training, Evaluation Training, Evaluation Evaluation Training, Evaluation Evaluation Training, Evaluation Training, Evaluation Training, Evaluation Training, Evaluation Evaluation Evaluation Evaluation Evaluation Training, Evaluation Evaluation Training, Evaluation Evaluation

NA 74

ND 37

82 15

0 0

16

0

19

12

16

13

30

7

12

5

55 18

0 1

19 8

0 4

8

4

19

1

12

5

15 34 4 7 22

0 0 0 0 8

16 3

0 0

4

0

Training, Evaluation Training, Evaluation Training, Evaluation Evaluation Evaluation

31

11

12

8

5

0

23 6

0 0

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

15

28.6% Research Paper No Research Paper

71.4%

Fig. 3. Availability of associated research papers among the 28 open-source ML robustness tools. Twenty tools had an identifiable associated research paper, while no associated paper was found for eight tools.

Evasion Attacks (89.1%)

Inference Attacks (Privacy) (3.5%) Backdoor Attacks (1.8%)

Preprocessor (33.0%)

Defense Type

Attack Type

Poisoning Attacks (4.4%)

Trainer (41.1%)

Detector (9.8%) Postprocessor (8.0%)

Model Extraction Attacks (1.3%) 0

20

40 60 Percentage (%) (a) Supported attack types

Transformer (8.0%) 80

0

10

20 30 Percentage (%) (b) Supported defense types

40

Fig. 4. Supported attack types (a) and defense types (b) across robustness tools.

Summary of RQ1: We identified 28 robustness tools released between 2016 and 2025. Most are academic, Pythonbased, and target deep learning models, with fewer tools supporting traditional machine learning, federated learning, graph learning, and multimodal vision–language models. 5.2

Tool Community Engagement (RQ2)

To examine community engagement with open-source robustness tools, we consider both social and project-level indicators, including stars, watchers, forks, and pull requests [51, 56].

16

Owotogbe et al.

5.2.1 Social Engagement. The social engagement data spans 2019 to 2025 and captures changes in repository popularity and community interest over multiple years. As shown in Table 4, several projects exhibit high star counts throughout the observation period, indicating sustained visibility and long-term engagement within the community. For example, Adversarial Robustness Toolbox (ART) maintains high levels of attention across all years, with star counts increasing steadily from 614 in 2019 to 837 in 2025. Similarly, CleverHans demonstrates persistent community interest, beginning with a high number of stars in 2019 (972) and continuing to be widely adopted through 2025. These patterns suggest that well-established robustness tools with broad applicability continue to attract users over extended periods. Table 4. GitHub star counts over time for robustness tools (2019-2025).

Tool Paddle-Adversarial-Toolbox (PAT) Counterfit AdverTorch DeepRobust Torchattacks Universal Robustness Evaluation Toolkit (URET) LLMart AI Robustness Testing Kit (AiR-TK) ByzFL Canary SEFI TextAttack Graph Robustness Benchmark (GRB) Adversarial Robustness Toolbox (ART) AdvBox Foolbox RobustVLM CleverHans AutoAttack PromptBench SecML-Torch (SecMLT) SecML Malware Robustness Gym DEEPSEC ASTRA-RL TextFlint ARES 2.0 OpenAttack AI Robustness Platform

2019 0 0 392 7 28 0 0 0 0 0 0 0 614 305 501 0 972 0 0 0 0 0 80 0 0 0 0 0

2020 0 0 309 314 202 0 0 0 0 0 1093 0 728 338 454 0 561 127 0 0 19 0 51 0 0 97 146 0

2021 13 454 203 234 437 0 0 0 0 0 622 43 769 185 367 0 540 184 0 0 68 349 26 0 503 162 232 0

2022 3 97 147 177 443 3 0 0 0 0 417 23 702 91 278 0 331 144 0 0 54 58 20 0 69 102 115 0

2023 1 143 121 159 424 19 0 0 0 13 449 20 843 91 233 0 360 108 1386 2 31 22 21 0 43 67 110 0

2024 0 122 93 104 378 8 0 5 0 83 424 5 761 47 226 108 288 97 1063 30 35 7 9 0 20 60 90 0

2025 0 83 57 80 220 2 42 0 31 23 332 7 837 28 151 41 221 74 319 69 35 8 18 25 13 33 70 4

In contrast, other projects exhibit early peaks followed by more gradual growth or stagnation. For instance, projects such as AdverTorch, AdvBox, and Foolbox show high star counts in earlier years that have since leveled off, suggesting an initial surge in adoption followed by stabilization. We also observe repositories that gained substantial traction only in recent years. Notably, projects such as PromptBench exhibit sharp increases in star counts beginning in 2023, reaching 1,386 stars in that year. This pattern suggests rapid adoption driven by emerging robustness concerns, particularly in response to newer AI model paradigms, e.g., large language models (LLMs). Likewise, LLMart shows star activity

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

17

175

Watcher count

150 125 100 75 50 25 0

Academic

Industry

Project origin

Individual

Fig. 5. Watcher count by repository source among the 28 open-source ML robustness tools.

only in the final year of observation, indicating a newly introduced tool that has begun to attract community attention. Finally, DeepRobust and TextAttack show consistent star counts after initial growth phases, reflecting sustained but stable interest rather than rapid expansion. We observed variation in social engagement across robustness tools (Figure 5), with watcher counts differing across repository sources. For example, Adversarial Robustness Toolbox (ART) and Counterfit attract relatively large audiences among industry-backed tools, while CleverHans has the highest watcher count in the dataset. These findings suggest that both industry-backed and research-focused robustness tools can attract visible public attention. These tools support robustness evaluation across multiple stages of the ML lifecycle and a wide range of attack scenarios. In particular, ART is also institutionally supported beyond its GitHub repository. It is hosted by the Linux Foundation AI & Data Foundation (LF AI & Data) as a graduated-stage project, after being contributed by IBM as an incubation-stage project in June 2020 and graduating in February 2022 [44]. This external governance and foundation support provide additional context for ART’s visibility and sustained engagement compared with many other robustnesstool repositories. At the same time, several academic repositories demonstrate substantial social engagement. For example, CleverHans has the most watchers in our dataset (185), while repositories such as AdvBox (50), Foolbox (41), and TextAttack (33) also attract consistent attention. These findings suggest that research-focused robustness tools can foster dedicated communities, indicating strong interest in methodological innovation and experimental robustness techniques within the academic community. 5.2.2 Project Engagement. Project-level engagement varies substantially across robustness tools, as reflected in pull request activity between 2019 and 2025 (Table 6). Overall, we observed that sustained PR activity is concentrated in a relatively small subset of robustness tools. Projects backed by established organizations or long-running research efforts tend to exhibit the highest and most consistent engagement. For example, Adversarial Robustness Toolbox shows

18

Owotogbe et al.

continuous PR activity from 2019 through 2024, with opened PRs increasing from 16 in 2019 to a peak of 59 in 2023, accompanied by very high numbers of closed PRs, including 285 in 2020 and 296 in 2021. This pattern reflects continuous development and active involvement from maintainers. Notably, while merged PRs are absent in the extracted ART records, the high number of closed PRs indicates that pull requests were actively reviewed or resolved in GitHub, even when they were not recorded as merged in the extracted metadata. Other robustness tools demonstrate moderate PR engagement across multiple years. For instance, TextAttack shows consistent activity between 2019 and 2024, with opened PRs peaking at 30 in 2021 and closed PRs reaching 237 in 2020. The sustained presence of closed PRs across several years reflects active review and integration processes during periods of high community use. Similarly, Foolbox and CleverHans exhibit repeated PR activity in earlier years, followed by declining PR activity after 2021. This pattern suggests that these repositories had periods of earlier community or maintainer engagement, but show reduced visible pull-request activity in the later years of the analysis window. In contrast, many robustness tools originating from smaller academic groups or individual contributors display sporadic or limited PR activity. Tools such as DeepRobust, AutoAttack, and TextFlint show that PR engagement is confined to one or two years, with few opened and closed PRs. This pattern suggests niche usage or maintenance driven primarily by the original developers rather than sustained community contribution. We also observe emerging engagement in a small number of repositories in later years. For example, PromptBench shows PR activity only beginning in 2024, with 2 open and 10 closed PRs, indicating the early stages of community interaction or maintainer-driven development. Similarly, SecMLTorch exhibits its first notable PR engagement in 2023–2024, suggesting growing adoption or renewed maintenance efforts. Across repository tools, merged PRs are rare compared to opened and closed PRs. When present, they tend to occur in repositories with higher overall PR volumes, such as AdvBox and Foolbox, suggesting selective integration practices. Across the dataset, merged PR counts are low, with most repositories reporting few or no merged PRs in many years. This observation suggests that, while issues and contributions are actively discussed and resolved, formal merge activity may be tightly controlled or underreported in GitHub metadata. Overall, the PR analysis highlights substantial heterogeneity in engagement patterns and underscores that sustained maintenance of robustness tools is limited to a few projects. Summary of RQ2: Community engagement is uneven across the 28 robustness-tool repositories. Stars, forks, watchers, and pull-request activity are concentrated in a small subset of tools, especially ART, CleverHans, Foolbox, TextAttack, AdvBox, and PromptBench. Several industry-backed or long-running repositories show higher visible engagement, but repository origin and age do not fully explain the variation. Tools with limited visible participation may face greater sustainability uncertainty because fewer observable contributors appear to be involved in issue discussion, pull-request activity, or repository monitoring. 5.3

Tool Maintenance (RQ3)

To assess repository maintenance, we applied the deterministic status definition described in Section 4. Of the 28 repositories, five (17.9%) were classified as active, 22 (78.6%) as inactive, and one (3.6%) as archived. The inactive classification indicates that no commit was recorded on the repository’s default branch during the 180 days preceding January 21, 2026. It does not necessarily imply that the software is unusable or permanently abandoned. The five active repositories were the Adversarial Robustness Toolbox (ART), SecML-Malware, ASTRA-RL, SecML-Torch, and LLMart. Maintenance activity varied considerably even within this group. ART exhibited substantially greater issue,

Mean issue-resolution time (days)

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

175 150 125 100 75 50 25 0

Active

19

Inactive Repository status

Fig. 6. Mean issue-resolution time by repository maintenance status among the 28 open-source ML robustness tools.

pull-request, and contributor activity than the other active repositories, whereas ASTRA-RL and LLMart were more recently established and had shorter project histories. The results should therefore be interpreted in relation to repository age, functional scope, and the number of observations available for each project. Descriptively, active repositories exhibited greater maintenance activity than inactive repositories across several measures. Active repositories had a median of 22 total issues, compared with 2.5 for inactive repositories, and a median of 16 closed issues, compared with 0. They also had medians of 55 closed pull requests and 38 merged pull requests, whereas both corresponding medians were 0 among inactive repositories. During the 24-month analysis window, active repositories had a median of 144 commits, 67 commits by the most active developer, and seven contributors. The corresponding medians for inactive repositories were 0. These descriptive differences indicate that observable development and collaboration were concentrated in a small subset of the repositories. Using the conventional unadjusted significance threshold of 𝑝 ≤ 0.05, nine of the 12 maintenance features differed between active and inactive repositories. Here, (p) denotes the p-value from the two-sided Mann-Whitney U test: smaller values indicate stronger evidence that the two status groups differ in the distribution of a feature. The nine features with unadjusted 𝑝 ≤ 0.05 were total issues (𝑈 = 93.0, 𝑝 = 0.0184), closed issues (𝑈 = 101.0, 𝑝 = 0.0022), open pull requests (𝑈 = 89.0, 𝑝 = 0.0210), closed pull requests (𝑈 = 105.0, 𝑝 = 0.0012), merged pull requests (𝑈 = 106.0, 𝑝 = 0.0005), total commits (𝑈 = 99.0, 𝑝 = 0.0045), maximum days without a commit (𝑈 = 89.0, 𝑝 = 0.0266), commits by the most active developer (𝑈 = 97.0, 𝑝 = 0.0067), and number of contributors (𝑈 = 102.5, 𝑝 = 0.0021). Forks, owner projects, and owner commits did not differ significantly. After applying the Holm correction for 12 comparisons, six features remained significant: closed issues, closed pull requests, merged pull requests, total commits, commits by the most active developer, and number of contributors. Given the small active group, these comparisons are interpreted as exploratory. Because the active group contained only five repositories, we also examined the repository-level values underlying each comparison rather than relying solely on the 𝑝-values. Several active-group means were strongly influenced by ART, which had substantially larger issue, pull-request, and contributor counts than the other active repositories. We therefore emphasize group medians and repository-level distributions when interpreting the results. We did not compute model-based feature importance because repository status was not predicted from the 12 maintenance features. Instead, status was assigned directly using the GitHub archived flag and the 180-day commit-recency rule. The 12 features were then used only to compare the observed maintenance profiles of active and inactive repositories.

20

Owotogbe et al.

Table 5. Maintenance-related issue and pull-request matches and resolution times. Resolution times are calculated only for matched records created and closed on or before January 21, 2026. Keyword categories overlap, and one record may appear in multiple rows. Keyword API Endpoint Changelog Deprecation Refactor Integration Backward compatibility Library update API update

Issues

PRs

Closed records

Mean days

Median days

85 1 2 20 8 7 1 -

235 9 528 84 69 50 15 -

294 7 509 95 75 56 15 -

54.41 47.86 24.89 22.12 22.67 34.96 50.56 -

4.87 15.08 1.45 2.43 1.15 3.61 32.75 -

Mean issue-resolution times varied considerably across repositories (Figure 6). Four of the five active repositories had mean issue-resolution times below 52 days, while ART was a clear exception, with a mean of 173.40 days across 903 closed issues. Among the other active repositories, SecML-Malware had a mean of 20.97 days, SecML-Torch 41.99 days, LLMart 51.54 days, and ASTRA-RL 0.04 days. The ASTRA-RL value is based on only one closed issue and should therefore not be treated as representative of its general maintenance responsiveness. Inactive repositories also exhibited substantial variation. Foolbox and OpenAttack had mean resolution times of 149.35 and 122.47 days, respectively, while TextAttack and CleverHans had means of 80.52 and 80.16 days. These repository-level values should be interpreted alongside the number of closed issues, because estimates based on few observations are less stable. The variation within both status groups indicates that issue-resolution time alone does not clearly distinguish active from inactive repositories. We next examined maintenance-related issues and pull requests using the nine keyword expressions defined in the methodology. Across the final 28 repositories, changelog produced the largest number of matches (𝑛 = 530), followed by API (𝑛 = 320), deprecation (𝑛 = 104), refactor (𝑛 = 77), and integration (𝑛 = 57). The expressions backward compatibility and endpoint matched 16 and 10 records, respectively. No eligible record matched the exact expressions library update or API update under the revised word-boundary matching procedure. Most matches occurred in pull requests rather than issues. For example, changelog appeared in 528 pull requests and two issues, while API appeared in 235 pull requests and 85 issues. Similarly, refactor appeared in 69 pull requests and eight issues, and integration appeared in 50 pull requests and seven issues. These counts indicate that the selected maintenance concerns were frequently associated with proposed code changes. However, they do not establish that each match represents a distinct maintenance task, because individual records could match more than one keyword. The manual validation sample indicated high precision. Of the 120 sampled records, 118 were judged relevant, two were judged not relevant, and none were coded unclear, giving an estimated precision of 98.33%. Precision for the api keyword was 93.55%; all other sampled keyword groups had 100% precision, although several groups had small validation samples. Table 5 shows substantial variation in the resolution times of keyword-matched maintenance records. Changelog, refactor, and deprecation matches had median resolution times between 1.15 and 2.43 days, indicating that many records containing these terms were closed quickly. API matches had a mean resolution time of 54.41 days but a median of only 4.87 days, indicating a strongly right-skewed distribution in which a smaller number of long-running records increased the mean. Backward-compatibility matches had a mean of 50.56 days and the largest median, at 32.75 days, although

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

21

this estimate is based on only 15 closed records. Endpoint matches had a mean of 47.86 days and a median of 15.08 days, while integration matches had a mean of 34.96 days and a median of 3.61 days. These comparisons are descriptive. The keyword categories overlap, sample sizes vary substantially, and several categories contain few or no records. The results therefore characterize which maintenance-related topics appeared in issues and pull requests and how long closed records remained open. They do not show how much developer time was spent on each record, how difficult the underlying code change was, or whether one maintenance topic required more work than another. Table 6. Yearly open, closed, and merged pull request activity for open-source robustness tools (2019-2025). Tool Counterfit AdverTorch DeepRobust Torchattacks AI Robustness Testing Kit (AiR-TK) TextAttack Adversarial Robustness Toolbox (ART) AdvBox Foolbox CleverHans AutoAttack PromptBench SecML-Torch (SecMLT) Robustness Gym TextFlint OpenAttack

Open PRs 2022 2023 10 0 0 0 3 0 2 5 0 0

2019 0 3 0 0 0

2020 0 2 3 0 0

2021 8 0 3 4 0

2 16

25 38

30 44

12 49

2 5 32 0 0 0 0 0 0

10 9 7 0 0 0 0 0 24

0 6 2 1 0 0 4 2 10

10 7 0 0 0 0 0 0 0

2024 0 0 0 0 0

2025 0 0 0 0 0

2019 0 28 0 0 0

2020 0 8 17 0 0

Closed PRs 2021 2022 2023 4 20 0 0 0 0 8 7 0 7 11 12 0 0 0

2024 0 0 0 0 14

2025 0 0 0 0 0

2019 0 0 0 0 0

2020 0 4 0 0 0

Merged PRs 2021 2022 2023 0 0 0 0 0 0 0 2 0 0 0 3 0 0 0

2024 0 0 0 0 0

2025 0 0 0 0 0

5 59

1 48

0 0

45 99

237 285

85 296

48 247

19 165

8 60

0 0

0 0

0 0

0 0

0 0

2 0

3 0

0 0

0 0 0 0 0 1 0 0 0

0 1 0 0 2 4 0 0 0

0 0 0 0 0 0 0 0 0

17 71 84 0 0 0 0 0 0

3 96 8 13 0 0 0 0 98

0 8 19 14 0 0 64 10 54

0 16 0 0 0 0 0 0 0

0 0 0 0 20 12 0 0 0

0 13 0 0 10 30 0 0 0

0 0 0 0 0 0 0 0 0

0 0 3 0 0 0 0 0 0

0 1 2 0 0 0 0 0 0

0 1 4 0 0 0 0 0 0

6 1 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0

0 3 0 0 0 1 0 0 0

0 0 0 0 0 0 0 0 0

Summary of RQ3: Five of the 28 robustness tools were classified as active, 22 as inactive, and one as archived. Active repositories showed higher observable development activity across several maintenance indicators, especially pullrequest activity, commits, contributors, and issue handling. Maintenance-related records frequently concerned API changes, dependency updates, changelog updates, refactoring, integration, deprecation, and backward compatibility. 5.4

Tool Project Lifespan (RQ4)

Unlike RQ3, which examines maintenance activity during the 24 months preceding the collection date, RQ4 considers how long robustness-tool repositories remain under active commit-based development. The primary survival analysis excluded the archived repository, PromptBench, and included 27 non-archived repositories. Twenty-two inactive repositories were treated as observed inactivity events, with the event date defined as 180 days after the final observed default-branch commit. The five active repositories were right-censored at January 21, 2026. Figure 7 provides a descriptive calendar-time view of the first and final commits for the three repository-status groups. The timestamps in this figure are expressed relative to January 1, 2019 only to show when development activity occurred in calendar time. This common calendar baseline was not used to calculate project-survival duration. Survival duration was measured separately from each repository’s own first commit, as described in Section 4. The descriptive timelines show that long project histories occur in both active and inactive repositories. Repository age alone therefore does not determine current maintenance status. Several repositories with long development histories were inactive at the collection date, while some recently established repositories remained active. However, an inactive classification indicates only that no commit occurred on the default branch during the final 180 days of observation; it does not establish that the software is unusable or permanently abandoned. Figure 8 presents the Kaplan-Meier survival estimate for the 27 non-archived repositories. Project age is measured from the first commit. For inactive repositories,

Owotogbe et al.

Repository status

22

Active Archived Inactive

−1000 −500

0

500 1000 1500 2000 Days since January 1, 2019 Activity First commit Final commit

2500

Estimated probability of remaining active

Fig. 7. Calendar-time distribution of first and final commits for active, inactive, and archived robustness-tool repositories. Dates are expressed as days relative to January 1, 2019 for descriptive comparison only; these values were not used as survival durations in the Kaplan-Meier analysis.

1.0

All robustness tools

0.8 0.6 Median = 2.89 years

0.4 0.2 0.0

0

2 4 6 Project age since first commit (years)

8

Fig. 8. Kaplan-Meier estimate of time to 180-day inactivity for the 27 non-archived robustness-tool repositories. Inactive repositories are treated as observed events 180 days after their final default-branch commit, while active repositories are right-censored at the collection date.

the observed duration ends at the 180-day inactivity event. For active repositories, the duration ends at the colection date and is right-censored. The estimated median time to 180-day inactivity was 40.63 months, or approximately 3.39 years. The estimated probability of remaining active was 92.44% after one year and 30.42% after five years. These results characterize continued observable development and do not imply that inactive repositories are necessarily unusable or permanently abandoned. The estimated probability of remaining under active commit-based development was 85.7% after one year, 62.3% after two years, and 46.8% after three years. It declined to 29.8% after four years, 25.5% after five years, and 5.7% after seven years. The estimated median survival time was 2.89 years, corresponding to approximately 34.7 months. In Kaplan-Meier terms, this is the project age at which the estimated probability of remaining active first fell to 0.5 or

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

23

below. The confidence interval widens substantially at later project ages because only a small number of repositories remain at risk. The estimates beyond approximately five years should therefore be interpreted cautiously. The flat tail of the curve does not imply that repositories were directly observed for nine or ten years under continued maintenance; it reflects the absence of additional observed inactivity events after the remaining long-lived repositories had been censored or removed from the risk set. Summary of RQ4: The primary Kaplan-Meier analysis excluded the archived repository and included 27 nonarchived repositories. Under the 180-day commit-based inactivity definition, 22 repositories experienced an observed inactivity event and five active repositories were right-censored at the data collection date. The estimated median time to 180-day inactivity was 40.63 months, or approximately 3.39 years. The estimated probability of remaining active was 92.44% after one year and 30.42% after five years. These results characterize continued observable commit-based development and do not imply that inactive repositories are necessarily unusable or permanently abandoned. 6

Discussion

This section interprets the empirical findings in relation to the long-term sustainability of open-source ML robustness tools. We discuss how repository visibility relates to maintenance, why compatibility work is central to robustnesstool upkeep, why project age alone does not establish sustainability, and what these findings imply for researchers, practitioners, and AI assurance documentation. 6.1

Increasing Visibility of Robustness Tooling

The goal of our work is to support the sustainability of open-source robustness tools by examining their maintenance and community engagement histories. Prior work shows that maintenance activity and visible project signals can influence trust in, awareness of, and adoption of AI and machine learning tools [16, 51]. Our findings, however, indicate that in robustness tooling, visibility and maintenance do not always move together. Engagement is concentrated in a small subset of tools. For example, the Adversarial Robustness Toolbox (ART) exhibits high numbers of stars, watchers, forks, and pull-request activity, reflecting sustained repository visibility and participation. CleverHans also accumulated substantial historical engagement, including the largest watcher count in our dataset, but it was classified as inactive under the 180-day commit-recency criterion. This contrast illustrates that high visibility does not guarantee continued maintenance. Maturity may partly shape these engagement patterns. Older repositories have had more time to accumulate stars, forks, watchers, citations, and community attention, whereas newer projects may not yet have had enough time to become visible to the broader robustness community. As a result, cumulative engagement measures should not be interpreted as direct evidence of current maintenance or practical adoption. A mature tool may remain widely known after visible development slows, while a newer tool may be actively maintained but not yet widely recognized. Similar concentration effects have been reported for fairness tools, where organizational backing, reputation, and community awareness shape engagement patterns [51]. Together, these results suggest that visibility and awareness are relevant to the sustainability of robustness tools, but they are insufficient on their own. This aligns with prior work identifying lack of awareness as a barrier to the adoption of responsible-AI tooling [52, 61]. Our findings suggest that improving discoverability and clearly signaling ongoing support may help foster a more diverse and sustainable ML robustness tooling ecosystem.

24

Owotogbe et al.

6.2

Maintenance Effort Centers on API, Dependency, and Compatibility Work

Our analysis shows that maintenance-related records frequently concerned API changes, changelog updates, refactoring, dependency integration, deprecation, and backward compatibility. These topics indicate that robustness-tool maintenance often involves keeping the tool usable as surrounding software changes, including ML framework APIs, package dependencies, supported model interfaces, and evaluation workflows. This finding mirrors observations in fairness tooling, where sustained maintenance is closely tied to managing software evolution rather than only introducing new algorithmic features [51]. Routine maintenance records, such as changelog updates and refactoring, were typically resolved more quickly. By contrast, API-related records had the longest mean resolution time in the keyword analysis, at 54.41 days, and backwardcompatibility records also showed relatively long resolution times. This suggests that compatibility work may require more coordination than routine updates, especially when changes affect how a tool interacts with ML frameworks, dependencies, model interfaces, or evaluation workflows. This interpretation is consistent with broader findings that machine-learning systems accumulate maintenance burdens through dependencies, configuration, interfaces, and hidden technical debt [68, 88]. This maintenance burden is especially important for robustness tools because they must track changes in ML framework APIs while also supporting evolving attack, defense, and evaluation practices. For example, tools that support evasion, poisoning, inference, backdoor, or model-extraction attacks may need to remain compatible with changing model interfaces, framework versions, dataset formats, and evaluation protocols. Prior work on adversarial robustness similarly emphasizes the continuing evolution of attack and defense techniques across application sectors [58]. These findings suggest that compatibility work is central to the sustainability of robustness tooling. At the data collection date, only five of the 28 repositories were classified as active, while 22 were inactive and one was archived. This concentration of recent development activity means that users should be cautious when relying on tools whose APIs, dependencies, or supported frameworks have not been updated recently. An inactive repository may still provide useful software, but its compatibility with current ML stacks should be verified before it is used for robustness evaluation.

6.3

Longevity Does Not Imply Sustainability

Our analysis of project lifespan further demonstrates that longevity alone is not a reliable indicator of sustainability. Several robustness tools have existed for many years, but not all continue to receive recent maintenance or community engagement. Inactive repositories often exhibit meaningful development histories and extended gaps since their last commit, suggesting that historical relevance or early visibility can persist after observable maintenance slows. Conversely, some active robustness tools have shorter lifespans but show recent commit activity at the data collection date. Several factors may help explain this pattern. Some robustness tools originate as research artifacts, where development may be closely tied to a publication, benchmark, or experimental contribution. Prior studies of open-source maintenance similarly show that research-driven or low-maintenance projects can experience reduced contributor activity over time [21, 51]. In addition, some tools target narrow attack categories, defense mechanisms, or model types and may reach a relatively stable state after their intended functionality is implemented. In such cases, low recent activity may reflect either stagnation or limited need for visible change, depending on whether the tool remains compatible with current ML frameworks and evaluation practices. Similar patterns have been reported in studies of open-source project survival, which emphasize recent activity and contributor engagement over project age as stronger indicators of project health [4, 13]. Our findings reinforce

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

25

this perspective in the context of robustness tooling. Project age captures how long a repository has existed, but sustainability depends on whether the project continues to receive maintenance attention, remains compatible with the surrounding ML ecosystem, and supports current robustness evaluation needs. The survival analysis confirms this distinction. Among the non-archived repositories in our dataset, the median time to 180-day inactivity was 40.63 months, or approximately 3.39 years. This result indicates that robustness tools can remain visible or historically important while still becoming inactive within a few years of initial development. Longevity should therefore be interpreted as one part of a repository’s trajectory, not as evidence of continued sustainability on its own [4]. 6.4

Implications for Researchers and Practitioners

The findings of this study have implications for researchers who develop robustness tools, practitioners who select them for evaluation tasks, and organizations that depend on robustness evidence in AI assurance processes. First, the dataset shows that open-source ML robustness tooling is fragmented and unevenly distributed across learning paradigms, attack categories, defense categories, and target systems. Evasion attacks dominate the available attack support, whereas poisoning, inference, backdoor, and model-extraction attacks are represented by fewer tools. This suggests that future robustness-tool research should be scoped against known robustness needs rather than only against existing popular repositories. Researchers proposing new tools should clearly position them relative to existing ones, identify the concrete capability gap they address, and evaluate against established baselines or benchmarks where appropriate [24]. Second, robustness-tool development should treat sustainability as a software-engineering concern rather than only as a research-output concern. Our maintenance analysis shows that long-term robustness tooling requires continued attention to software evolution. Maintenance records in the dataset frequently involved API changes, dependency updates, integration work, refactoring, changelog updates, deprecation handling, and backward compatibility. APIrelated records had the longest mean resolution time in the keyword-based maintenance analysis, at 54.41 days. This indicates that robustness tools remain usable not only because they implement attacks or defenses, but also because they continue to work with changing ML frameworks, model interfaces, dependencies, and evaluation practices. Researchers and maintainers should therefore document supported framework versions, provide stable interfaces where possible, publish release notes, use dependency lock files or environment specifications, automate compatibility tests where feasible, and make contribution guidance visible [68, 88]. Third, practitioners should evaluate maintenance state together with functional coverage. Stars, forks, watchers, and citations can indicate visibility or historical importance, but they do not guarantee that a repository is currently maintained. Before integrating a robustness tool into an evaluation workflow, practitioners should check whether the tool supports the required attack or defense category, whether it remains compatible with the target ML framework, whether the repository is archived, and whether recent commits, issues, or pull requests indicate ongoing maintenance. These checks are consistent with prior repository-mining work emphasizing recent activity and contributor engagement as important signals of open-source project health [21, 51]. The survival analysis reinforces the need for periodic reassessment: among the non-archived repositories in our dataset, the median time to 180-day inactivity was 40.63 months, or approximately 3.39 years [4]. Thus, long-term integration should not rely on project age, star count, or publication status alone. Finally, robustness tools should be treated as versioned and evolving software dependencies in AI assurance documentation. When robustness evaluations contribute to safety, security, or compliance decisions, records should identify the exact tool artifact used, including the repository URL, commit identifier or release version, execution date, dependencies,

26

Owotogbe et al.

supported ML framework version, evaluation configuration, known limitations, and archived or maintenance status at the time of use. The maintenance-status record should also include the observation date and the rule used to assign the status, such as the 180-day commit-recency criterion used in this study. This is particularly relevant for AI Bills of Materials, where recording only a tool name is insufficient to capture maintenance and compatibility risk. Preserving execution environments, dependency specifications, container files, and configuration artifacts can improve traceability even if the upstream repository later changes or becomes inactive [54, 63]. Organizations that depend on robustness tools should also consider monitoring upstream changes and maintaining contingency plans for replacing or internally supporting critical evaluation dependencies. 7

Threats to Validity

We discuss potential threats to the validity of this study and the steps taken to mitigate them, following the classification of validity threats established by Wohlin et al. [84] and Campbell and Stanley [14]. Construct Validity. We operationalize community engagement and maintenance using repository-level metrics, including commits, issues, pull requests, stars, watchers, and forks. Although these measures are widely used in software engineering research [35, 39], they do not capture all dimensions of software use, engagement, or maintenance quality. Informal communication through Slack, Discord, mailing lists, and other external channels is not represented in GitHub metadata. Similarly, package downloads, downstream dependencies, private forks, internal deployments, and industrial use are not observable through the collected repository metrics. We therefore interpret these measures as indicators of publicly observable GitHub engagement rather than direct evidence of practical adoption. Cumulative engagement measures are also affected by repository age. Older repositories have had more time to accumulate stars, watchers, forks, issues, and pull requests than recently released projects. We examined annual star acquisition and yearly pull-request activity to provide temporal context, but we did not fully normalize all engagement measures by repository age. Consequently, comparisons between repositories with substantially different release years should be interpreted cautiously. The repositories also differ substantially in functional scope. Broad toolkits supporting multiple attack categories, model types, or ML frameworks may naturally generate more issues, pull requests, and commits than specialized tools with a narrower feature set. We recorded attack coverage, defense coverage, target system, and lifecycle phase to contextualize these differences, but the number of repositories within individual functional categories was too small for reliable subgroup comparisons. Observed differences in engagement and maintenance may therefore partly reflect variation in tool scope rather than maintenance quality alone. Commit recency is an imperfect proxy for maintenance state. Repositories with no recent commits may represent abandoned projects, mature and stable tools, or software maintained through release branches, package registries, forks, or private development channels. Our 180-day rule provides a reproducible measure of observable development activity but cannot fully distinguish these cases. Accordingly, the inactive label should be interpreted as the absence of recent commits on the default branch rather than definitive evidence that the software is unusable or abandoned. Our classification retains archived repositories as a category distinct from inactive repositories. The single archived repository, PromptBench, was excluded from the Mann-Whitney U comparison so that the statistical analysis compared only active and inactive repositories. To reduce reliance on any single indicator, we combined multiple repository metrics and supplemented automated extraction with manual inspection of all 28 repositories. Internal Validity. Our analysis relies on repository-level artifacts retrieved through the GitHub API, including commits, issues, pull requests, and social signals. These artifacts may be affected by transient API inconsistencies, delayed updates,

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

27

repository renaming, or subsequent changes to repository metadata. To reduce this risk, we used automated datacollection scripts with standardized queries, collected data on January 21, 2026, and manually inspected the extracted data for all 28 repositories. For repository-status classification, the first author applied the predefined archival and commit-recency rules, and the second author verified the repository identifiers, archived flags, first and last commit dates, and resulting labels. Any discrepancies were resolved by rechecking the GitHub evidence and rerunning the relevant extraction steps. Because the final status labels were generated by a deterministic rule rather than by subjective independent coding, we do not report an inter-rater agreement coefficient. We also applied the same analysis windows and feature definitions to all repositories. Nevertheless, residual inaccuracies may remain because of deleted historical artifacts, rewritten commit histories, undocumented repository changes, or limitations of the GitHub API External Validity. Our study focuses exclusively on publicly available open-source ML robustness tool projects on GitHub. Consequently, the findings may not generalize to robustness tools developed in closed-source settings, hosted on alternative platforms (e.g., GitLab, Bitbucket), or distributed outside public version control systems. Additionally, our focus on Python-based tools reflects the predominance of Python in the ML ecosystem but means that robustness tooling implemented in other languages is not represented. To broaden coverage, we employed a systematic keyword-based search strategy informed by prior work and iteratively refined through manual inspection [51, 56]. The GitHub discovery procedure may also favor established and highly visible repositories. For each keyword, repository-search results were restricted to Python projects, ranked by stars in descending order, and limited to the first 10 results. Code-search results were likewise limited to the first 10 results returned by GitHub. These choices bounded the manual screening effort but may have excluded recently released, specialized, or low-visibility tools appearing below the selected cut-off. Accordingly, the resulting dataset should be interpreted as a systematically curated collection of publicly discoverable robustness tools rather than an exhaustive census of the ecosystem. The keyword vocabulary was derived from the Seed Toolset and therefore reflects the terminology used by established robustness projects. Although we combined natural-language concepts with technical package and module identifiers, tools described using substantially different terminology may not have been retrieved. We did not estimate the recall of the 43-keyword set against an independent gold-standard population. Keyword sensitivity is therefore a remaining source of selection uncertainty. Furthermore, our seed set, derived from the prior comparative analysis by Agarwal and Nene [2], provides an established anchor set of recognized robustness tools that reduces selection bias toward only newly discovered repositories. Despite these efforts, our dataset likely represents a subset of robustness tooling in practice, and tools using different terminology or distribution channels may not be captured. Conclusion Validity. Our statistical conclusions are based on 28 robustness-tool repositories: five active, 22 inactive, and one archived. The Mann-Whitney U comparisons therefore involve only five active and 22 inactive repositories, with the archived repository excluded. These small and unequal group sizes limit statistical power, increase sensitivity to individual repositories, and reduce the stability of estimated group differences. We used two-sided non-parametric tests because the repository metrics were highly skewed and the groups were unequal in size. However, statistical significance should not be interpreted as evidence of a large or causal effect. We therefore report the exact 𝑈 statistics and 𝑝-values, interpret the findings as exploratory associations, and emphasize the corresponding group medians and repository-level distributions. Replication on larger and more diverse samples is needed to determine whether the observed differences generalize beyond the studied repositories. The survival estimates are also uncertain at longer project ages because few repositories remain at risk. The widening confidence intervals in the Kaplan-Meier curve reflect this loss of precision. Survival probabilities beyond approximately five years should therefore be interpreted

28

Owotogbe et al.

cautiously. Comparisons by project origin are especially uncertain because the academic, industry, and individual groups are small and unequal in size. We therefore treat origin-based differences as descriptive and do not infer that organizational origin causes differences in engagement, maintenance, or survival. 8

Conclusion

In this paper, we presented a repository-mining study of 28 open-source ML robustness tools, examining their functional scope, observable GitHub engagement, maintenance activity, and project longevity. The study adapts established repository-mining methods to robustness tooling and provides a curated dataset, statistical analyses, and a documented replication package. The results show that engagement and recent development activity are concentrated in a small subset of repositories. At the collection date of January 21, 2026, five repositories were classified as active, 22 as inactive, and one as archived under the 180-day commit-recency rule. The inactive label indicates the absence of recent default-branch commits and does not necessarily mean that a tool is unusable or permanently abandoned. Exploratory Mann-Whitney U tests identified active-inactive differences in nine of the 12 maintenance features at the unadjusted (𝑝 ≤ 0.05) threshold, with six features remaining significant after Holm correction. These differences were most evident for pull-request activity, commits, contributors, and issue handling. Maintenance-related records frequently concerned API changes, changelog updates, deprecation, refactoring, integration, and backward compatibility. APIand backward-compatibility-related records had relatively long observed resolution times, although these keyword categories overlapped and the resolution-time distributions were strongly skewed. These findings identify recurring maintenance concerns but do not establish that such tasks are more complex or burdensome than maintenance work in other software domains. The Kaplan-Meier analysis was conducted on the 27 non-archived repositories, with 22 observed inactivity events and five right-censored active repositories. The estimated median time to 180-day inactivity was 40.63 months, or approximately 3.39 years. The estimated probability of remaining active declined from 92.44% after one year to 30.42% after five years. Historical visibility and project age should therefore not be used alone to infer continued maintenance. For software-engineering practice, robustness tools should be managed as evolving dependencies rather than as one-time evaluation artifacts. Teams should record the exact version or commit used, preserve execution environments, monitor compatibility and maintenance signals, and prepare for the possibility that an upstream repository becomes inactive. For AI Bills of Materials, the tool version, repository identifier, observation date, maintenance state, supported framework versions, dependency environment, and evaluation configuration should be preserved to support later traceability and reproducibility. Future work should replicate the study on larger and more diverse tool ecosystems, examine package downloads and downstream dependencies, and conduct interviews with maintainers and users. Longitudinal follow-up would help determine how repository status, contributor participation, compatibility work, and project survival change over time. Future studies could also measure practical adoption by identifying downstream repositories that import robustness-tool packages or invoke their APIs, classifying the purposes of those projects, and examining which robustness capabilities are used in practice. 9

Acknowledgments

References [1] Ayman M Abdalla, Mohammad M Abdallah, and Mosa I Salah. 2017. A BRief Program Robustness Survey. International Journal of Software Engineering & Applications (IJSEA) 8, 1 (2017), 1–10. doi:10.5121/ijsea.2017.8101

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

29

[2] Avinash Agarwal and Manisha J. Nene. 2025. Advancing Trustworthy AI: A Comparative Evaluation of AI Robustness Toolboxes. SN Computer Science 6, 3 (2025), 234. doi:10.1007/s42979-025-03785-w [3] AgentMaker. 2021. Paddle-Adversarial-Toolbox (PAT): A Deep Learning Security Library Based on PaddlePaddle. https://github.com/AgentMaker/ Paddle-Adversarial-Toolbox. Accessed: 2026-01-21. [4] Adem Ait, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2022. An Empirical Study on the Survival Rate of GitHub Projects. In Proceedings of the 19th International Conference on Mining Software Repositories. Association for Computing Machinery, New York, NY, USA, 365–375. doi:10.1145/ 3524842.3527941 [5] Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software Engineering for Machine Learning: A Case Study. In Proceedings of the IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice. IEEE, Piscataway, NJ, USA, 291–300. doi:10.1109/icse-seip.2019.00042 [6] Paolo Arcaini, Andrea Bombarda, Silvia Bonfanti, and Angelo Gargantini. 2021. ROBY: A Tool for Robustness Analysis of Neural Network Classifiers. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation. IEEE, Piscataway, NJ, USA, 442–447. doi:10.1109/icst49551.2021.00057 [7] Mohammad Azees, Abdulhassan Alshomali, and Wurood Albayati. 2023. Mining GitHub for Factors That Affect Open Source Software Sustainability. https://www.researchgate.net/publication/373968962 [8] Mateus V. Bertoncello, Gustavo Pinto, and Igor Steinmacher Wiese. 2020. Pull Requests or Commits? Which Method Should We Use to Study Contributors’ Behavior?. In Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, Piscataway, NJ, USA, 467–477. doi:10.1109/saner48275.2020.9054855 [9] Bethge Lab, Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel. 2024. Foolbox: Fast adversarial attacks to benchmark the robustness of machine learning models. https://github.com/bethgelab/foolbox. Version 3.3.4; accessed 2026-01-21. [10] BorealisAI. 2019. AdverTorch: A Toolbox for Adversarial Robustness Research. https://github.com/BorealisAI/advertorch. Version 0.2, accessed 2026-01-21. [11] Hudson Borges and Marco Tulio Valente. 2018. What’s in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform. Journal of Systems and Software 146 (2018), 112–129. doi:10.1016/j.jss.2018.09.016 [12] Houssem Ben Braiek and Foutse Khomh. 2025. Machine learning robustness: A primer. In Trustworthy AI in medical imaging. Elsevier, 37–71. [13] Fabio Calefato, Marco Aurélio Gerosa, Giuseppe Iaffaldano, Filippo Lanubile, and Igor Steinmacher. 2022. Will You Come Back to Contribute? Investigating the Inactivity of OSS Core Developers in GitHub. Empirical Software Engineering 27, 3 (2022), 76. doi:10.1007/s10664-021-10012-6 [14] Donald T. Campbell and Thomas D. Cook. 1990. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin, Boston, MA, USA. [15] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. 2019. On Evaluating Adversarial Robustness. arXiv:1902.06705 [cs.LG] https://arxiv.org/abs/1902.06705 [16] Huaming Chen and M. Ali Babar. 2024. Security for Machine Learning-Based Software Systems: A Survey of Threats, Practices, and Challenges. Comput. Surveys 56, 6 (2024), 39. doi:10.1145/3638531 [17] Pin-Yu Chen and Sijia Liu. 2023. Holistic adversarial robustness of deep learning models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 15411–15420. [18] CleverHans Lab. 2021. CleverHans: An Adversarial Example Library for Benchmarking Machine Learning Models. https://github.com/cleverhanslab/cleverhans. Version 4.0.0, accessed: 2026-01-21. [19] CleverHans Lab. 2021. CleverHans: An Adversarial Example Library for Constructing Attacks, Building Defenses, and Benchmarking Machine Learning Models. https://github.com/cleverhans-lab/cleverhans. Version 4.0.0, accessed August 6, 2026. [20] Jailton Coelho and Marco Tulio Valente. 2017. Why Modern Open Source Projects Fail. In Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering. Association for Computing Machinery, New York, NY, USA, 186–196. doi:10.1145/3106237.3106246 [21] J. Coelho, M. T. Valente, L. Milen, and L. L. Silva. 2020. Is This GitHub Project Maintained? Measuring the Level of Maintenance Activity of Open-Source Projects. Information and Software Technology 122 (2020), 106287. doi:10.1016/j.infsof.2020.106287 [22] Cory Cornelius, Marius Arvinte, Sebastian Szyller, Weilin Xu, and Nageen Himayat. 2025. LLMart: Large Language Model Adversarial Robustness Toolkit. https://github.com/IntelLabs/LLMart. Version 2025.06; accessed 2026-01-21. [23] Valerio Cosentino, Javier L.Cánovas Izquierdo, and Jordi Cabot. 2017. A Systematic Mapping Study of Software Development with GitHub. IEEE Access 5 (2017), 7173–7192. doi:10.1109/ACCESS.2017.2682323 [24] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, and Matthias Hein. 2021. RobustBench: A Standardized Adversarial Robustness Benchmark. arXiv:2010.09670 [cs.LG] https://arxiv.org/abs/2010.09670 [25] Francesco Croce and Matthias Hein. 2020. AutoAttack: Reliable Evaluation of Adversarial Robustness. https://github.com/fra31/auto-attack. GitHub repository, accessed 2026-01-21. [26] Laura Dabbish, Colleen Stuart, Jason Tsay, and James Herbsleb. 2012. Social Coding in GitHub: Transparency and Collaboration in an Open Software Repository. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. Association for Computing Machinery, New York, NY, USA, 1277–1286. doi:10.1145/2145204.2145396 [27] Luca Demetrio, Battista Biggio, and contributors. 2024. secml-malware: A Python Library for Adversarial Robustness Evaluation of Windows Malware Classifiers. https://github.com/pralab/secml_malware. Version 0.4. Accessed: 2026-01-21.

30

Owotogbe et al.

[28] Malinda Dilhara, Ameya Ketkar, and Danny Dig. 2021. Understanding Software 2.0: A Study of Machine Learning Library Usage and Evolution. ACM Transactions on Software Engineering and Methodology 30, 4 (2021), 43. doi:10.1145/3453478 [29] DSE-MSU. 2023. DeepRobust: A PyTorch Library for Adversarial Attacks and Defenses on Images and Graphs. https://github.com/DSE-MSU/ DeepRobust. Accessed: 2026-01-21. [30] Aryaz Eghbali and Michael Pradel. 2020. No strings attached: An empirical study of string-related software bugs. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 956–967. [31] Kevin Eykholt, Taesung Lee, Doug Schales, Jiyong Jang, Ian Molloy, and Masha Zorin. 2020. URET: Universal Robustness Evaluation Toolkit for Adversarial Evasion. https://github.com/IBM/URET. GitHub repository, accessed 2026-01-21. [32] Jonas Gamalielsson and Björn Lundell. 2014. Sustainability of Open Source software communities beyond a fork: How and why has the LibreOffice project evolved? Journal of systems and Software 89 (2014), 128–145. [33] Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, Christopher Ré, and contributors. 2021. Robustness Gym: An Evaluation Toolkit for Machine Learning Models. https://github.com/robustness-gym/robustness-gym. Version 0.1.2. Accessed: 2026-01-21. [34] Dou Goodman, Hao Xin, Wang Yang, Wu Yuesheng, Xiong Junfeng, and Zhang Huan. 2020. AdvBox: A Toolbox to Generate Adversarial Examples. https://github.com/advboxes/AdvBox. GitHub repository, accessed 2026-01-21. [35] Georgios Gousios, Bogdan Vasilescu, Alexander Serebrenik, and Andy Zaidman. 2014. Lean GHTorrent: GitHub Data on Demand. In Proceedings of the 11th Working Conference on Mining Software Repositories. Association for Computing Machinery, New York, NY, USA, 384–387. doi:10.1145/ 2597073.2597126 [36] Wenbo Guo, Zhengzi Xu, Chengwei Liu, Cheng Huang, Yong Fang, and Yang Liu. 2023. An empirical study of malicious code in pypi ecosystem. In 2023 38th IEEE/ACM Int. Conf. on Automated Software Engineering (ASE). IEEE, 166–177. [37] Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In Proceedings of the 7th International Conference on Learning Representations. OpenReview.net, Online, 1–1. arXiv:1903.12261 [cs.LG] https://arxiv.org/abs/1903.12261 [38] Yacine Izza and João Marques-Silva. 2023. The Pros and Cons of Adversarial Robustness. In Proc. of the International Conference on Agents and AI, Vol. 2. SciTePress, Setúbal, Portugal, 386–397. doi:10.5220/0013166300003890 [39] Eirini Kalliamvakou, Leif Singer, Georgios Gousios, Daniel M. German, and Daniela Blincoe. 2014. The Promises and Perils of Mining GitHub. In Proceedings of the 11th Conference on Mining Software Repositories. Association for Computing Machinery, New York, NY, USA, 92–101. doi:10.1145/2597073.2597074 [40] Samer Khamaiseh, Steven Chiacchira, Aibak Aljadayh, Deirdre Jost, et al. 2023. AiR-TK: AI Robustness Testing Kit. https://github.com/LAiSRSK/AiRobustnessTestingKit-AiR-TK-. MIT License, accessed: 2026-01-21. [41] Nafiz Imtiaz Khan, Priyal Soni, Arjun Ashok, and Vladimir Filkov. 2025. OSSPREY: AI-Driven Forecasting and Intervention for OSS Project Sustainability. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 4074–4077. [42] Sayedhassan Khatoonabadi, Diego Elias Costa, Rabe Abdalkareem, and Emad Shihab. 2023. On Wasted Contributions: Understanding the Dynamics of Contributor-Abandoned Pull Requests: A Mixed-Methods Study of 10 Large Open-Source Projects. ACM Transactions on Software Engineering and Methodology 32, 1 (2023), 15. doi:10.1145/3530785 [43] Hoki Kim. 2020. Torchattacks: A PyTorch Library for Adversarial Attacks. https://github.com/Harry24k/adversarial-attacks-pytorch. Version 3.5.1, accessed 2026-01-21. [44] LF AI & Data Foundation. 2022. Adversarial Robustness Toolbox. LF AI & Data Foundation. https://lfaidata.foundation/projects/adversarialrobustness-toolbox/ Graduated in February 2022. Contributed by IBM in June 2020. Accessed: June 26, 2026. [45] Xiang Ling, Shouling Ji, Jiaxu Zou, Jiannan Wang, Chunming Wu, Bo Li, and Ting Wang. 2019. DEEPSEC: A Uniform Platform for Security Analysis of Deep Learning Models. https://github.com/ryderling/DEEPSEC. Accessed: 2026-01-21. [46] Jia Liu and Yaochu Jin. 2023. A comprehensive survey of robust deep learning in computer vision. Journal of Automation and Intelligence 2, 4 (11 2023), 175–195. doi:10.1016/j.jai.2023.10.002 [47] LPD EPFL, Marc González, Rachid Guerraoui, Rafael Pinot, Geovani Rizk, John Stephan, and François Taïani. 2025. ByzFL: A Research Framework for Robust Federated Learning. https://github.com/LPD-EPFL/byzfl. Version v0.0.11; accessed 2026-01-21. [48] Pooya Rostami Mazrae, Alexandre Decan, Tom Mens, and Mairieli Wessel. 2026. An empirical study of the evolution of GitHub Actions workflows. Journal of Systems and Software (2026), 112824. [49] Microsoft, Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. 2023. PromptBench: A Unified Library for Evaluating and Understanding Large Language Models. https://github.com/microsoft/promptbench. Accessed: 2026-01-21. [50] Microsoft Corporation. 2022. Counterfit. https://github.com/Azure/counterfit. Version 1.1.0, accessed: 2026-01-21. [51] Sadia Afrin Mim, Fatemeh Vares, Andrew Meenly, and Brittany Johnson. 2025. An Investigation into Open Source Fairness Tool Sustainability. In Proceedings of the 1st International Workshop on Responsible Software Engineering. Association for Computing Machinery, New York, NY, USA, 21–28. doi:10.1145/3711919.3728679 [52] Sina Mohseni, Haotao Wang, Chaowei Xiao, Zhiding Yu, Zhangyang Wang, and Jay Yadawa. 2023. Taxonomy of Machine Learning Safety: A Survey and Primer. Comput. Surveys 55, 8 (2023), 38 pages. doi:10.1145/3551385 [53] Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian M. Molloy, and Ben Edwards. 2019. Adversarial Robustness Toolbox v1.0.0. https://arxiv.org/abs/1807.01069

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

31

[54] Sabato Nocera, Massimiliano Di Penta, Fatima Ahmed, Simone Romano, and Giuseppe Scanniello. 2025. What We Know about AIBOMs: Results from a Multivocal Literature Review on Artificial Intelligence Bill of Materials. ACM Transactions on Software Engineering and Methodology (2025). [55] Marco Ortu, Giuseppe Destefanis, Daniel Graziotin, Michele Marchesi, and Roberto Tonelli. 2020. How Do You Propose Your Code Changes? Empirical Analysis of Affect Metrics of Pull Requests on GitHub. IEEE Access 8 (2020), 110897–110907. doi:10.1109/access.2020.3002663 [56] Joshua Owotogbe, Indika Kumara, Dario Di Nucci, Damian Andrew Tamburri, and Willem-Jan van den Heuvel. 2025. Chaos Engineering in the Wild: Findings from GitHub. arXiv preprint arXiv:2505.13654 (2025). [57] Andrei Paleyes, Raoul Gabriel Urma, and Neil D. Lawrence. 2023. Challenges in Deploying Machine Learning: A Survey of Case Studies. Comput. Surveys 55, 6 (2023), 38 pages. doi:10.1145/3533378 [58] Sotiris Pelekis, Thanos Koutroubas, Afroditi Blika, Anastasis Berdelis, Evangelos Karakolis, Christos Ntanos, Evangelos Spiliotis, and Dimitris Askounis. 2025. Adversarial Machine Learning: A Review of Methods, Tools, and Critical Industry Sectors. Artificial Intelligence Review (2025), 32 pages. doi:10.1007/s10462-025-11147-4 Online first. [59] PRALab. 2025. SecML-Torch: A Library for Robustness Evaluation of Deep Learning Models. https://github.com/pralab/secml-torch. Version 1.3, accessed 2026-01-21. [60] QData. 2024. TextAttack: A Python Framework for Adversarial Attacks and Data Augmentation in NLP. https://github.com/QData/TextAttack. Version 0.3.10, accessed: 2026-01-21. [61] Perusahaan Grup Qi, An Xin, Yuekang Li, Yi Liu, Tianwei Zhang, and Yang Liu. 2023. “Security Is Not My Field, I’m a Stats Guy”: A Qualitative Root Cause Analysis of Barriers to Adversarial Machine Learning Defenses in Industry. In Proceedings of the USENIX Security Symposium. USENIX Association, Boston, MA, USA, 3763–3780. [62] Joaquin Quiñonero-Candela, Anton Sugiyama, and Neil Lawrence. 2022. Dataset Shift in Machine Learning. MIT Press, Cambridge, MA, USA. [63] Petar Radanliev, Omar Santos, Carsten Maple, and Kayvan Atefi. 2026. Operationalising artificial intelligence bills of materials for verifiable AI provenance and lifecycle assurance. Frontiers in Computer Science 8 (2026), 1735919. [64] Mohammad Masudur Rahman and Chanchal K. Roy. 2014. An Insight into the Pull Requests of GitHub. In Proceedings of the 11th Working Conference on Mining Software Repositories. Association for Computing Machinery, New York, NY, USA, 364–367. doi:10.1145/2597073.2597121 [65] Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, New York, NY, USA, 33–44. doi:10.1145/ 3351095.3372873 [66] Jonas Rauber, Wieland Brendel, and Matthias Bethge. 2017. Foolbox: A Python Toolbox to Benchmark the Robustness of Machine Learning Models. arXiv:1707.04131 [cs.LG] https://arxiv.org/abs/1707.04131 [67] Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. 2024. RobustVLM: Code for Robust CLIP and Robust Vision-Language Models. https://github.com/chs20/RobustVLM. Accessed 2026-01-21. [68] D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Vinay Phillips, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems, Vol. 28. Curran Associates, Inc., Red Hook, NY, USA. [69] Karthik Shivashankar, Ghadi S. Al Hajj, and Antonio Martini. 2025. Scalability and Maintainability Challenges and Solutions in Machine Learning: A Systematic Literature Review. arXiv:2504.11079 [cs.SE] https://arxiv.org/abs/2504.11079 [70] Yanqi Su, Zhenchang Xing, Xin Peng, Xin Xia, Chong Wang, Xiwei Xu, and Liming Zhu. 2021. Reducing bug triaging confusion by learning from mistakes with a bug tossing knowledge graph. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 191–202. [71] Jiazheng Sun et al. 2023. Canary SEFI: An Adversarial Robustness Evaluation Framework for Deep Learning Models. https://github.com/NeoSunJZ/ Canary_Master. Apache-2.0 License, accessed: 2026-01-21. [72] Jiazheng Sun, Li Chen, Chenxiao Xia, Dexin Zhang, Rui Huang, Zhen Qiu, and Weijia Xiong. 2023. CANARY: An Adversarial Robustness Evaluation Platform for Deep Learning Models on Image Classification. Electronics 12, 17 (2023), 3665. doi:10.3390/electronics12173665 [73] System Integration and Security Lab (SISL), Allie Griffith, et al. 2025. ASTRA-RL: Adaptive Stress Testing for Robust AI. https://github.com/sisl/astra-rl. Version 0.0.1, accessed 2026-01-21. [74] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing Properties of Neural Networks. In Proceedings of the International Conference on Learning Representations. ICLR, Banff, AB, Canada, 10 pages. arXiv:1312.6199 [75] TextFlint Team. 2022. TextFlint: A Unified Multilingual Robustness Evaluation Toolkit for NLP. https://github.com/textflint/textflint. Version 0.1.0, accessed: 2026-01-21. [76] THU-ML Group. 2022. ARES 2.0: Adversarial Robustness Evaluation for Safety. https://github.com/thu-ml/ares. Apache-2.0 License, accessed: 2026-01-21. [77] THUDM. 2021. GRB: Graph Robustness Benchmark. https://github.com/THUDM/grb. Version 0.1.0, accessed: 2026-01-21. [78] THUNLP. 2021. OpenAttack (Release 1.0). https://github.com/thunlp/OpenAttack/releases. Latest release accessed: 2026-01-21. [79] Ozan K. Tonguz and Federico Taschin. 2025. On the Measurement of Distribution Shift in Machine Learning Systems. IEEE Intelligent Systems 40, 2 (2025), 45–54. doi:10.1109/mis.2024.3524798

32

Owotogbe et al.

[80] Trusted-AI. 2025. Adversarial Robustness Toolbox (ART). https://github.com/Trusted-AI/adversarial-robustness-toolbox. Version 1.20.1, accessed 2026-01-21. [81] Ipek Ustunboyacioglu, Indika Kumara, Dario Di Nucci, Damian Andrew Tamburri, and Willem-Jan Van Den Heuvel. 2024. Data quality assessment in the wild: Findings from GitHub. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 120–129. [82] Marat Valiev, Bogdan Vasilescu, and James Herbsleb. 2018. Ecosystem-Level Determinants of Sustained Activity in Open-Source Projects: A Case Study of the PyPI Ecosystem. In Proceedings of the 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for Computing Machinery, New York, NY, USA, 644–655. doi:10.1145/3236024.3236062 [83] Zhiyuan Wan, Gail C Murphy, and Xin Xia. 2020. Predicting code context models for software development tasks. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 809–820. [84] Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, and Björn Regnell. 2012. Experimentation in Software Engineering. Springer, Berlin, Heidelberg. doi:10.1007/978-3-642-29044-2 [85] Yanran Xie. 2024. AI-Robustness-Platform: A PyTorch-Based System for Adversarial Attack and Defense. https://github.com/xieyanran/AIRobustness-Platform. Accessed: 2026-01-21. [86] Pulei Xiong, Scott Buffett, Shahrear Iqbal, Philippe Lamontagne, Mohammad Mamun, and Heather Molyneaux. 2022. Towards a robust and trustworthy machine learning system development: An engineering perspective. Journal of Information Security and Applications 65 (2022), 103121. [87] Peiyu Xiong, Michael Tegegn, Jaskeerat Singh Sarin, Shubhraneel Pal, and Julia Rubin. 2024. It Is All About Data: A Survey on the Effects of Data on Adversarial Robustness. Comput. Surveys 56, 7 (2024), 34 pages. doi:10.1145/3627817 [88] Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2022. Machine Learning Testing: Survey, Landscapes and Horizons. IEEE Transactions on Software Engineering 48, 1 (2022), 1–21. doi:10.1109/tse.2019.2962027 [89] Xunhui Zhang, Yue Yu, Georgios Gousios, and Ayushi Rastogi. 2023. Pull Request Decisions Explained: An Empirical Overview. IEEE Transactions on Software Engineering 49, 2 (2023), 849–871. doi:10.1109/tse.2022.3165056 [90] Jiayuan Zhou, Michael Pacheco, Zhiyuan Wan, Xin Xia, David Lo, Yuan Wang, and Ahmed E Hassan. 2021. Finding a needle in a haystack: Automated mining of silent vulnerability fixes. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 705–716. [91] Shurui Zhou, Bogdan Vasilescu, and Christian Kästner. 2019. What the Fork: A Study of Inefficient and Efficient Forking Practices in Social Coding. In Proceedings of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for Computing Machinery, New York, NY, USA, 350–361. doi:10.1145/3338906.3338918

Related documents

• Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study#792404
• Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study#788404
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues#820515
• Demystifying Machine Learning: Applications in African Environmental Science and Engineering#684900
• Benchmarking the Titans: A Multi-Dimensional Empirical Evaluation of LLM Code Generation Quality in the .NET Ecosystem#482210