Biomedical open source software: Crucial packages and hidden heroes.

Biomedical open source software: Crucial packages and hidden heroes - PMC Skip to main content An official website of the United States government Here's how you know Here's how you know Search Log in Primary site navigation Search Log in Search PMC Full-Text Archive Search in PMC Journal List User Guide As a library, NLM provides access to scientific literature. Inclusion in an NLM database does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: PMC Disclaimer | PMC Copyright Notice PLoS Comput Biol . 2026 May 13;22(5):e1014260. doi: 10.1371/journal.pcbi.1014260 Biomedical open source software: Crucial packages and hidden heroes Eva Maxfield Brown 1, # , Stephan Druskat 2, # , Laurent Hébert-Dufresne 3, # , James Howison 4, *, # , Daniel Mietchen 5, 6, # , Andrew Nesbitt 7, # , João Felipe Pimentel 8, # , Boris Veytsman 9, 10, # Editor: Varun Dutt 11 Author information Article notes Copyright and License information 1 Information School, University of Washington, Seattle, Washington, United States of America 2 Institute of Software Technology, German Aerospace Center (DLR), Berlin, Germany 3 Department of Computer Science, University of Vermont, Burlington, Vermont, United States of America 4 School of Information, The University of Texas at Austin, Austin, Texas, United States of America 5 Leibniz Institute for Information Infrastructure, FIZ Karlsruhe, Berlin, Germany 6 Institute for Globally Distributed Open Research and Education (IGDORE), Jena, Germany 7 Ecosyste.ms, London, United Kingdom 8 Instituto de Computação, Universidade Federal Fluminense, Niterói, Rio de Janeiro, Brazil 9 Chan Zuckerberg Initiative, Redwood City, California, United States of America 10 School of Systems Biology, George Mason University, Fairfax, Virginia, United States of America 11 Indian Institute of Technology Mandi - Kamand Campus: Indian Institute of Technology Mandi, INDIA ✉ * E-mail: jhowison@ischool.utexas.edu The authors have declared that no competing interests exist. # Contributed equally. Roles Eva Maxfield Brown : Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft Stephan Druskat : Conceptualization, Investigation, Methodology, Software, Validation, Writing – original draft, Writing – review & editing Laurent Hébert-Dufresne : Formal analysis, Investigation, Methodology, Visualization, Writing – original draft James Howison : Conceptualization, Writing – review & editing Daniel Mietchen : Conceptualization, Writing – review & editing Andrew Nesbitt : Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft João Felipe Pimentel : Conceptualization, Writing – review & editing Boris Veytsman : Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing Varun Dutt : Editor Received 2024 Dec 25; Accepted 2026 Apr 20; Collection date 2026 May. © 2026 Brown et al This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. PMC Copyright notice PMCID: PMC13235917  PMID: 42127143 Abstract Despite the importance of scientific software for research, it is often not formally recognized nor rewarded. This is especially true for foundational libraries, which are hidden below packages visible to the users (and thus doubly hidden, since even the packages directly used in research are frequently not visible in the paper). Research stakeholders like funders, infrastructure providers, and other organizations need to understand the complex network of computer programs that contemporary research relies upon. In this work, we use the CZ Software Mentions Dataset to map the upstream dependencies of software used in biomedical papers and find the packages critical to scientific software ecosystems. We propose centrality metrics for the network of software dependencies, analyze three ecosystems (PyPI, CRAN, Bioconductor), and determine the packages with the highest centrality. Author summary Scientists today use software as an important tool for their research. The progress of science depends on the people who write software being properly recognized, rewarded, and incentivized. This means that those who fund research or promote scientists need to know the impact of software and identify the most important programs. The universe of scientific software is complex: some people write programs used by researchers, while some people write programs used by other programs. The latter are often even less visible than the former, but they are important for the progress of science. In this work, we establish metrics to measure the impact of both types of software. We introduce the network of dependencies (program A used by researchers depends on programs B, C, and D) and use it to calculate the impact of a large number of packages used in the biomedical literature. In doing so, we identify important software packages, regardless of their direct visibility to end users. We conclude with a discussion of the limitations of our methodology and approaches to improve network validity. 1 Introduction Since the second half of the last century, a computer has become as ubiquitous a tool of a scientific lab as balance scales and a Bunsen burner were in previous ages. As a consequence, computer software is now crucial to research, bringing new methods and new scale, while offering new potential for reproducibility and extension. This is true not only for the natural sciences and mathematics but for scholarship more broadly (e.g., sociology and the humanities), making the software revolution both wide and deep. Yet we have very limited insight into the software actually used in research. This lack of infrastructural understanding means we are limited in our ability to reward developers and maintainers, encourage collaboration and coordination, and direct science funding in a well-informed manner. Scientific software is often invisible in publications, because citation practices in science have not changed at the same pace as software has become crucial. For example, software is infrequently and inconsistently formally cited [ 1 – 6 ]. There have been recent efforts to extract informal citations from the full text of articles [ 7 – 10 ] and to evaluate the “importance” of software packages by looking at papers that cite them [ 11 ]. Unfortunately, publications sometimes do not mention all the software used in the course of research. Besides this, there is another kind of invisibility of scientific software. The programs visible to the end user may rely on many other software packages (known in the software world as dependencies). While the end users may mention a package at the top of the dependency stack, they are likely not even aware of the full set of packages that are further below. These may be packages that the user-facing program depends on directly ( direct dependencies ), or indirectly, where the direct dependencies in turn may depend on other packages. These latter packages thus become indirect, transitive dependencies of the user-facing program. This complexity in the network of dependencies has a number of implications, including those for security [ 12 , 13 ] and computational reproducibility [ 14 ]. In particular, much of the work undertaken to develop, maintain, test, and distribute the underlying software is not directly visible in the publication record itself, and is not included in derivatives such as citation networks and knowledge graphs. The situation resembles the famous XKCD cartoon where “all modern infrastructure” critically depends on “a project some random person in Nebraska has been thanklessly maintaining since 2003” [ 15 ]. The word “thanklessly” is important in this context: being unknown, these critical pieces of software get much less recognition and credit than they deserve—and than the science needs. The absence of recognition may lead to dire consequences, for example, if, due to the lack of funded maintenance, the underlying libraries become vectors for malware attacks. There have been proposals to assign credit to these packages using the dependency structure to calculate what has been termed transitive credit [ 16 , 17 ] but so far, the problem has not yet been solved. To implement the proposed measure, software projects must start to publicize the packages they rely on in a way that enables recognition and citation, beyond the technical dependency already recorded in manifest files such as pyproject.toml , DESCRIPTION , cargo.toml , pom.xml , etc. One way for projects to do this is the inclusion of citation information for their own software outputs as well as for their direct dependencies, e.g., a citation file in the Citation File Format (CFF) [ 18 ]. Were such citation information available for the complete dependency stack of a program, transitive credit could be implemented by building weighted software citation networks [ 19 ]. At present, though, the situation is quite different: sometimes even the maintainers of the lower-stack software are not aware of the upper-stack programs that depend on their work. They need this knowledge when making breaking changes to their packages, which might negatively influence the software that depends on them [ 20 ]. By classifying software packages as visible to the end users or primarily important for other packages, we follow the ideas described by Donald E. Stokes in his famous book, Pasteur’s Quadrant [ 21 ]. There, Stokes distinguishes between applied research with the results visible to the general public, and pure research, which is less visible, but important for the applied research. He discussed a two-dimensional diagram of scientific works, with the importance of the works for the applications on the x axis and the quest for fundamental understanding on the y axis. Thus, the upper left quadrant is occupied by the purely fundamental research personified by Niels Bohr. The lower right quadrant is occupied by purely applied research personified by Thomas Edison. The upper right quadrant is occupied by the works that have both fundamental and practical value. Stokes chose Louis Pasteur as an example of such works. While this picture is rather simplified (Bohr’s works have significant practical value, and Edison’s experiments stimulated fundamental research), it gives important insight into scientific endeavor. Following these ideas, we can put the software packages into a two-dimensional plane with the axes corresponding to the frequency of software mentions and a measure of network centrality ( Fig 1 ). The majority of the packages will likely occupy the lower left corner of the plot, having a small number of mentions from the authors of scientific papers and limited centrality from usage in other software packages. Popular packages are used by many authors. We are interested in the “Nebraska” packages, not very visible, but of critical importance. We use the term “Nebraska” packages throughout this article in reference to the original XKCD cartoon which noted that much of highly critical software is often less visible and less well known [ 15 ]. Of course, one can think about highly visible and critical “Pasteur” packages, which are used both directly and as a foundation for other libraries. Fig 1. Classification of software packages inspired by Stokes’ classification system in [ 21 ]. Open in a new tab “Nebraska” packages are software projects which have few mentions in research articles, but are highly central in a dependency network. “Pasteur” packages are both highly visible with lots of mentions and are highly central in a dependency network. We had a fortunate opportunity to explore these ideas due to the generosity of the Chan Zuckerberg Initiative (CZI). On 24–27 October 2023, CZI hosted a hackathon on Mapping the Impact of Research Software in Science (see https://github.com/chanzuckerberg/software-impact-hackathon-2023 ). One of the projects at this hackathon was Tracing the dependencies of open source software mentioned in the biomedical literature [ 22 ]. In this project, we explored the dependencies of the open source software packages mentioned in the CZ Software Mentions dataset [ 9 ]. The dataset was created using a machine learning system trained on the SoftCite gold standard dataset of manually annotated software mentions [ 8 ] and consists of extractions from 2.4 million biomedical papers. The methodology and its limitations are discussed in [ 10 ]. In particular, the work on the disambiguation of packages can be improved. Later, we discuss how these limitations influence our results. We decided to limit our study to open-source packages: first, because the dependencies of closed-source packages are not public, and second, because we believe in the importance of open source for open and reproducible science. Here, we explore approaches to making the software package infrastructure underlying science more visible. We examine our findings to develop questions to better understand the idea of criticality and opportunities for improving science through adjustments to the research software ecosystems. 2 Materials and methods 2.1 Network construction We combined three datasets: The CZ Software Mentions dataset [ 9 ], which was constructed using a machine learning model trained on the SoftCite Gold Standard annotations [ 8 ], the Ecosyste.ms software dependency dataset [ 23 ], which is built by gathering and normalizing the dependency information from multiple software ecosystems, including one for the Python programming language (PyPI) and two for the R programming language (CRAN and Bioconductor), and finally we used OpenAlex [ 24 ] for information on how frequently papers had been cited. The software authorship data were scraped from GitHub metadata. The CZ Software Mentions dataset [ 9 , 10 ] includes mentions from 2.4 million biomedical papers and identifies which mentions were traced to which ecosystem. In this way, package names from CRAN, Bioconductor, and PyPI are parsed by the software mentions extraction model, collected, and finally disambiguated. We collected information for each package, including its dependencies, using the latest release of the package at the time of data collection (October 2023). Dependencies were then recursively retrieved using the most recent release and following dependencies until the full list of transitive dependencies was obtained. Limitations of our dependency resolution approach are discussed later. After this data processing, we produced a two-mode network, with nodes for papers and nodes for software packages. Edges from papers to software packages were added when a package was mentioned in the full text of the article. Edges between packages were added when metadata descriptions indicated a required dependency. The full code for the processing is available at [ 22 ]. The network is available at [ 25 ] in GEXF format [ 26 ] and has four classes of nodes: paper: papers from the CZ Software Mentions dataset. They are identified by their Digital Object Identifier (DOI). To estimate the impact of the papers, we separately downloaded their citation numbers from OpenAlex [ 24 ] as of November 2023 (a copy is available in the data subdirectory of [ 22 ]). pypi, cran, bioconductor: Software from the corresponding ecosystems. We used CZI ID [ 9 , 10 ] as the identifier. We did not attempt to identify the same software across the ecosystems (see the discussion below). The edges are directed and weighted. An edge from one software node A to another software node B means that software A depends on software B as determined by the corresponding metadata. An edge from a paper node to a software node means that the given software is mentioned in the paper, and the weight corresponds to the number of citations the paper received. In what follows, we will consider both the weighted and unweighted versions of this network, as utilization of the weights has both positive and negative aspects to consider during analysis and interpretation. 2.2 Network analysis To analyze the network, we relied on directed centrality analysis [ 27 ]. There are several possible options for centrality measures. Our choice was determined by the following considerations. First, we wanted a centrality metric that can account for papers even if they have no incoming edges in our network (in other words, papers do not receive centrality but should contribute to the centrality of software packages). This criterion excludes eigenvector centrality [ 28 ]. Second, we wanted a centrality metric that can account for the weight of edges from papers to software packages (i.e., some papers contribute more centrality because they are more highly cited.). This criterion excludes the PageRank algorithm [ 29 ], which normalizes weights of out-degree. We found that Katz centrality [ 30 ] satisfies all our criteria. Katz centrality can be interpreted as the importance of the node for the diffusion processes on the network. This corresponds to the intuition that innovation is a diffusion-like process [ 31 ]. Katz centrality gives us the attenuation factor β as a free parameter to control the importance of papers (which we set equal to 1), such that paper nodes contribute a factor proportional to β times their citation count to the software packages they mention (we follow the formulation of Katz centrality in [ 32 ]). In turn, software packages contribute a factor proportional to their own centrality to their dependencies. A package can therefore be central by receiving mentions from well-cited papers, by having central dependents, or by a combination of both. We additionally provide analysis for three variations of our network: unweighted, weighted, and the largest connected component from the weighted graph. The unweighted form of our network ignores the weights added to the edges between the seeding articles and their software mentions. The weighted network includes the article citation count weights. The differences between the unweighted and weighted versions of our analysis allow us to parse different subtypes of the results. Specifically, comparing the weighted and unweighted networks allows us to identify software which is only important due to its utilization and reference in high citation count articles. 3 Results In Fig 2 , we show the overall network of software packages connected by their dependencies within each ecosystem and interconnected through the papers that mention them. Edges from papers to software packages are directed and weighted by the number of citations the paper received. Edges between software dependencies are directed from a dependent to a dependency. Fig 2. (a) Network visualization of software packages from three ecosystems (from CRAN in green, PyPI in blue, and Bioconductor in pink) connected through their dependencies within their ecosystem and interconnected through papers that mention them. Open in a new tab We label the top 3 most central packages in each ecosystem: ggplot2 [ 33 ], SAM [ 34 ], and PRISMA [ 35 ] for CRAN, velvet [ 36 ], tophat and pymol [ 37 ] for PyPI and DESeq2 [ 38 ], edgeR [ 39 ] and limma [ 40 , 41 ] for Bioconductor. The core of the network is dominated by CRAN and PyPI dependencies, despite the fact that three of the five most central packages come from Bioconductor. (b) The top part of the above network, with papers added (in grey) to illustrate how PRISMA [ 35 ] can be central due to many mentions in papers. We found a dense core of popular packages that receive many mentions (e.g., ggplot2 in CRAN [ 33 ], tophat in PyPI and limma in Bioconductor [ 40 , 41 ]), some of which have many dependencies themselves (e.g., ggplot2 [ 33 ]). We also found packages specific to certain communities (e.g., PRISMA in CRAN [ 35 ] or pymol in PyPI [ 37 ]). Analyzing the entire ecosystem of software dependencies, we found that roughly 10% of software packages are part of dependency loops (i.e., cycles in the dependency networks). Interestingly, we found no cycles in the connected components of networks that have received software mentions, and their dependencies. We discuss this finding below. Note that contemporary package managers usually resolve the dependency loops by installing all packages in the loop simultaneously, so the loops, while indicating rather sloppy software practices, are not fatal. In Fig 3 , we show the distribution of packages from the three investigated ecosystems over Katz centrality and mention counts. We used three different ways to calculate centrality: (a) unweighted graph, (b) weighted complete graph, and (c) the largest connected components of the weighted graph (calculated separately for each ecosystem). It is remarkable that all three models give rather similar results for many packages. In the next section, we discuss the insights from the cases when the different methods do not agree. As expected, we found a dense cluster of packages with low Katz centrality and low mention counts, i.e., the “Majority” quadrant (see Fig 1 ). Fig 3. Distribution of packages by Katz centrality and counts of their mentions in papers. Open in a new tab Katz centrality is calculated for an unweighted graph, for a weighted graph with all nodes, or just for the largest connected cluster (LCC) for each ecosystem. In the calculations, we assumed β = 1. Fig 3 suggests large differences between the packages in the majority quadrant and the packages in the rest of the system. This is further illustrated by Tables 1 and 2 . The sum of the median and half of the interquartile range is several orders of magnitude smaller than the maximal value for both the number of mentions and centrality. Table 1. Summary statistics for the number of mentions in papers. “Number” is the number of packages, “Dependency only” is the fraction of packages not mentioned in any papers and pulled by the dependencies only, “Median” is the median number of mentions, “IQR” is the interquartile range for mentions count, “Max” is the maximal number of mentions, and “Gini” is the Gini coefficient for the mentions count. Ecosystem Number Dependency only Mentions count Median IQR Max Gini bioconductor 1018 0.047 24 68 21257 0.843712143943309 cran 3594 0.11324429604897 9 34 21960 0.889196914901722 pypi 5596 0.14349535382416 4 17 20322 0.888156736619396 Open in a new tab Table 2. Summary statistics for Katz centrality. “Median” is the median centrality, “IQR” is the interquartile range, “Max” is the maximal centrality, and “Gini” is the Gini coefficient. Ecosystem Median IQR Max Gini Unweighted graph bioconductor 0.001 0.00065127467783916 0.217934004136626 0.549779090129221 cran 0.00116646210956267 0.00048602587898445 0.277617982075916 0.496401393691949 pypi 0.00108092155486141 0.00019441035159378 0.191008170440888 0.351314534735671 Weighted, complete graph bioconductor 0.00143155318286753 0.000322302811767757 0.105238137865536 0.358203102844196 cran 0.00138817278338669 0.000147428626544985 0.128618817548222 0.253046882585352 pypi 0.00138275023345159 0.00010438408625076 0.07377243622961 0.180193304141016 Weighted, LCC only bioconductor 0.00288301570993583 0.00075920092370992 0.210742476047381 0.371663779294751 cran 0.00279343047601127 0.000369917791154937 0.257562976941226 0.280390754327007 pypi 0.00278528636383631 0.00031286964272138 0.147731480149716 0.220244615942456 Open in a new tab A convenient measure of this inequality is the Gini coefficient [ 42 ]. Gini coefficients close to one characterize a very unequal distribution, while Gini coefficients close to zero correspond to a uniform distribution. As seen from Table 1 , the distribution of mention counts is wildly unequal with high Gini coefficients. The distribution of centrality ( Table 2 ) is smoother, especially for the weighted graph, but is still rather unequal. Perhaps an even more robust measure of “Nebraska” properties of the ecosystem is the number of packages that are not mentioned in any paper, but are in the dependencies of the packages that are mentioned (we are grateful to the anonymous reviewer of PLOS Computational Biology for suggesting this metric). It is shown in Table 1 . In Tables 3 , 4 , and 5 , we show the packages in “Pasteur”, Popular, and “Nebraska” packages using the following criteria: Table 3. Software quadrants, unweighted graph. Software Ecosystem Mentions Centrality Count Percentile Value Percentile “Pasteur” PRISMA cran 15797 0.9997 0.27762 1 DESeq2 bioconductor 21257 0.9999 0.19888 0.9997 ggplot2 cran 14441 0.9995 0.23407 0.9999 pymol pypi 14604 0.9996 0.19101 0.9996 PERMANOVA cran 21960 1 0.1219 0.9992 limma bioconductor 13451 0.9993 0.21793 0.9998 SAM cran 13048 0.9992 0.14425 0.9994 edgeR bioconductor 12923 0.9991 0.17137 0.9995 lme4 cran 10510 0.999 0.13191 0.9993 WGCNA cran 13915 0.9994 0.09721 0.9988 vegan cran 10012 0.9989 0.12121 0.9991 tophat pypi 7642 0.9987 0.1184 0.999 STAR cran 9091 0.9988 0.11052 0.9989 .......... scipy pypi 1318 0.9921 0.02081 0.9916 numpy pypi 962 0.9884 0.0176 0.99 .......... napari pypi 11 0.5873 0.00118 0.6104 Popular GSVA bioconductor 2937 0.9969 0.00108 0.361 MAST bioconductor 2091 0.995 0.00108 0.361 FSA cran 499 0.9782 0.00108 0.361 Boruta cran 441 0.9754 0.00108 0.361 MixSIAR cran 377 0.9714 0.00117 0.4989 DRIMSeq bioconductor 335 0.9678 0.00117 0.4989 netDx bioconductor 174 0.9419 0.00107 0 COCOA bioconductor 162 0.9378 0.00117 0.4989 emoji pypi 161 0.9374 0.00107 0 aviso pypi 160 0.937 0.00117 0.4989 SiZer cran 159 0.9366 0.00117 0.4989 searchlight pypi 137 0.9273 0.00107 0 “Nebraska” vctrs cran 2 0.2516 0.02438 0.9928 withr cran 2 0.2516 0.02438 0.9928 isoband cran 0 0 0.02438 0.9928 gss cran 7 0.4991 0.01212 0.9849 SuppDists cran 3 0.3287 0.00567 0.9658 lfda cran 1 0.1232 0.0049 0.9581 ggtext cran 7 0.4991 0.00482 0.957 texttable pypi 1 0.1232 0.00478 0.9569 affyio bioconductor 5 0.4326 0.00473 0.956 dnaio pypi 1 0.1232 0.00472 0.9558 requests pypi 7 0.4991 0.00419 0.9499 tifffile pypi 4 0.3863 0.00409 0.9488 Open in a new tab Table 4. Software quadrants, weighted graph (full). Software Ecosystem Mentions Centrality Count Percentile Value Percentile “Pasteur” PRISMA cran 15797 0.9997 0.12862 1 DESeq2 bioconductor 21257 0.9999 0.08735 0.9996 SAM cran 13048 0.9992 0.10545 0.9999 limma bioconductor 13451 0.9993 0.10524 0.9998 ggplot2 cran 14441 0.9995 0.07401 0.9995 PERMANOVA cran 21960 1 0.04861 0.999 edgeR bioconductor 12923 0.9991 0.10309 0.9997 WGCNA cran 13915 0.9994 0.05195 0.9992 pymol pypi 14604 0.9996 0.04661 0.9989 tophat pypi 7642 0.9987 0.07377 0.9994 vegan cran 10012 0.9989 0.05172 0.9991 lme4 cran 10510 0.999 0.03738 0.9986 .......... scipy pypi 1318 0.9921 0.00762 0.9881 numpy pypi 962 0.9884 0.0083 0.9891 .......... napari pypi 11 0.5873 0.00136 0.2967 Popular GSVA bioconductor 2937 0.9969 0.00136 0.1214 MAST bioconductor 2091 0.995 0.00136 0.1214 GSA cran 844 0.9868 0.00136 0.2163 FSA cran 499 0.9782 0.00136 0.1508 Boruta cran 441 0.9754 0.00136 0.0968 MixSIAR cran 377 0.9714 0.00138 0.4265 AntWeb cran 282 0.9616 0.00138 0.4265 emoji pypi 161 0.9374 0.00138 0.4356 eureqa pypi 149 0.9331 0.00138 0.4265 searchlight pypi 137 0.9273 0.00136 0.259 pathfindR cran 133 0.9259 0.00137 0.4023 ChIPXpress bioconductor 125 0.9213 0.00138 0.4129 pypet pypi 125 0.9213 0.00136 0 ADAPTS cran 125 0.9213 0.00137 0.3452 “Nebraska” calculate_expression pypi 3 0.3287 0.01872 0.9956 zscore pypi 5 0.4326 0.00851 0.9893 webpages pypi 2 0.2516 0.00707 0.9872 setuptools pypi 6 0.4689 0.00572 0.9825 gpviz pypi 1 0.1232 0.00453 0.9759 glib pypi 3 0.3287 0.00397 0.9711 ego pypi 5 0.4326 0.00326 0.9607 xtick pypi 1 0.1232 0.00309 0.9575 pauvre pypi 5 0.4326 0.00292 0.9531 mappy pypi 4 0.3863 0.00292 0.9531 rle cran 2 0.2516 0.00287 0.9522 LineagePulse bioconductor 1 0.1232 0.0027 0.9474 Open in a new tab Table 5. Software quadrants, weighted graph (LCC component only). Software Ecosystem Mentions Centrality Count Percentile Value Percentile “Pasteur” PRISMA cran 15797 0.9996 0.25756 1 DESeq2 bioconductor 21257 0.9999 0.17492 0.9995 SAM cran 13048 0.999 0.21116 0.9999 limma bioconductor 13451 0.9991 0.21074 0.9998 ggplot2 cran 14441 0.9994 0.1482 0.9994 PERMANOVA cran 21960 1 0.09735 0.9988 edgeR bioconductor 12923 0.9989 0.20643 0.9996 WGCNA cran 13915 0.9993 0.10402 0.999 pymol pypi 14604 0.9995 0.09333 0.9986 tophat pypi 7642 0.9984 0.14773 0.9993 vegan cran 10012 0.9986 0.10357 0.9989 lme4 cran 10510 0.9988 0.07485 0.9983 .......... scipy pypi 1318 0.99 0.01525 0.9851 numpy pypi 962 0.9854 0.01662 0.9863 .......... napari pypi 11 0.5351 0.00273 0.2816 Popular GSVA bioconductor 2937 0.996 0.00272 0.1185 MAST bioconductor 2091 0.9937 0.00272 0.1185 GSA cran 844 0.9833 0.00272 0.2167 FSA cran 499 0.9724 0.00272 0.148 Boruta cran 441 0.969 0.00272 0.0923 MixSIAR cran 377 0.9639 0.00276 0.3878 AntWeb cran 282 0.9516 0.00276 0.3878 emoji pypi 161 0.9215 0.00276 0.397 searchlight pypi 137 0.9091 0.00273 0.254 pathfindR cran 133 0.9073 0.00275 0.3677 ChIPXpress bioconductor 125 0.9018 0.00276 0.3765 pypet pypi 125 0.9018 0.00272 0 “Nebraska” calculate_expression pypi 3 0.2982 0.0375 0.9944 zscore pypi 5 0.3883 0.01705 0.9865 setuptools pypi 6 0.4215 0.01146 0.978 newick pypi 9 0.4957 0.00806 0.9649 xtick pypi 1 0.1305 0.00618 0.9469 nanolyse pypi 9 0.4957 0.00595 0.9443 splatter pypi 8 0.4742 0.00584 0.9422 pauvre pypi 5 0.3883 0.00584 0.9413 mappy pypi 4 0.3474 0.00584 0.9413 rle cran 2 0.232 0.00574 0.9402 scone pypi 8 0.4742 0.00574 0.9398 LineagePulse bioconductor 1 0.1305 0.0054 0.9344 Open in a new tab “Pasteur”: the highest sum of mentions and centrality percentile. To provide a “sanity check”, we added the data for three important packages: numpy , scipy , and napari . Popular: the highest mentions percentile while centrality percentile is less than 0.5. “Nebraska”: the highest centrality percentile while mentions percentile is less than 0.5. These metrics calculated for all packages are available as CSV files in the repository [ 22 ] ( https://github.com/borisveytsman/SoftwareImpactHackathon2023_Tracing_dependencies/tree/main/data ). 4 Discussion Generally, the results verify our intuition that the majority of software packages will have a low Katz centrality and relatively few mentions. The outliers are then the most interesting packages, because they are central and often mentioned (“Pasteur”), or because their centrality outweighs their mentions (“Nebraska”), or vice versa (popular packages). The absence of cycles in the network of open-source software packages used in our sample of biomedical science is quite interesting. It suggests a more robust design structure in the universe of scientific software than in the general software world. The lack of cycles also improves our analysis with Katz centrality, as the absence of loops excludes feedback mechanisms that can artificially inflate the centrality of packages on a loop. Packages with low mentions and high Katz centrality are therefore even more critical, as they represent essential dependencies of packages that enable large volumes of research without receiving direct attention. These are the “Nebraskan” packages referenced by [ 15 ] that we set out to find. Fig 3 and Tables 3–5 provide examples of these packages. As expected, most of these software packages are libraries used by other packages: vctrs [ 43 ] (a package for type manipulation in R), withr [ 44 ] (a set of tools for safely running functions that change global variables), isoband [ 45 ] (a package to generate isolines from gridded data), pauvre [ 46 ] (a package for plotting Oxford Nanopore data), newick [ 47 ] (read and write Newick data formatted files), setuptools [ 48 ] (a packaging library for Python projects). It is interesting that the prominent R “Nebraska” packages belong to the tidyverse system [ 49 ], and thus are co-authored by Hadley Wickham. To check the results, we included in the tables three popular Python packages: scipy [ 50 ], numpy [ 51 ], and napari [ 52 ]. We see that scipy and numpy are in high percentiles of package by centrality and mentions for all three versions of network studied. The newer and more specialized package napari is lower in the ratings, but still is above 50% percentile. All this seems reasonable. Similarly we were pleased to note that the analysis identified the tifffile [ 53 ] package as part of the Nebraska quadrant and quick inspection of the online materials confirm this as a high quality, widely used package with one person development team, albeit in California and not physically in Nebraska. The differences between the unweighted and weighted versions of our analysis, shown in Fig 3 , can help us parse different subtypes of these outliers. For example, the velvet [ 54 ] package is central solely because it is mentioned by one highly cited paper (and dozens of less highly cited ones). Such characteristics do not make a package central in the unweighted version of the analysis, since citation counts are ignored there. Central packages under this representation, therefore, also need to have other packages depend on them and cannot solely rely on mentions from influential papers. It is interesting that cran and bioconductor dominate the “Nebraska” quadrant for the unweighted graph, while for the weighted graph, pypi is overrepresented. The reason for this difference may warrant additional research. Altogether, our paper tries to quantify impact as importance and to provide a set of metrics behind the intuition. Of course, this is by itself a perilous process: there could be various flavors of both impact and importance, and people may reasonably disagree about the details. Our contribution is to propose a certain quantification, which agrees with our intuitive understanding. In addition to possible problems with the exact definition of importance, our work has a number of limitations. The network analysis is limited by the quality of disambiguation and linking in the CZ Software Mentions dataset (see the discussion in [ 10 ]). Homonyms (different software packages with the same or close names) may change the network statistics, and preliminary research points to considerable potential for incorrect linking in the CZ Software Mentions dataset due to homonyms [ 6 ]. Notably, this affects the packages velvet and tophat (see Fig 2a ), which have been falsely linked to Python packages. velvet was linked to a package that provides signal processing and communications algorithms in Python [ 36 ]. Meanwhile, all of the mentions of “velvet” point to different versions of algorithms for de novo short-read assembly using de Bruijn graphs in genomics, as described in [ 55 ] and implemented mostly in C [ 54 ]. tophat was linked to a package that provides a framework for collaborative and multiplayer mobile applications [ 56 ]. However, over 99% of mentions point to different software called TopHat , a splice junction mapper for RNA-Seq reads, implemented mostly in C ++ [ 57 , 58 ], and the remainder to the Top-Hat transform algorithm used for baseline subtraction in image processing. While the linking errors do not change the centrality values and our interpretation of them – both rely on disambiguation, not linking – they led to mislabeling of different, mostly non-PyPI packages as PyPI packages for the visualization in Fig 2 . Sometimes an open source package also shares a name with a proprietary software (e.g., PRISM, PACE), which leads to confusion and incorrect mention counts. Nonetheless, we think that this contribution is useful, both as a demonstration of this approach and because our analysis is reproducible. This means that limitations can be addressed in improvements over time. For example, as datasets emerge with improvements to disambiguation these datasets can be incorporated into the analysis and comparisons or updates produced. Our approach is limited to the open-source packages available in one of the chosen ecosystems. This excludes proprietary software like Excel or GraphPad Prism [ 59 ], and open-source non-package software like Gephi [ 60 ] (a compiled application). In addition, our work excludes software projects such as BLAS [ 61 ], LAPACK [ 62 ] or the Linux Kernel itself. These packages are critically important but are often lower-level system dependencies that may not be clearly cited or referenced in an article’s text or provided in a software library’s dependency list. Without such references or dependencies, it is hard to detect their specific utilization. Without authors including references to lower-level software libraries in their articles, and without explicit dependency on lower-level libraries from mentioned software, we simply cannot include such software in our networks. The number of missing packages from the ecosystems investigated may be estimated from Table 14 of [ 10 ], which estimates the coverage for GitHub at 64.39%, the coverage for CRAN at 8.36%, the coverage for PyPI at 5.86%, and the coverage for Bioconductor at 3.23%. GitHub coverage can be used as a proxy for the share of the open source software: while some packages—perhaps growing in number—are not hosted by GitHub, their fraction remains low [ 51 ]. If we assume that each package exists only in one ecosystem (there are packages present in several, but we neglect this effect), the combined coverage by CRAN, PyPI, and Bioconductor is 17.45%, i.e., between a quarter and one third of the open source packages. It is very likely that each of these non-package pieces of software do depend on packages in our focus ecosystems. For open-source non-package software, it may be feasible to identify source repository URLs and then identify packages depended on by the software. That would need to be done by directly inspecting the source code, possibly using Software Bill of Materials (SBOM) tools [ 12 ]. On the other hand, some non-package software does not have source code available (including proprietary GUI software such as SPSS, and cloud-based services). Nonetheless, these pieces of code may very well be based on packages in our focal ecosystem (especially if the latter use non-restrictive licenses such as BSD, MIT). In the longer term, there is a possibility that the requirement to provide Software Bills of Materials (SBOMs) for U.S. federal government purchasers will give insight into the packages on which proprietary software relies [ 63 ]. In our dependency network, we always used the latest version of any package. In reality, dependencies change between the versions. A study of development logs often reveals messages like “deleted the dependence on XYZ” or “added the dependence on ABC”. These changes often make the work of package management software very difficult. A software dependency graph is a living network, constantly evolving, with links added and deleted. In this study, we effectively collapse the time, making an (imperfect) snapshot of the long movie. A future improvement would be to use each package ecosystem’s specific dependency resolution algorithm to compute the full transitive dependency tree for each mentioned software package. An even further extension would be to attempt to do this for the version of the package most likely used by a specific publication, based on aligning publication date with the current version used at that time. Our method captures the off-the-shelf software packages used for each article. It does not capture ad hoc software written specifically for the given paper, which may be contained in the code accompanying the papers. This code may load software libraries and thus add software dependencies that are absent from our graph. The dependencies reflected in the package metadata do not necessarily reflect the actual usage of the software. A library may be loaded, but not used for the actual execution by a given paper. A more reliable method would be to reproduce the analysis done in each paper and capture the actual library calls—which is probably prohibitively difficult for the number of papers covered. Many Python and R packages ultimately depend on libraries outside their ecosystems (most often C and Fortran libraries). These dependencies are not easily captured by the package management infrastructure and thus are outside of our analysis. Our dependency network does not include the concept of alternatives, when package A depends on the functionality that can be provided either by package B 1 , or package B 2 . Some package managers like Debian’s apt have the possibility of specifying alternatives, but this is not a common feature of package managers. Different communities place different emphasis on infrastructure elements such as tests. This may lead to the over-representation of testing infrastructure packages in some cases, for example, in the Python ecosystem. We do not try to attempt to decide which elements of the infrastructure are “actually important”; instead, we rely on the judgment of the community of users and developers while recognizing that such judgment may be different across the landscape. Our current approach does not filter or subset the network into different time periods (i.e., by year) and as such, some of the software identified as central to our network such as velvet and tophat are perhaps historical artifacts that have not been updated in many years. These examples demonstrate that future analysis of software dependency networks should consider temporal data to understand how particular software projects (and more broadly, computational methods) rise in adoption and then decline as new methods become available. There are different software package repositories and ecosystems for Python and R with greatly overlapping dependency graphs. We used PyPI, CRAN and Bioconductor for this study. Alternative repositories that provide packages for Python and R, such as Conda channels (conda-forge, bioconda, etc.), are also worth investigating as there may be some software projects distributed solely through Conda channels. Conda provides alternate packaging standards from PyPI and CRAN and as such, over a large distribution, the same set of packages sourced from PyPI may have a slightly different dependency graph than one sourced from Conda. Conda also has more cross-language dependencies listed and may allow deeper traversal to find infrastructural libraries like BLAS; other improvements could include using PyPI Wheels or HomeBrew dependency systems. 5 Conclusions and future work In this work, we investigated the network of software packages used in the biomedical literature available through PubMed Central. We demonstrated that these packages follow the structure of “Stokes’ diagram” (famous for “Pasteur’s quadrant”), with some packages highly visible to the end users, and some packages less visible, but important in the network due to their dependencies. We discussed the use of Katz centrality in discovering the important packages and found examples of such packages in the biomedical field. These findings and insights, as well as the underlying methodology, can be used to identify critical, but low-visibility, open-source scientific software in need of targeted funding and support. Of course, there are many ways to extend this work. It would be interesting, for instance, to analyze common workflows for different disciplines, perhaps using co-occurrences of mentions, and map them into the dependency network. This might help to discover packages important for specific sub-fields of biomedical sciences. Similarly, adding temporal dependencies to our graph may help to discover and predict development trends. Additionally, individual dependencies may differ in the actual impact that they have on the dependent software (i.e., a dependency that is only minimally utilized by the downstream software itself may be given less weight). Developing and applying a metric that describes these differences in impact would make it possible to refine our approach to quantifying the importance of research software dependencies. Data Availability Name: SoftwareImpactHackathon2023_Tracing_dependencies https://github.com/borisveytsman/SoftwareImpactHackathon2023_Tracing_dependencies doi: 10.5281/zenodo.10031223 . Funding Statement The authors are grateful to the Chan Zuckerberg Initiative for making this research possible and for sponsoring the Mapping the Impact of Research Software in Science hackathon. L.H.-D. acknowledges support from the National Institute of General Medical Sciences under the 2P20GM125498 Centers of Biomedical Research Excellence Award and from Google Open Source through the Open-Source Complex Ecosystems And Networks (OCEAN) project. D.M. acknowledges support from the German Research Council (DFG) through the MaRDI project (460135501) as well as through the BASE4NFDI/KGI4NFDI project (521453681). J.F.P. acknowledges support from CNPq and FAPERJ (E-26/210.478/2024). J.H. acknowledges that this material is based upon work supported by the US National Science Foundation under Grant Nos. SMA- 1064209 (SciSIP), OCI-0943168 (VOSS) and ACI-1453548 (CAREER) as well as support from the Alfred P. Sloan Foundation, Grant/Award Number: 2016-7209 for the creation of the SoftCite dataset. The Chan Zuckerberg Initiative and the acknowledged funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. References 1. Howison J, Herbsleb JD. Scientific software production: incentives and collaboration. In: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, 2011. 513–22. 10.1145/1958824.1958904 [ DOI ] [ Google Scholar ] 2. Howison J, Deelman E, McLennan MJ, Ferreira da Silva R, Herbsleb JD. Understanding the scientific software ecosystem and its impact: current and future measures. Res Eval. 2015;24(4):454–70. doi: 10.1093/reseval/rvv014 [ DOI ] [ Google Scholar ] 3. Singh Chawla D. The unsung heroes of scientific software. Nature. 2016;529(7584):115–6. doi: 10.1038/529115a [ DOI ] [ PubMed ] [ Google Scholar ] 4. Howison J, Bullard J. Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature. JASIST. 2016;67(9):2137–55. doi: 10.1002/asi.23538 [ DOI ] [ Google Scholar ] 5. Knowles R, Mateen BA, Yehudi Y. We need to talk about the lack of investment in digital research infrastructure. Nat Comput Sci. 2021;1(3):169–71. doi: 10.1038/s43588-021-00048-5 [ DOI ] [ PubMed ] [ Google Scholar ] 6. Druskat S, Hong NPC, Buzzard S, Konovalov O, Kornek P. Don’t mention it: an approach to assess challenges to using software mentions for citation and discoverability research. arXiv. 2024. doi: 10.48550/arXiv.2402.14602 [ DOI ] [ Google Scholar ] 7. Schindler D, Bensmann F, Dietze S, Krüger F. SoMeSci- a 5 star open data gold standard knowledge graph of software mentions in scientific articles. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 2021. 4574–83. 10.1145/3459637.3482017 [ DOI ] [ Google Scholar ] 8. Du C, Cohoon J, Lopez P, Howison J. SoftCite dataset: a dataset of software mentions in biomedical and economic research publications. JASIST. 2021;72(7):870–84. doi: 10.1002/asi.24454 [ DOI ] [ Google Scholar ] 9. Istrate AM, Veytsman B, Li D, Taraborelli D, Torkar M, Williams I. CZ Software Mentions: a large dataset of software mentions in the biomedical literature. 2022. https://datadryad.org/stash/dataset/doi:10.5061/dryad.6wwpzgn2c 10. Istrate AM, Li D, Taraborelli D, Torkar M, Veytsman B, Williams I. A large dataset of software mentions in the biomedical literature. arXiv. 2022. doi: 10.48550/ARXIV.2209.00693 [ DOI ] [ Google Scholar ] 11. Bogart C, Howison J, Herbsleb J. Guiding development work across a software ecosystem by visualizing usage data. arXiv e-prints. 2020. doi: 10.48550/arXiv.2012.05987 [ DOI ] [ Google Scholar ] 12. Hatta M. The Nebraska problem in open source software development. Annal Business Admin Sci. 2022;21(5):91–102. doi: 10.7880/abas.0220914a [ DOI ] [ Google Scholar ] 13. Goodin D. What we know about the xz utils backdoor that almost infected the world. Ars Technica. 2024. https://arstechnica.com/security/2024/04/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world/ 14. Samuel S, Mietchen D. Computational reproducibility of Jupyter notebooks from biomedical publications. Gigascience. 2024;13:giad113. doi: 10.1093/gigascience/giad113 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 15. Munroe RP. Dependency. 2020. https://xkcd.com/2347/ 16. Katz DS. Transitive credit as a means to address social and technological concerns stemming from citation and attribution of digital products. Journal of Open Research Software. 2014. doi: 10.5334/jors.be [ DOI ] [ Google Scholar ] 17. Katz DS, Smith AM. Implementing transitive credit with JSON-LD. arXiv. 2014. doi: 10.48550/arXiv.1407.5117 [ DOI ] [ Google Scholar ] 18. Druskat S, Spaaks JH, Chue Hong N, Haines R, Baker J, Bliven S. Citation File Format. 2021. doi: 10.5281/zenodo.5171937 [ DOI ] [ Google Scholar ] 19. Druskat S. Software and dependencies in research citation graphs. Comput Sci Eng. 2020;22(2):8–21. doi: 10.1109/MCSE.2019.2952840 [ DOI ] [ Google Scholar ] 20. Bogart C, Kästner C, Herbsleb J, Thung F. When and how to make breaking changes: policies and practices in 18 open source software ecosystems. ACM Trans Softw Eng Methodol. 2021;30(4). doi: 10.1145/3447245 [ DOI ] [ Google Scholar ] 21. Stokes DE. Pasteur’s quadrant: basic science and technological innovation. Washington, D. C.: Brookings Institute Press; 1997. [ Google Scholar ] 22. Brown EM, Nesbitt A, Hébert-Dufresne L, Veytsman B, Pimentel JF, Druskat S. Exploring the dependencies of the CZI mentions dataset. 2023. https://github.com/borisveytsman/SoftwareImpactHackathon2023_Tracing_dependencies 23. Nesbitt A. Package and dependency metadata for CZI hackathon: mapping the impact of research software in science. Zenodo; 2023. [ Google Scholar ] 24. Priem J, Piwowar H, Orr R. OpenAlex: a fully-open index of scholarly works, authors, venues, institutions, and concepts. arXiv e-prints. 2022;arXiv:2205.01833. doi: 10.48550/arXiv.2205.01833 [ DOI ] [ Google Scholar ] 25. Brown EM. A Dependency graph for 460,000 papers and their software mentions from the CZI software mentions dataset. 2023. doi: 10.5281/zenodo.10048132 [ DOI ] [ Google Scholar ] 26. GEXF Working Group. GEXF file format. 2009. https://gexf.net/ 27. Borgatti SP, Everett MG. Three perspectives on centrality. the oxford handbook of social networks. Oxford University Press; 2020. 333–51. 10.1093/oxfordhb/9780190251765.013.22 [ DOI ] [ Google Scholar ] 28. Bonacich P. Some unique properties of eigenvector centrality. Soc Netw. 2007;29(4):555–64. doi: 10.1016/J.SOCNET.2007.04.002 [ DOI ] [ Google Scholar ] 29. Brin S, Page L. The anatomy of a large-scale hypertextual Web search engine. Comp Netw ISDN Syst. 1998;30(1–7):107–17. doi: 10.1016/s0169-7552(98)00110-x [ DOI ] [ Google Scholar ] 30. Katz L. A new status index derived from sociometric analysis. Psychometrika. 1953;18(1):39–43. doi: 10.1007/BF02289026 [ DOI ] [ Google Scholar ] 31. Rogers EM. Diffusion of innovations, 5th Edition. Free Press; 2003. [ Google Scholar ] 32. Sun J, Tang J. A survey of models and algorithms for social influence analysis. In: Aggarwal CC, editor. Social network data analytics. Boston, MA: Springer US; 2011. 177–214. [ Google Scholar ] 33. Wickham H. ggplot2. Wiley Interdisciplinary Reviews: Computational Statistics. 2011;3(2):180–5. [ Google Scholar ] 34. Ravikumar P, Lafferty J, Liu H, Wasserman L. Sparse additive models. J Royal Stat Soc Series B: Stat Methodol. 2009;71(5):1009–30. doi: 10.1111/j.1467-9868.2009.00718.x [ DOI ] [ Google Scholar ] 35. Krueger T, Gascon H, Krämer N, Rieck K. Learning stateful models for network honeypots. In: Proceedings of the 5th ACM workshop on Security and artificial intelligence, 2012. 37–48. 10.1145/2381896.2381904 [ DOI ] [ Google Scholar ] 36. Wood S. Velvet. 2015. https://pypi.org/project/velvet 37. DeLano WL. Pymol: An open-source molecular graphics tool. CCP4 Newsl Protein Crystallogr. 2002;40(1):82–92. [ Google Scholar ] 38. Love MI, Huber W, Anders S. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome Biol. 2014;15(12):550. doi: 10.1186/s13059-014-0550-8 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 39. Robinson MD, McCarthy DJ, Smyth GK. edgeR: a Bioconductor package for differential expression analysis of digital gene expression data. Bioinformatics. 2010;26(1):139–40. doi: 10.1093/bioinformatics/btp616 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 40. Smyth GK. Limma: linear models for microarray data. In: Bioinformatics and computational biology solutions using R and Bioconductor. Springer; 2005. 397–420. [ Google Scholar ] 41. Ritchie ME, Phipson B, Wu D, Hu Y, Law CW, Shi W, et al. limma powers differential expression analyses for RNA-sequencing and microarray studies. Nucleic Acids Res. 2015;43(7):e47. doi: 10.1093/nar/gkv007 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 42. Gini C. Concentration and dependency ratios [English translation of the original 1909 paper]. Rivista di Politica Economica. 1997;87:769–89. [ Google Scholar ] 43. Wickham H, Henry L, Vaughan D. vctrs: Vector Helpers. 2023. 44. Hester J, Henry L, Müller K, Ushey K, Wickham H, Chang W. withr: Run code ‘with’ temporarily modified global state. https://CRAN.R-project.org/package=withr 45. Wickham H, Wilke CO, Pedersen TL. Isoband: generate isolines and isobands from regularly spaced elevation grids. 2022. 46. Schultz D, Ebbert M, De Coster W. Pauvre. 2021. https://pypi.org/project/pauvre/ 47. Forkel R. Newick. 2025. https://pypi.org/project/newick/ 48. Python Packaging Authority. setuptools. https://pypi.org/project/setuptools/ 49. Wickham H, Averick M, Bryan J, Chang W, McGowan L, François R, et al. Welcome to the Tidyverse. J O Sourc Softw. 2019;4(43):1686. doi: 10.21105/joss.01686 [ DOI ] [ Google Scholar ] 50. Virtanen P, Gommers R, Oliphant TE, Haberland M, Reddy T, Cournapeau D, et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods. 2020;17(3):261–72. doi: 10.1038/s41592-019-0686-2 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 51. Trujillo MZ, Hébert-Dufresne L, Bagrow J. The penumbra of open source: projects outside of centralized platforms are longer maintained, more academic and more collaborative. EPJ Data Science. 2022;11(1):31. [ Google Scholar ] 52. Napari contributors. Napari: a multi-dimensional image viewer for python. 2019. 53. Gohlke C. Cgohlke/tifffile: v2026.4.11. 2026. 54. Zerbino DR, Foret S, Gurney JM, Slater G, Birney E, Marshall J. Velvet. Software Heritage; 2014. [ Google Scholar ] 55. Zerbino DR, Birney E. Velvet: algorithms for de novo short read assembly using de bruijn graphs. Genome Res. 2008;18(5):821–9. doi: 10.1101/gr.074492.107 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 56. The TopHat developers. TopHat. 2012. https://pypi.org/project/TopHat 57. Trapnell C, Pachter L, Salzberg SL. TopHat: discovering splice junctions with RNA-Seq. Bioinformatics. 2009;25(9):1105–11. doi: 10.1093/bioinformatics/btp120 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 58. Kim D, Pertea G, Trapnell C, Pimentel H, Kelley R, Salzberg SL. TopHat2: accurate alignment of transcriptomes in the presence of insertions, deletions and gene fusions. Genome Biol. 2013;14(4):R36. doi: 10.1186/gb-2013-14-4-r36 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ] 59. Swift ML. GraphPad Prism, data analysis, and scientific graphing. J Chem Inf Comput Sci. 1997;37(2):411–2. doi: 10.1021/ci960402j [ DOI ] [ Google Scholar ] 60. Bastian M, Heymann S, Jacomy M. Gephi: an open source software for exploring and manipulating networks. Int Conf Weblogs Soc Media. 2009;3(1):361–2. doi: 10.1609/icwsm.v3i1.13937 [ DOI ] [ Google Scholar ] 61. Blackford LS, Petitet A, Pozo R, Remington K, Whaley RC, Demmel J. An updated set of basic linear algebra subprograms (BLAS). ACM Transact Mathematical Softw. 2002;28(2):135–51. [ Google Scholar ] 62. Anderson E, Bai Z, Bischof C, Blackford LS, Demmel J, Dongarra J. LAPACK users’ guide. SIAM; 1999. [ Google Scholar ] 63. Howison J, Ram K. Support scientific software infrastructure by requiring SBOMs for federally funded research. 2024. https://fas.org/publication/sboms-hardware/ PLoS Comput Biol. doi: 10.1371/journal.pcbi.1014260.r001 Decision Letter 0 Varun Dutt Author information Copyright and License information Roles Varun Dutt : Academic Editor © 2026 Varun Dutt This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. PMC Copyright notice 4 Feb 2025 -->PCOMPBIOL-D-24-02238 Biomedical Open Source Software: Crucial Packages and Hidden Heroes PLOS Computational Biology Dear Dr. Veytsman, Thank you for submitting your manuscript to PLOS Computational Biology. After careful consideration, we feel that it has merit but does not fully meet PLOS Computational Biology's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript within 60 days Apr 06 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at ploscompbiol@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pcompbiol/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: * A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. This file does not need to include responses to formatting updates and technical items listed in the 'Journal Requirements' section below. * A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. * An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, competing interests statement, or data availability statement, please make these updates within the submission form at the time of resubmission. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter We look forward to receiving your revised manuscript. Kind regards, Varun Dutt, Ph.D Academic Editor PLOS Computational Biology Mark Alber Section Editor PLOS Computational Biology Journal Requirements: 1) Please ensure that the CRediT author contributions listed for every co-author are completed accurately and in full. At this stage, the following Authors/Authors require contributions: Andrew Nesbitt, Boris A Veytsman, Daniel Mietchen, Eva Maxfield Brown, James Howison, João Felipe Pimentel, Laurent Hébert-Dufresne, and Stephan Druskat. Please ensure that the full contributions of each author are acknowledged in the "Add/Edit/Remove Authors" section of our submission form. The list of CRediT author contributions may be found here: https://journals.plos.org/ploscompbiol/s/authorship#loc-author-contributions 2) We ask that a manuscript source file is provided at Revision. Please upload your manuscript file as a .doc, .docx, .rtf or .tex. If you are providing a .tex file, please upload it under the item type u2018LaTeX Source Fileu2019 and leave your .pdf version as the item type u2018Manuscriptu2019. 3) Please provide an Author Summary. This should appear in your manuscript between the Abstract (if applicable) and the Introduction, and should be 150-200 words long. The aim should be to make your findings accessible to a wide audience that includes both scientists and non-scientists. Sample summaries can be found on our website under Submission Guidelines: https://journals.plos.org/ploscompbiol/s/submission-guidelines#loc-parts-of-a-submission 4) Please upload all main figures as separate Figure files in .tif or .eps format. For more information about how to convert and format your figure files please see our guidelines: https://journals.plos.org/ploscompbiol/s/figures 5) Please amend your detailed Financial Disclosure statement. This is published with the article. It must therefore be completed in full sentences and contain the exact wording you wish to be published. 1) State what role the funders took in the study. If the funders had no role in your study, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." 6) Your current Financial Disclosure states, "The authors are grateful to the Chan Zuckerberg Initiative for making this re-search possible and for sponsoring the Mapping the Impact of Research Softwarein Science hackathon. L.H.-D. acknowledges support from the National Insti-tute of General Medical Sciences under the 2P20GM125498 Centers of Biomed-ical Research Excellence Award and from Google Open Source through theOpen-Source Complex Ecosystems And Networks (OCEAN) project. D.M. ac-knowledges support from the German Research Council (DFG) through theMaRDI project (460135501) as well as through the BASE4NFDI/KGI4NFDIproject (521453681). J.F.P. acknowledges support from CNPq and FAPERJ(E-26/210.478/2024)." However, your funding information on the submission form does not indicate any funds. Please ensure that the funders and grant numbers match between the Financial Disclosure field and the Funding Information tab in your submission form. Note that the funders must be provided in the same order in both places as well. Please indicate by return email the full and correct funding information for your study and confirm the order in which funding contributions should appear. Please be sure to indicate whether the funders played any role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. Reviewers' comments: Reviewer's Responses to Questions Reviewer #1: This manuscript by Nesbitt et al. presents a study of the graph of citations between papers and software tools as well as between software tools themselves. The main thesis put forward is that the Katz centrality metric is a good way to measure the importance of papers and software tools and can capture tools that are under-appreciated. While this is interesting, the manuscript is lacking in important details and, in my opinion, fails to provide a convincing argument for this thesis. 1. The authors justify the use of the Katz centrality metric as fulfilling some basic criteria, but other than excluding two related metrics (eigencentrality and PageRank) the argument for Katz centrality is not very well developed, neither from a theoretical (e.g., deriving from a set of axioms) or from a purely empirical perspective. 2. The authors write that "We found that Katz centrality [Katz, 1953] satisfies all our criteria. This approach gives us the attenuation factor β as a free parameter to control the importance of papers..." This requires more detail. The original [Katz, 1953] paper does not mention a parameter β (in fact, it is typeset without the use of Greek letters, and mentions only the attenuation factor a, which today we would typeset as α). The authors should clarify what they mean by this and present the actual formulas they use. Is β the attenuation factor? It does not seem to be the case as it would not be related to the importance of papers (at least in my understanding, but I may be wrong given the lack of detail in the manuscript). 3. The authors acknowledge that the "velvet" is a misassignment. The velvet assembler is hardly under-appreciated (google scholar gives it >11,000 citations: https://genome.cshlp.org/content/18/5/821.short , https://scholar.google.com/scholar?cites=355244555212756650 ). Having had a look at their dataset, it seems that the highly cited paper mentioned in the text appears to be 10.1093/bioinformatics/btu170 (the Trimmomatic paper) which is marked as citing "velvet", but reading that paper, it cites the velvet assembler, not the velvet package on PyPI. Without this example, there are no other examples of under-appreciated tools in the manuscript, which was a centrally motivating factor for the study. In fact, from Figure 3, I would even be tempted to say that citations do a good job of capturing the centrality of tools, although it is also difficult to say because this figure does not provide a lot of detail for the papers with <5000 citations (as the dots overlap and are impossible to distinguish), which constitute the majority of papers. 4. The code is available, but references paths such as `~/Downloads/pruned-network.csv` which are not directly usable. Reviewer #2: Open-source software or libraries used in a scientific context are notoriously seldom cited, if they are cited at all. As highlighted by the authors, the problem of identifying packages crucial to the proper functioning of scientific infrastructure appears even more challenging when considering the breadth of packages used as dependencies hidden to the users of scientific software. How can we, then, as scientists using software everyday, support critical software packages when they cannot be identified so easily? In their manuscript, the authors proposed a method to identify such critical packages by merging information from a large public dataset of software mentions in the biomedical literature (CZI Software Mentions Dataset) and the dependencies trees of the mentioned software into a single network. Because this dataset was born out of data extracted from biomedical papers, their network has the potential to be representative of the open-source biomedical software infrastructure. While this work has a lot of potential, the manuscript suffers from a serious lack of tangible findings that would confirm the validity of the approach. The results consist in a dense graph from which users cannot draw conclusions, and the name of 9-16 packages identified as the most “central” given the authors’ metrics. There is little to no investigation of these packages, which are in any case far too few to allow the authors to conclude anything about the approach. The only package for which some scrutiny has been applied is held as a new-found critical package that is hardly mentioned in the literature (velvet), but the authors themselves cast shadows on the veracity of that finding since it only appears because its original method is historically important (see first major comment). The authors identify and clearly state many interesting limitations to their approach (which is laudable), whether from a consequence of imperfect data or of their metrics itself. But they do not attempt to quantify how much these limitations are impacting their measure through validation of the prominent packages or known/expected packages (see major comments 2 and 3). Finally, I find the manuscript difficult to parse and suggest several axis of improvements, as well as additional information that would make it easier for readers to judge the method. In its current state, I do not think that the manuscript can be published. I do believe, however, that both the dataset and the approach have the potential to yield interesting results should informative data analysis be applied. ## Major comments 1. One of the main result (if not the only one that is actually discussed) of the paper is the identification of “velvet” as a “Nebraskan” package. Two aspects of that result bother me: firstly in the way it is introduced, and secondly in what it actually says about the method. In the discussion section, “velvet” is first introduced as the only (for lack of another one mentioned) “Nebraskan” package, and the authors then state that the package disappears as soon as the paper citation counts are not taken into account (unweighted graph). In another section of the discussion, they double down on by highlighting again that it is mostly, if not only, central because of one highly cited paper (presumably the original publication). In my opinion, this totally undermines the method, which aims at identifying specifically “Nebraskan” packages. Indeed, the only identified “Nebraskan” package only pops in the analysis because the original method is highly cited (11000 citations)... While I cannot exclude that the package is still used (there are a few issues from 2022 in the Github repository), it clearly is not a dependency of other software (from the author’s own analysis), it is 17 years old (last commit 11 years ago) and the paper is probably cited because of its historical importance. A quick question to genomics specialist made it clear that newer tools are more useful nowadays: Abyss, PacBio, Nanopore. Genomics being such an active field, I have a hard time believing that velvet is still crucial to the scientific infrastructure. To justify the power of the method, better (so far unknown) examples should be highlighted to the readership. 2. As exemplified by velvet, the manuscript lacks analysis validating the method. Only 9 (fig 2, or 16 in fig 3) packages are mentioned, of which a single one is investigated. I strongly suggest examining closely a non-negligible number of packages (e.g. 50-100, number should be adjusted based on how much work it ends up being, while maintaining enough statistics to be able to draw conclusions) that fall into the “Nebraskan” and “Pasteur” categories (given ad-hoc thresholds on mentions and centrality) and identifying the amount of “potential impostors” (e.g. like velvet) in that pool. The authors should go through the list of identified packages and compile metrics and metadata information to validate whether the package could belong to the assigned category: - Is it an homonym? - Centrality in weighted vs unweighted graphs - Number of mentions in the CZI dataset - Download statistics (which are available in all three package ecosystems investigated) - Average citation / paper with edges to this package, highest citation count for paper with edge (aka guarding against velvet-like packages) - Date of the latest commit in Github / Date of the latest package version - Rate of updates (e.g. average commits / months in the last year or versions / year in the last 5 years) - Number of Github stars if applicable - … None of these are perfect proxy metrics (e.g. download statistics will not be comparable between ecosystems, CI/CD can skew the downloads etc.), but they can all inform to a certain extent on whether there might be issues with the particular software packages. Additionally, assigning a software category to these packages would allow refining the analysis and help understand what type of packages the method is able to classify. I don’t know if ontologies already exist, but categories could be along the lines of: specific analysis (BLAST, velvet), generic computation (scikit-learn), infrastructure / data structures (numpy, pytest, pandas), rendering (ggplot2, matplotlib). The results could be show in tables, for instance per categories and per ecosystem, before being discussed. 3. An additional important experiment to validate the method is to build a list of known packages and inspect where they appear in the [Mentions, Centrality] graph. While the authors correctly state that different fields will assign importance to different packages, and such importance will indeed reflect mostly direct use of package (as opposed to usage through dependencies), this does not mean that one should not investigate what happens to clearly important scientific software package. Such a list can easily be built by surveying colleagues or asking in forums (biostars and image.sc already cover a large range of biomedical applications, missing mainly the medical fields). The authors could make sure that they are building a list that covers examples in categories spanning application of a particular analysis, generic analysis package, data i/o, rendering or more infrastructure ones. Examples that come to mind are for instance dplyr (I am limited in my knowledge of R), xgboost, pandas, matplotlib, scipy, scikit-learn, dask, pytorch, ggplot2 (obviously that one is already clearly identified in the current analysis, although it is assumed that the readers are familiar with it and it is not discussed). The centrality and mentions measures could be reported for these packages per ecosystem and discussed in comparisons to those identified via thresholding the metrics (Nebraskan and Pasteur), as well as some of the other metrics. 4. I find the manuscript difficult to read for multiple reasons: - The Stokes-like diagram is central in the introduction, without describing what a “Pasteur” package is, forcing readers to google the Stokes diagram. “Nebraskan” and “Pasteur” are mentioned in the text a lot, without exemplifying it in the result figures. - Mentioned software packages should be identified with references and links to the package as well as with a short description of what they are. Otherwise, readers are forced to investigate on their own what these packages are (e.g. I had to google velvet, and until I read the discussion I had no idea why this package was picked up by the analysis at all). - Discussion about the packages are done piece-wise in multiple places and make it difficult to follow the logic regarding the package (velvet). - A substantial part of the discussion is actually results (metrics for velvet, appearance and its disappearance in the different graphs, package coverage). 5. The authors state that they are using the CZI Software Mentions Dataset in the introduction, but do not describe it. Readers have to wait for the discussion to learn about potential pitfalls (cluster of names, homonyms) that may impede the analysis. In addition, the authors should add a few numbers to highlight the importance of the dataset: e.g. number of papers and number of software in the dataset. 6. As opposed to what is stated in multiple places (e.g. point 9 in the discussion) in the paper, PyPI is not a package manager, but a package repository (same for CRAN?). pip/conda are indeed package managers for the Python ecosystem. 7. First paragraph introduction: “This is true not only for the sciences”, prefer the term “natural sciences” to not appear derogatory to sociology. 8. Paragraph 2 and 4 in introduction repeat each other. 9. The authors state that “We also found less central packages mentioned heavily by specific communities (e.g. PRISMA in CRAN […]”, but it seems to me that PRISMA is the most central package identified by their method (Fig 3) solely based on the metrics. 10. The authors should mention in the result sections how many software packages are identified through the dependency trees, that is the total number of software packages in their graph. 11. The result section or the supplementary materials should provide distributions of centrality and of mentions, as well as averages. A comparison of how the distribution of centrality measures changes between unweighted and weighted would be interesting. 12. Could the authors discuss the largest connected component? How do they interpret it? How come it seems to be cross-ecosystem? If a paper mentions both python and R package, does that make their dependency tree a single connected component? 13. I tried running the Github example notebooks, but these failed due to json errors or missing json (e.g. pypi_with_mentions.ndjson). I have not investigated further. I would suggest making sure that the notebooks run, and that the README explains clearly how to get the data, run the analysis and explore the results in order to benefit the readership! ## Minor comments 1. It would be nice to add explanations about how Katz centrality is calculated in your analysis in the supplementary materials of the paper. 2. The authors noted that “we found that roughly 10% of software packages are part of dependency loops”. I am a bit confused by this, do they expect these to be working software? Can such software build in the R and Python world? 3. When the authors state that “It suggests a more robust design structure in the universe of scientific software than in the general software world.”, I would precise “open source software world” as company code / closed source software are often better organized due to internal constraints. There are other mentions of this general software world. 4. When the authors state that “The lack of cycles also improves our analysis with Katz centrality, as it excludes feedback mechanisms that can artificially inflate the centrality of packages on a loop”, do they mean that the lack of cycles exclude feedback mechanisms? This makes sense to me, but on the first read I had the impression that the Katz centrality calculation excluded feedback mechanisms. 5. It is hinted at, but worth insisting on for the readership that, at a time where research institutions have a hard time battling constant cybersecurity targeting, open source software are potential entry ways for malicious actors (similar to the XZ Utils case). Reviewer #3: This paper takes steps to tackle an important and timely issue in bioinformatics, to credit the hidden labour of key software resources and to drive targeted sustainability efforts for key but invisible software packages. Software is the life blood of data driven biology research. Tractably building a credible evidence base in the complex network of modern software is fundamental to these goals. The paper is well written and balanced, reporting an early effort using the CZI software mentions dataset and the Ecosyste.ms dataset. It is balanced because the datasets have a number of limitations, the analysis makes a number of assumptions and there are fundamental issues in the way software is packaged and deemed important that make this an exploratory exercise. The authors make these limitations explicit, effectively laying down a roadmap of challenges for future work and other researchers to take up. Given that “perfect is the enemy of good” these challenges should not hold back the effort. The network analysis and weighting choices are convincingly presented and chimes with intuition. • It would have been interesting to approach leaders in the CRAN, PyPi and Bioconductor community for their views in the results - are there surprises or is it confirming intuition? • The lack of dependency cycles is suggested as robust design - is it just less complexity or more homogeneity in the scientific software world? Or less legacy? Or smaller well connected development communities? • Figure 3 the weighted complete graph and the LCC cluster pattern is identical (as could be expected) so it’s unclear what the point is of showing this is. • Figure 2 - apart from the labelled packages the rest of the figure is tricky to interpret. Unless it is further discussed its value is not clear • The paper focus was in three packages in each ecosystem. What about the next 10? • The bioscience literature community is interesting - there are dedicated journals for bioinformatics and others for scientific discovery that may mention the immediate software. Are there any differences in the citation (and hence weighting) pattern between different classes of journal ? As this is a first step a future work paragraph on how the community might prepare cleaner datasets and further analysis to address limitations of the work in order to fulfil the ultimate credit and funding goals ********** Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes Reviewer #3: Yes ********** PLOS authors have the option to publish the peer review history of their article ( what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy . Reviewer #1: No Reviewer #2: No Reviewer #3: No [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] Figure resubmission: While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step. If there are other versions of figure files still present in your submission file inventory at resubmission, please replace them with the PACE-processed versions. Reproducibility: To enhance the reproducibility of your results, we recommend that authors of applicable studies deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols --> PLoS Comput Biol. 2026 May 13;22(5):e1014260. doi: 10.1371/journal.pcbi.1014260.r002 Author response to Decision Letter 1 Article notes Copyright and License information Collection date 2026 May. PMC Copyright notice 20 Jun 2025 Attachment Submitted filename: rebuttal.pdf pcbi.1014260.s002.pdf (161.7KB, pdf) PLoS Comput Biol. doi: 10.1371/journal.pcbi.1014260.r003 Decision Letter 1 Varun Dutt Author information Copyright and License information Roles Varun Dutt : Academic Editor © 2026 Varun Dutt This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. PMC Copyright notice 14 Aug 2025 -->PCOMPBIOL-D-24-02238R1 Biomedical open source software: Crucial packages and hidden heroes PLOS Computational Biology Dear Dr. Howison, Thank you for submitting your manuscript to PLOS Computational Biology. After careful consideration, we feel that it has merit but does not fully meet PLOS Computational Biology's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript within 60 days Oct 14 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at ploscompbiol@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pcompbiol/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: * A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. This file does not need to include responses to formatting updates and technical items listed in the 'Journal Requirements' section below. * A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. * An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, competing interests statement, or data availability statement, please make these updates within the submission form at the time of resubmission. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter We look forward to receiving your revised manuscript. Kind regards, Varun Dutt, Ph.D Academic Editor PLOS Computational Biology Mark Alber Section Editor PLOS Computational Biology Journal Requirements: 1) Please ensure that the CRediT author contributions listed for every co-author are completed accurately and in full. At this stage, the following Authors/Authors require contributions: Eva Maxfield Brown, Stephan Druskat, Laurent Hébert-Dufresne, James Howison, Andrew Nesbitt, João Felipe Pimentel, Daniel Mietchen, and Boris A Veytsman. Please ensure that the full contributions of each author are acknowledged in the "Add/Edit/Remove Authors" section of our submission form. The list of CRediT author contributions may be found here: https://journals.plos.org/ploscompbiol/s/authorship#loc-author-contributions 2) The file inventory includes files for Figures 2a, and 2b. We would recommend either combining these into a single Figure 2.tiff file with separate internal panels, or renumbering them as individual figures, as we are not able to publish multiple components of a single figure as separate files. Note: If the reviewer comments include a recommendation to cite specific previously published works, please review and evaluate these publications to determine whether they are relevant and should be cited. There is no requirement to cite these works unless the editor has indicated otherwise. Reviewers' comments: Reviewer's Responses to Questions Reviewer #4: In this article, the authors combine two metrics (popularity and centrality) to identify specific classes of biomedical open source software. The authors rightly point out that software maintenance and funding is crucial to modern science. They add that their approach could be used to highlight the importance of some software packages and help their recognition and funding. The authors use software from three ecosystems. PyPI, Cran, and Bioconductor. This covers a lot of the bioinformatics / -omics tools, but does not cover much of the imaging software used by other part of the community. While this is a limitation, I think it is ok to start with a subset of biomedical research software and let other communities apply the method on their own field. The authors provide the software to do so. Disclaimer : while I am aware that the paper has been reviewed before, I was not in the first set of reviewers. The reviewers already covered important points. While this is an interesting proof of concept, It is hard to estimate this work because, as any metric, this metric is largely arbitrary. This does not make it uninsightful, but I am not currently convinced. I have some questions & concerns before being able to fully assess the manuscript : - Elephants in the room My main surprise is to not see cornerstone libraries such as Blas & Lapack, or even higher level libraries such as numpy / scikit learn appear in this graph (while scikit-learn appears in the mentions dataset). In a way, there is nothing more central than those libraries ; as far as I know, most python scientific libraries will depend on numpy, that uses Blas & Lapack. I am surprised that they do not show up in the analysis. Why is it so ? It is hard for me to trust the results knowing that. - Cutoff Where should the analysis stop ? In a way the code of the python/R interpreter, or of clang/gcc, is probably the most critical to this. Or even the windows OS and Linux Kernel ? Is there a cutoff somewhere ? - Anaconda vs Pypi Anaconda offers a very structured model for dependencies and distributing through conda-forge is much stricter on the dependencies than PyPI. Could looseness of the PyPi requirement system explain why obvious packages such as numpy/scipy/scikit lean do not appear ? - The need for a metric : Unfortunately science is very much metric-driven. While I believe the authors propose this in good faith, metrics come at a price as they can become decision tools (which is implied by the authors when they mention the importance of recognition for funding) ; then often the metric itself is optimized, instead of what it should measure(Goodhart's law). I nontheless think that this work is interesting out of curiosity. - History vs current situation As the authors mention, software is a living environment. Some libraries will disappear and others will appear. It seems that the author's analysis give a lot of weight on history, as evidenced by velvet, but also tophat. Tophat has not been updated for many years, nor is being actively forked recently. Is it still actually a cornerstone ? More generally, how do you distinguish history from the current situation ? Reviewer #5: Biomedical open source software: Crucial packages and hidden heroes is well-written and interesting. It addresses a very important topic of recognition and credit/incentives for open-source scientific software used—and frequently developed in—the academic setting. The traditional transactional nature of academic collaboration and authorship does not capture well software development efforts, let alone software maintenance efforts. Package and library dependencies in particular are easily taken for granted and their developers and maintainers are unsung heroes. In particular, researchers may rarely mention how they loaded data or performed basic numerical calculations. Thus, the goal of this paper to develop robust methods and metrics that can account for their role is very commendable and very important. The traditional publishing side of academia has—for better or worse—the ubiquitous impact factors and h-indexes. Similar metrics for software contributions could help legitimize scientific software development efforts, particularly for early career researchers. I think these issues will only be further magnified both in the field of computational biology, but also in other scientific areas that may not been traditionally “computational”, with the growth of machine learning and challenges of big data. To my knowledge, the approach here using Katz centrality is novel, not having been applied to the topic at hand. However, it is well motivated and explained in the text. While the dataset used is somewhat out of date at this point, this is not a major limitation, because the method should still provide insight into the dataset and then in the future be applied to the latest data. As a result, the work is of significant importance to the field and well suited for publication. Major comments I’m not very familiar with CRAN and Bioconductor, but I am quite familiar with the scientific Python ecosystem. As a result, I must say that I am quite puzzled and somewhat troubled by the lack of core packages like numpy, scipy, and matplotlib figuring anywhere in the results. I fully expected to see them in the “Pasteur” quadrant. Just taking numpy as an example, pymol, a “Pasteur” package, has numpy as a dependency, as does pace, a “Popular” package. Whether a result of the specifics of the underlying data set or some aspect of the methodology, I think it’s important that a package like numpy be accounted for somehow/somewhere in the text. Was it is simply considered a far outlier and as such discarded? Is it somehow a function of the age of the dataset (I have a hard time believing this)? With that said, I was impressed that the package tifffile was identified as a “Nebraska” package! It indeed is the most powerful package for loading tiff files in Python (that I know of), but libraries used to load data, like images, are unlikely to be mentioned in a publication. Further, tifffile is literally developed and maintained by a single person, although I believe they are in California and not Nebraska. Minor comments - Page 3, The legend of Figure 1: it would be best to also have “Nebraska” and “Pasteur” quadrants briefly explained in the legend, to make the overall figure able to stand on it’s own. “Nebraska” in particular is quite a logical leap, even for those familiar with the XKCD cartoon. - Page 3, “Nebraska” packages: it would be nice to connect back to the XKCD cartoon as mentioned earlier. In the earlier context on Page 2, the emphasis is on “thanklessly” and I had glossed over the “Nebraska” part of the quote. Thus, even as someone familiar with the cartoon, I was briefly confused by “Nebraska” on page 3. Alternately, consider on Page 2 to say that the term “Nebraska” will be used for such packages/libraries. - Page 5, Results, a number of packages/libraries are mentioned here, but no references are provided, which is a bit ironic given the nature of the work. Some packages/libraries do get references later on, e.g. vctrs, reference 34, but I think all should be cited either in the text or in a table. This would also help fully disambiguate packages/libraries. For example, `tophat` on PyPI is a mobile app framework from 2012, so I have some doubt it that’s really the “Pasteur” package—perhaps it’s `top-hat` a “Recommendation system in TensorFlow”? - Page 5, Results, “(a) unweighted graph, (b) weighted complete graph, and (c) the largest connected components of the weighted graph” — it would be helpful to clarify the differences between these three approaches. There is some interpretation of the differences in the Discussion, but here where these 3 models are first mentioned it would be helpful to clarify the differences in approach. ********** Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #4: None Reviewer #5: Yes ********** PLOS authors have the option to publish the peer review history of their article ( what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy . Reviewer #4: No Reviewer #5: No [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] Figure resubmission: While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step. If there are other versions of figure files still present in your submission file inventory at resubmission, please replace them with the PACE-processed versions. Reproducibility: To enhance the reproducibility of your results, we recommend that authors of applicable studies deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols--> PLoS Comput Biol. 2026 May 13;22(5):e1014260. doi: 10.1371/journal.pcbi.1014260.r004 Author response to Decision Letter 2 Article notes Copyright and License information Collection date 2026 May. PMC Copyright notice 6 Nov 2025 Attachment Submitted filename: ResponseToReviewers.pdf pcbi.1014260.s003.pdf (58.9KB, pdf) PLoS Comput Biol. doi: 10.1371/journal.pcbi.1014260.r005 Decision Letter 2 Varun Dutt Author information Copyright and License information Roles Varun Dutt : Academic Editor © 2026 Varun Dutt This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. PMC Copyright notice 12 Feb 2026 -->PCOMPBIOL-D-24-02238R2 Biomedical open source software: Crucial packages and hidden heroes PLOS Computational Biology Dear Dr. Howison, Thank you for submitting your manuscript to PLOS Computational Biology. After careful consideration, we feel that it has merit but does not fully meet PLOS Computational Biology's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Apr 14 2026 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at ploscompbiol@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pcompbiol/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: * A letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. This file does not need to include responses to formatting updates and technical items listed in the 'Journal Requirements' section below. * A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. * An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, competing interests statement, or data availability statement, please make these updates within the submission form at the time of resubmission. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter We look forward to receiving your revised manuscript. Kind regards, Varun Dutt, Ph.D Academic Editor PLOS Computational Biology Mark Alber Section Editor PLOS Computational Biology Journal Requirements: If the reviewer comments include a recommendation to cite specific previously published works, please review and evaluate these publications to determine whether they are relevant and should be cited. There is no requirement to cite these works unless the editor has indicated otherwise. Reviewers' comments: Reviewer's Responses to Questions Comments to the Authors: Please note here if the review is uploaded as an attachment. Reviewer #5: The Authors addressed my comments; however, after adding numpy, scipy, and napari to the analysis, they don't cite these packages when they appear in the text, e.g. page 12. Again, I find this very ironic given the nature of the work at hand. Please ensure that all of the packages discussed are cited properly! Reviewer #6: As a new reviewer of this paper, I took time to reflect on the manuscript and the reviewer reports and responses. I agree with the other reviewers that the true core software libraries in scientific computing are things like BLAS, which are not covered by the data scope of the paper. At the same time I agree with the authors that their analysis nevertheless has value because it scopes a method and applies it to demonstrated networked software dependencies and risks within a narrower field of science. It contributes because it shows how immense value and interdependency emerges from the decentralized work of scientists. We should care about these libraries because, as the authors point out, incentives and of the maintainers in this part of the software stack is different from incentives in OSS coding in general. Reviewer #7: I applaud the authors' efforts to improve this manuscript in previous rounds. As a new reviewer added just to this round, I don't think I have too many major comments, except that as a user of the CZI dataset, I have deep concerns of using this dataset for quantitative works. In our preliminary examination, we found pretty strong software name disambiguation problem, which was not captured by CZI's evaluation. I believe by focusing on just three software ecosystems, this problem can be alleviated, however, it cannot be totally solved. The one example that I hope the authors could check is Prism, which is a popular software name but is very poorly disambiguated in this dataset. So while the authors acknowledged this as a limitation, in my opinion, this is more than just a limitation that the authors should consider before designing this research. A related comment is that, in my opinion, some more basic software, such as scipy and numpy, are way less frequently mentioned than they should be. This is an inevitable bias in scientific methods reporting: not every actually used software will be reported in the scientific papers. I hope the authors could reflect on this bias and talk about alternative ways to trace the impact of software (such as through citations) if they see fit. ********** Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #5: Yes Reviewer #6: Yes Reviewer #7: Yes ********** PLOS authors have the option to publish the peer review history of their article ( what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy . Reviewer #5: No Reviewer #6: No Reviewer #7: No Figure resubmission: While revising your submission, we strongly recommend that you use PLOS’s NAAS tool ( https://ngplosjournals.pagemajik.ai/artanalysis ) to test your figure files. NAAS can convert your figure files to the TIFF file type and meet basic requirements (such as print size, resolution), or provide you with a report on issues that do not meet our requirements and that NAAS cannot fix.--> After uploading your figures to PLOS’s NAAS tool - https://ngplosjournals.pagemajik.ai/artanalysis , NAAS will process the files provided and display the results in the "Uploaded Files" section of the page as the processing is complete. If the uploaded figures meet our requirements (or NAAS is able to fix the files to meet our requirements), the figure will be marked as "fixed" above. If NAAS is unable to fix the files, a red "failed" label will appear above. When NAAS has confirmed that the figure files meet our requirements, please download the file via the download option, and include these NAAS processed figure files when submitting your revised manuscript. Reproducibility: To enhance the reproducibility of your results, we recommend that authors of applicable studies deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols PLoS Comput Biol. 2026 May 13;22(5):e1014260. doi: 10.1371/journal.pcbi.1014260.r006 Author response to Decision Letter 3 Article notes Copyright and License information Collection date 2026 May. PMC Copyright notice 15 Apr 2026 Attachment Submitted filename: SoftwareDependenciesPaper_Response3rdround.pdf pcbi.1014260.s004.pdf (39.2KB, pdf) PLoS Comput Biol. doi: 10.1371/journal.pcbi.1014260.r007 Decision Letter 3 Varun Dutt Author information Copyright and License information Roles Varun Dutt : Academic Editor © 2026 Varun Dutt This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. PMC Copyright notice 20 Apr 2026 Dear Dr Howison, We are pleased to inform you that your manuscript 'Biomedical open source software: Crucial packages and hidden heroes' has been provisionally accepted for publication in PLOS Computational Biology. Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests. Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated. IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript. Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS. Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. Best regards, Varun Dutt, Ph.D Academic Editor PLOS Computational Biology Mark Alber Section Editor PLOS Computational Biology *********************************************************** Based upon the revisions, the manuscript may be accepted in its current form. PLoS Comput Biol. doi: 10.1371/journal.pcbi.1014260.r008 Acceptance letter Varun Dutt Author information Copyright and License information Roles Varun Dutt : Academic Editor © 2026 Varun Dutt This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. PMC Copyright notice PCOMPBIOL-D-24-02238R3 Biomedical open source software: Crucial packages and hidden heroes Dear Dr Howison, I am pleased to inform you that your manuscript has been formally accepted for publication in PLOS Computational Biology. Your manuscript is now with our production department and you will be notified of the publication date in due course. The corresponding author will soon be receiving a typeset proof for review, to ensure errors have not been introduced during production. Please review the PDF proof of your manuscript carefully, as this is the last chance to correct any errors. Please note that major changes, or those which affect the scientific understanding of the work, will likely cause delays to the publication date of your manuscript. Soon after your final files are uploaded, unless you have opted out, the early version of your manuscript will be published online. The date of the early version will be your article's publication date. The final article will be published to the same URL, and all versions of the paper will be accessible to readers. For Research, Software, and Methods articles, you will receive an invoice from PLOS for your publication fee after your manuscript has reached the completed accept phase. If you receive an email requesting payment before acceptance or for any other service, this may be a phishing scheme. Learn how to identify phishing emails and protect your accounts at https://explore.plos.org/phishing . Thank you again for supporting PLOS Computational Biology and open-access publishing. We are looking forward to publishing your work! With kind regards, Judit Kozma PLOS Computational Biology | Carlyle House, Carlyle Road, Cambridge CB4 3DN | United Kingdom ploscompbiol@plos.org | Phone +44 (0) 1223-442824 | ploscompbiol.org | @PLOSCompBiol Associated Data This section collects any data citations, data availability statements, or supplementary materials included in this article. Supplementary Materials Attachment Submitted filename: rebuttal.pdf pcbi.1014260.s002.pdf (161.7KB, pdf) Attachment Submitted filename: ResponseToReviewers.pdf pcbi.1014260.s003.pdf (58.9KB, pdf) Attachment Submitted filename: SoftwareDependenciesPaper_Response3rdround.pdf pcbi.1014260.s004.pdf (39.2KB, pdf) Data Availability Statement Name: SoftwareImpactHackathon2023_Tracing_dependencies https://github.com/borisveytsman/SoftwareImpactHackathon2023_Tracing_dependencies doi: 10.5281/zenodo.10031223 . Articles from PLOS Computational Biology are provided here courtesy of PLOS ACTIONS View on publisher site PDF (1.2 MB) Cite Collections Permalink RESOURCES Similar articles Cited by other articles Links to NCBI Databases Follow NCBI NCBI on X (formerly known as Twitter) NCBI on Facebook NCBI on LinkedIn NCBI on GitHub NCBI RSS feed Connect with NLM NLM on X (formerly known as Twitter) NLM on Facebook NLM on YouTube National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894 Web Policies FOIA HHS Vulnerability Disclosure Help Accessibility Careers NLM NIH HHS USA.gov Back to Top

Related documents

• Beyond the digital health tsunami: reaffirming biomedical and health informatics as the scientific foundation of digital healthcare transformation#367419
• Beyond the digital health tsunami: reaffirming biomedical and health informatics as the scientific foundation of digital healthcare transformation.#232354
• Reimagining cardiac care with AI, LLMs, blockchain, and metaverse.#4378
• From Tool Invocation to Source-Mechanism Exploration: Protected White-Box DSE for Open-Source EDA#630075
• From Tool Invocation to Source-Mechanism Exploration: Protected White-Box DSE for Open-Source EDA#628213