Beyond Compliance: A Large Scale Study on the Completeness and Consistency of the GitHub SBOMs

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Beyond Compliance: A Large Scale Study on the Completeness and Consistency of the GitHub SBOMs

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

softwarearchitecture softwareengineering testing

software engineering, software architecture, testing

Noname manuscript No. (will be inserted by the editor)

Beyond Compliance: A Large Scale Study on the Completeness and Consistency of the GitHub SBOMs Kawsar Ahmed Bhuiyan · Mohamed Bilel Besbes · Rachna Raj · Adam Al Assil · Diego Elias Costa

arXiv:2607.04614v1 [cs.SE] 6 Jul 2026

Received: date / Accepted: date

Abstract Modern software development relies heavily on open-source components. Reusing components accelerates innovation but increases exposure to supply-chain attacks exploiting known vulnerabilities. Software Bills of Materials (SBOMs) improve software supply chain transparency by enumerating components, their versions, and their provenance. GitHub, the largest opensource development hosting platform, now automatically generates SBOMs for repositories, providing valuable metadata for risk assessment. Yet, it is unclear whether GitHub SBOMs can serve as a reliable source for vulnerability and license analysis, and how incomplete or inconsistent metadata may affect different programming ecosystems. To address this, we conduct a largescale analysis of 10,000 GitHub repositories across ten programming language ecosystems, evaluating GitHub SBOMs against three other popular SBOM generators: Syft, Trivy, and the Microsoft SBOM Tool. Our study finds a lack of NTIA compliance in GitHub SBOMs, though core metadata is consistently present. We also find that component version and license information availability is highly dependent on the programming ecosystem. Compared with Kawsar Ahmed Bhuiyan REALISE Lab, Concordia University, Montréal, Canada E-mail: [email protected] Mohamed Bilel Besbes REALISE Lab, Concordia University, Montréal, Canada E-mail: [email protected] Rachna Raj REALISE Lab, Concordia University, Montréal, Canada E-mail: [email protected] Adam Al Assil REALISE Lab, Concordia University, Montréal, Canada E-mail: [email protected] Diego Elias Costa REALISE Lab, Concordia University, Montréal, Canada E-mail: [email protected]

Bhuiyan et al.

the other three tools, GitHub yields results similar to the Microsoft SBOM Tool and often outperforms Syft and Trivy in providing version and license information. Finally, we discuss potential shortcomings of the GitHub SBOM Tool, directly related to how each ecosystem manages its dependencies. Keywords Software bill of materials · Software supply chain · Empirical study · SBOM Generators 1 Introduction The increasing reliance on third-party and open-source software components has become a defining characteristic of contemporary software development [64]. While this trend enables faster innovation and reduces engineering costs, it also increases the risk of exploitation, as attackers may target known vulnerabilities in these components, potentially undermining system integrity, exposing sensitive data, or causing significant operational and economic damage [26][16][42]. These challenges are intensified by the growing scale and complexity of software dependency networks, which often span thousands of interrelated packages [29][48]. To address these concerns, the Software Bill of Materials (SBOM) has emerged as a key mechanism for enhancing transparency in the software supply chain. An SBOM provides a structured inventory of all software components, including open-source and proprietary modules, along with their dependencies and relationships [54] [67]. By making this information transparent, SBOMs allow downstream users to assess potential cybersecurity and licensing risks [10] [21]. The strategic importance of SBOMs was recognized in the U.S. Executive Order on Improving the Nation’s Cybersecurity, issued in May 2021, which mandated the use of SBOMs in federal software procurement [62]. To promote standardization, the National Telecommunications and Information Administration (NTIA) established the Software Transparency initiative, defining guidelines for machine-readable SBOMs that specify minimum required elements such as supplier and component names, version information, dependency relationships, and other essential metadata [36]. These guidelines aim to harmonize SBOM practices across organizations, enabling scalable and interoperable adoption. Within this context, GitHub has emerged as a significant platform for SBOM generation and dissemination. GitHub hosts millions of open-source repositories and provides native support for SBOM creation by leveraging its dependency graph [19]. Hence, GitHub is a compelling target for empirical investigation, particularly given its ecosystem diversity and scale. As users may adopt GitHub SBOMs as part of their compliance and quality assessment [38][8], it is crucial to assess their compliance and consistency and to compare them with other widely used SBOM generation tools. In this paper, we present a large-scale empirical study of GitHub-generated SBOMs across 10 popular programming language ecosystems: C, C++, C#, Python, PHP, Java, JavaScript, Go, Swift, and Rust. We collect SBOM data

Title Suppressed Due to Excessive Length

from 10,000 GitHub repositories, and design our study to answer the following research questions: RQ1: How compliant and complete are GitHub SBOMs? We examine whether GitHub SBOMs include the key metadata required for software supply chain activities, focusing on the compliance to NTIA minimum elements. We then evaluate the presence of version and license information of GitHub SBOMs’ components. We found that GitHub SBOMs include most core metadata, such as component names, unique identifiers, and dependency relationships, but none fully meet NTIA minimum requirements due to missing supplier information. Version information is generally available for top-level dependencies, while transitive dependencies are only listed when their parent dependencies include versions. License information is inconsistently reported across dependency components. RQ2: How consistent is the GitHub SBOM Tool compared to other SBOM generators? This question explores how GitHub SBOMs differ from those produced by three widely used tools in terms of metadata coverage and consistency across ecosystems. We found that GitHub SBOMs generally report more dependencies and provide version and license information more frequently than those generated by Trivy and Syft. However, SBOMs generated by the Microsoft SBOM Tool tend to more consistently include version information. RQ3: How useful is the GitHub SBOM Tool compared to other SBOM generators in vulnerability tracking? This question evaluates how the GitHub SBOM Tool compares against other SBOM tools in supporting automated vulnerability tracking. We found that GitHub SBOM Tool and Microsoft SBOM Tool tend to list more components, and more frequently include unique identifiers (PURL) that can be used for vulnerability tracking. Syft and Trivy, on the other hand, frequently include components that lack precise version information. Consequently, GitHub SBOMs can be mapped to more vulnerabilities than other tools’ SBOMs across nearly all languages. However, the accuracy of vulnerability detection depends on whether exact version information is provided for the components. The main contributions of our paper are as follows: – We conduct the first comprehensive empirical study of GitHub-generated SBOMs using 10,000 repositories (1,000 repositories per programming language) across 10 programming languages, evaluating SBOM quality and completeness against the NTIA minimum element requirements. – We compare four major SBOM generators, revealing variations in version coverage, availability of unique identifiers, and license reporting across programming ecosystems. – We evaluate the practical usefulness of SBOMs for vulnerability tracking. – We publish our replication package1 to help foment more research on the tooling and SBOM quality topic. 1

https://doi.org/10.5281/zenodo.18883005

Bhuiyan et al.

We organize this paper across nine remaining sections. Section 2 provides key definitions and contextual information related to Software Bill of Materials (SBOMs), laying the foundation for understanding the scope and relevance of our study. Section 3 describes our study design in detail, including project selection and SBOM generation. Sections 4, 5, and 6 present our findings for each research question, with each section first outlining the specific approach used to address the question and then discussing the corresponding results. Section 7 further discusses four aspects of our results, including tool performance, component agreement between tools, and recurring patterns observed across ecosystems. Section 8 reviews the related work, and Section 9 discusses the threats to validity of our study. Finally, Section 10 concludes the paper.

2 Background 2.1 What is an SBOM A Software Bill of Materials (SBOM) is a formal record that contains details and supply chain relationships of the various components used in building software [35]. An SBOM lists all software components, including relevant component metadata, and describes the relationships between them. For example, an SBOM for a web application might list open-source libraries such as React, Lodash, and Axios, including their versions, licenses, and their relationships within the application’s dependency hierarchy. SBOMs offer several key benefits that enhance software development and security processes[8]. They provide clear visibility into all software components and their dependencies, enabling teams to manage and track both direct and transitive dependencies efficiently. This transparency helps organizations quickly identify and prioritize vulnerabilities, improving incident response and risk management. Additionally, SBOMs support license compliance by allowing legal teams to verify that all components adhere to organizational policies. Beyond internal benefits, SBOMs also create a competitive advantage for vendors, as they are increasingly required in procurement processes and signal a commitment to software quality and security [70]. The SPDX format. SPDX (System Packet Data Exchange, formerly Software Packet Data Exchange) is an open standard for communicating Software Bill of Materials information, including provenance, licensing, security, and related metadata [56]. By providing a common format, SPDX helps reduce redundant efforts across organizations and communities, facilitating the sharing of critical data and thereby streamlining compliance, security, and reliability processes [41]. Figure 1 visually outlines these key attributes, categorizing them into mandatory and optional components in the SPDX v2.3 format. The core required attribute in any SPDX document is the Creation Information section, which provides the foundational metadata necessary for the document’s identification and compatibility with various tools. Additionally, the SPDX

Title Suppressed Due to Excessive Length

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

{ " sbom " : { " SPDXID " : " SPDXRef - DOCUMENT " , " spdxVersion " : " SPDX -2.3 " , " creationInfo " : { " created " : " 2021 -09 -01 T00 :00:00 Z " , " creators " : [ " Tool : GitHub . com - Dependency - Graph " ] }, " name " : " github / example " , " dataLicense " : " CC0 -1.0 " , " d o c u m e n t N a m e s p a c e " : " https :// spdx . org /... " , " packages " : [ { " name " : " rails " , " SPDXID " : " SPDXRef - Package " , " versionInfo " : " 1.0.0 " , " d o w n l o a d L o c a t i o n " : " NOASSERTION " , " filesAnalyzed " : false , " l i c e n s e C o n c l u d e d " : " MIT " , " l ic en se D ec la re d " : " MIT " , " copyrightText " : " Copyright ( c ) 1985 " , " externalRefs " : [ { " r e f e r e n c e C a t e g o r y " : " PACKAGE MANAGER " , " referenceType " : " purl " , " r e f e r e n c e L o c a t o r " : " pkg : gem / rails@1 .0.0 " } ] } ], " relationships " : [ { " r e l a t i o n s h i p T y p e " : " DEPENDS_ON " , " spdxElementId " : " SPDXRef - Repository " , " r e l a t e d S p d x E l e m e n t " : " SPDXRef - Package " } ] }

27 28

Fig. 1 SPDX Document Overview [55]

29 30 31 32 33 34 35 36 37 38 39 40 41

}

Listing 1 GitHub-generated Example [20]

SBOM

format supports several optional sections that allow for a detailed description of the software and its components, including Package Information, File Information, Snippet Information, Other Licensing Information, Relationships, Annotations, and Review Information. Terminology: Components, packages, and dependencies. SPDX defines in the package information of an SBOM all components that compose the software system. In this paper, we use the term component to refer to all components in a software project. However, when an analysis discriminates between the types of dependency components, we refer to them either as toplevel dependencies or transitive dependencies of a software project.

2.2 GitHub & SBOM Generation GitHub holds a pivotal role in contemporary software development due to its widespread adoption as a collaborative platform for version control, code sharing, and project management. It serves as a central repository for millions of open source repositories, facilitating efficient teamwork and transparency. While numerous tools exist for generating Software Bill of Materials (SBOMs),

Bhuiyan et al.

the GitHub SBOM Tool stands out for its native integration within the GitHub ecosystem. This integration relies on repositories’ dependency graphs, which summarize manifest and lock files, along with dependencies submitted via the dependency submission API. A dependency graph provides information on dependencies’ versions, licenses, manifest sources, and known vulnerabilities, including transitive dependency paths when supported. Importantly, the SBOM generated from this data stays synchronized with the repository’s current dependencies and can be exported in the standard SPDX format via the GitHub UI or REST API [19]. This reduces manual work and helps maintain an up-to-date view of a project’s software supply chain. Listing 1 illustrates a GitHub SBOM (short for GitHub-generated SBOM) in the SPDX v2.3 JSON format. It shows how the GitHub SBOM Tool captures SPDX metadata such as the SPDX version, creation information, and document namespace. The packages section details a dependency (rails 1.0.0) along with its license (MIT), and package URL. The relationships section explicitly records dependency relationships, indicating which packages the repository depends on.

2.3 NTIA compliance The National Telecommunications and Information Administration (NTIA) has established guidelines for Software Bill of Materials (SBOM) compliance [34, 57, 36]. As detailed in Table 1, NTIA compliance requires SBOMs to include seven fundamental data fields that collectively enable comprehensive software component tracking and risk assessment. These minimum elements serve distinct but interconnected purposes in establishing supply chain transparency. The significance of each field, as outlined in the table, demonstrates how NTIA’s requirements address critical aspects of software security management, from tracing distribution sources and maintaining component inventories to identifying vulnerable versions and understanding inter-component dependencies. Additionally, compliant SBOMs must support automation and be machine-readable, typically delivered in standardized formats such as SPDX, CycloneDX, or SWID tags [40]. This regulatory framework represents a significant shift toward mandating supply chain transparency, particularly for software vendors serving federal government clients, while establishing a foundation for broader industry adoption of systematic software component tracking and risk assessment practices that leverage the structured data fields specified in NTIA’s minimum requirements.

3 Methodology Our goal is to study the completeness and consistency of SBOMs generated by GitHub across 10 programming ecosystems. Figure 2 illustrates our methodology. We begin by selecting 1,000 projects from each programming language,

Title Suppressed Due to Excessive Length

Table 1 NTIA Minimum Requirements for SBOMs [36] Data Field

Description

Reason of significance

Supplier Name

The name of an entity that creates, defines, and identifies components.

It helps trace the distribution source for a specific package as the same component may be supplied through different suppliers (e.g., maven repository, Gradle central repo).

Component Name

Designation assigned to a unit of software defined by the original supplier.

It is crucial to maintain an inventory list of all components used in a software such as packages and files. In GitHub’s generated SBOMs, mainly packages are used.

Version of the Component

Identifier used by the supplier to specify a change in software from a previously identified version.

Versions ensure compatibility and up-to-dateness as it identifies any changes to the software. Therefore, having it is important because we can identify which versions are vulnerable and need updates.

Other Unique Identifiers

Other identifiers that are used to identify a component, or serve as a look-up key for relevant databases.

Identifying components could be confusing so having a unique identifier for each component would make things easier.

Dependency Relationship

Characterizing the relationship that an upstream component X is included in software Y.

Dependency relationships specify different relationship types between components of the software such as which packages depend on others.

Author of SBOM Data

The name of the entity that creates the SBOM data for this component.

Identifies who created the SBOM as there are different tools such as GitHub’s dependency graph.

Timestamp

Record of the date and time of the SBOM data assembly.

Timestamps verify a record of when the SBOM was created for a specific software. It is important because as projects get updated, the SBOM should also change.

Bhuiyan et al.

aiming to capture active, highly popular, and collaborative software repositories (Section 3.1). For each repository, we generate a GitHub SBOM, yielding a total of 10,000 SBOMs (Section 3.3). To enable comparative analysis, we also generate SBOMs for the same repositories using three widely used tools: Syft, Trivy, and the Microsoft SBOM Tool (Section 3.3). We evaluate GitHub SBOMs for NTIA minimum compliance, version coverage, and license availability (RQ1, Section 4). We then compare GitHub SBOMs with SBOMs generated by other tools to assess differences in metadata completeness and dependency detection (RQ2, Section 5). Finally, we examine the practical usefulness of SBOMs for detecting known vulnerabilities (RQ3, Section 6). Together, these analyses provide a comprehensive view of SBOM completeness, metadata quality, and practical utility across programming languages and tooling.

Dataset Selection & Filtering Repositories from 10 popular languages

>= 5 Contributors

SBOMs Extraction

+1 commits in last 2 years

Sort by ? Count

10,000 Projects

GitHub SBOM Tool

10,000 GitHub SBOMs

Trivy

9,997 Trivy SBOMs

Syft

10,000 Syft SBOMs

Microsoft SBOM Tool

9,746 MS SBOMs

Analysis and Research Questions RQ2. SBOM Tools' Comparison

RQ1. SBOM Completeness 10, 000 GitHub SBOMs

Extract NTIA Compliance Features

Analysis of the Results (RQ 1.1)

Extract NTIA Compliance Features

Extract Components' Versions

Analysis of the Results (RQ 1.2)

Extract Components' Licenses

Analysis of the Results (RQ 1.3)

RQ3. Vulnerability Tracking

Normalise SBOM's output across tools

10,000 GitHub SBOMs

Query OSV.Dev

10,000 Syft SBOMs

Compare Package Count

10,000 Syft SBOMs

Compare Vulnerability Detection

9,997 Trivy SBOMs

Compare Metadata Availability

9,997 Trivy SBOMs

Summarize Tool Performance

9,746 MS SBOMs

Summarise Tool Performance

9,746 MS SBOMs

10,000 GitHub SBOMs

Fig. 2 Methodology overview of this study.

3.1 Project Selection To analyze SBOMs generated by GitHub, we target open-source repositories that are widely used and actively maintained by multiple contributors. These projects are more likely to reflect mature development practices and realistic software supply-chain behaviours. This choice is motivated by prior evidence that GitHub contains a substantial amount of repositories that are personal or inactive, and that naive sampling can therefore lead to datasets that are not representative of collaborative software development [24]. Within this population, we strategically select projects from ten widely used programming languages: C, C++, C#, Java, PHP, Python, JavaScript, Go, Rust, and Swift. These languages were chosen because they are consistently reported among the most popular and widely adopted languages across complementary popularity indicators, including the TIOBE Program-

Title Suppressed Due to Excessive Length

ming Community Index [59], GitHub’s Octoverse language rankings [18], and large-scale developer surveys from Stack Overflow [52]. Step 1: Contributor Threshold. We select only repositories with at least five unique contributors. This criterion helps us focus on collaborative and community-driven projects (instead of personal/toy repositories), which are more likely to reflect real-world development and maintenance practices. Similar contributor-based thresholds are commonly used in prior empirical studies in software engineering [45, 23]. Step 2: Recent Maintenance. The software repository must have shown at least one commit within the two years prior to our data extraction. This condition ensures that selected projects are minimally maintained. For the filtering in Steps 1 and 2, we use the SEART GitHub Search (GHS) platform [49], a web-based tool designed to support reproducible sampling of GitHub repositories for mining software repositories studies. SEART-GHS maintains a continuously updated dataset of GitHub repository metadata and provides a query interface that enables filtering projects based on multiple criteria, such as programming language, repository activity, popularity indicators, and licensing information [11]. SEART-GHS has been adopted in prior empirical software engineering studies to select representative sets of actively developed repositories [14] [30] [47], demonstrating its suitability for constructing reliable and reproducible datasets. Step 3: Popularity by Stars. We select the top 1,000 most popular projects per programming language based on their GitHub star count, resulting in a total of 10,000 projects across 10 programming languages. GitHub allows users to star repositories to express interest in or appreciation of a project. Prior empirical research has shown that the number of stars is commonly used as a proxy for project popularity, where repositories with higher star counts tend to receive greater community attention and adoption [6]. Following this established practice, we prioritize projects with higher star counts to capture influential and widely adopted software systems. This focus on popular repositories increases the likelihood that the analyzed projects reflect mature development practices and have broader ecosystem impact, thereby improving the generalizability and relevance of our findings. Following these steps, we selected 1,000 repositories for each of the 10 programming languages, resulting in a total dataset of 10,000 repositories. We present the distribution of the selected projects across different characteristics in Figure 3. 3.2 Selection of SBOM Generation Tools To compare the quality of GitHub SBOMs, we select the other widely used SBOM generation tools: – Syft: An open-source CLI tool developed by Anchore that generates SBOMs from container images, filesystems, and source directories, supporting multiple formats including SPDX and CycloneDX [1].

Bhuiyan et al.

Language

#Stars

#Forks

#Watchers

Size (B)

C 1.4k

123.7k

41.8k

2.5k

36m

1.1k

120k

25.2k

4.2k

10m

2.1k

190.4k

74.7k

7.5k

11m

3.3k

145.3k

40.7k

3.4k

1.7k

150.1k

54.3k

4.5k

35m

4.1k

236.4k

79.1k

6.7k

27m

79.7k

24.4k

4.4k

6.3k

358.5k

63.5k

9.8k

12m

1.2k

103.3k

10.6k

1.8k

265

48k

7.9k

2.5k

C++

Java

JavaScript

PHP

Python

Rust

Swift

Fig. 3 Statistical Distribution of the Selected 10,000 Repositories

– Trivy: A security and compliance scanner maintained by Aqua Security that can produce SBOMs as well as detect vulnerabilities in container images, filesystems, and Git repositories [4]. – Microsoft SBOM Tool: A free command-line tool provided by Microsoft that generates SBOMs for a wide range of project types, supporting multiple programming languages and producing output in SPDX and CycloneDX formats [31]. We selected these tools because they are free, open-source, and support multiple operating systems (Linux, Windows, and macOS). These tools are

Title Suppressed Due to Excessive Length

also frequently used in previous studies [69] [61] and industry overviews [44] [50], demonstrating their relevance and adoption.

3.3 SBOM Generation Upon selecting our dataset of the most popular 10,000 open-source repositories across 10 programming languages, we aim to export their SBOMs using the GitHub SBOM generator, Trivy, Syft, and Microsoft SBOM Tool. GitHub SBOM Tool. To extract SBOMs generated from the recently added GitHub SBOM Tool, we use the GitHub REST API for SBOMs [20] to programmatically generate the SBOM for each selected repository by issuing a GET request to the /repos/owner/repo/dependency-graph/sbom endpoint, which returns an SPDX-compatible SBOM derived from GitHub’s dependency graph when enabled. All of the SBOMs were obtained in SPDX JSON format, as GitHub’s SBOM REST API currently exports SBOMs in this format [20]. For each language, we initiated SBOM extraction starting from the most popular repositories and proceeded down the list. Since GitHub requires the dependency graph to be enabled in a repository to export an SBOM [19], we received a 404 error when attempting to generate SBOMs via the REST API for repositories where the dependency graph was disabled by the owners. In such cases, we skipped the repository and continued to the next most popular one. This iterative process was repeated until we successfully obtained 1,000 valid SBOMs per language. Extraction from Local Repositories. All three SBOM generator tools, Trivy, Syft, and Microsoft SBOM Tool, provide support for SBOM generation using local repositories [3, 2, 32]. Following the GitHub-based SBOM generation, all 10,000 repositories were cloned to local storage immediately thereafter, and all local SBOM generation runs were performed against these same cloned snapshots. By cloning directly after the GitHub phase, the local snapshots reflect the same repository state that was present during GitHub SBOM generation, minimizing the risk of source divergence between the two extraction methods. Each tool was then executed against the same locally cloned repository snapshot, ensuring identical source code, dependency manifests, and project structures were analyzed across all three tools. Each tool has parameters that control the sensitivity of the SBOM generation. We configured each tool to run under the most permissive license-discovery settings available to obtain the maximum possible license coverage Trivy. For Trivy, SBOMs were generated from the locally cloned repositories using Trivy’s filesystem scanning mode. For each repository, we executed Trivy with SPDX JSON output enabled. We enabled --license-full, which instructs it to scan not only component metadata but also the full contents of each file for embedded license expressions. Trivy successfully generated SBOMs for 9,997 repositories. Three failures occurred due to excessive analysis time

Bhuiyan et al.

on large multi-module Java projects and a stack-overflow issue triggered by its XML-based Go module parsing. Syft. Syft operates exclusively on local inputs such as source code directories, container images, or packaged artifacts [2]. For each cloned repository, Syft was executed with the repository directory as input and configured to emit SPDX JSON output. Syft successfully generated SBOMs for all 10,000 repositories in the dataset. Microsoft SBOM Tool. The Microsoft SBOM Tool similarly requires local repository inputs [32]. For each repository, SBOM generation involved creating a temporary manifest directory and executing the tool with repository metadata parameters. The Microsoft SBOM Tool internally queries the ClearlyDefined [9] API, an open-source crowdsourced database of license and attribution metadata, to retrieve license information. The resulting SPDX JSON files were extracted from the generated manifest directory. The Microsoft SBOM Tool exhibited a higher failure rate, particularly on repositories containing recursive symbolic-link structures or missing build-generated files. These failures stem from limitations in its hashing mechanism, which cannot correctly handle symlink loops or absent files, resulting in a total of 9,746 successfully generated SBOMs.

4 RQ1: How compliant and complete are GitHub SBOMs? Understanding the completeness of GitHub-generated SBOMs is essential because the usefulness of any SBOM depends on the accuracy and availability of the information it provides [40]. While specific needs may vary by use case, all applications require a consistent and systematic approach to defining and identifying software components and their relationships. An SBOM typically includes both the primary software component and its dependency components. In this study, we focus on the latter. We assess the completeness of the generated SBOMs with respect to three main aspects: (1) the NTIA minimum requirements (see Table 1), (2) the presence of version information for dependencies, and (3) the availability of licensing information for dependencies. Accordingly, we structure RQ1 into three sub-questions: – RQ 1.1: Do GitHub SBOMs satisfy the NTIA minimum requirements? – RQ 1.2: To what extent do GitHub SBOMs include version information for dependencies? – RQ 1.3: To what extent do GitHub SBOMs provide licensing information for dependencies?

Title Suppressed Due to Excessive Length

4.1 Approach To evaluate the completeness of GitHub SBOMs, we analyze each SBOM and aggregate the results by programming language to identify ecosystem-level patterns. Assessing NTIA Compliance of GitHub-Generated SBOMs. According to the NTIA guidance on the Minimum Elements for an SBOM, an SBOM should list all primary components along with their dependencies [36]. We analyze each SBOM file to identify the NTIA-required elements (listed in Table 1) using the corresponding SPDX field names and specification references. – Component Name: The primary component, i.e., the software system for which the SBOM was generated, is identified from the top-level name field of the SBOM document. Dependency components’ names are found by inspecting the name field within each entry of the packages array. – Version of the Component: The version of each component—including the primary component and third-party library packages—is extracted from the versionInfo field located in its respective entry within the packages array. – Supplier Name: The supplier is extracted by checking the supplier or PackageSupplier field within each package entry. – Other Unique Identifiers: Unique identifiers for components are found in the SPDXID field, while the SBOM document itself is uniquely referenced through the documentNamespace field. Examples of commonly used unique identifiers are Common Platform Enumeration (CPE), Software Identification (SWID) tags, and Package Uniform Resource Locators (PURL) [36]. We tried to look for CPE, SWID or PURL by inspecting the externalRefs field of each entry of the packages array. We examine the externalRefs field of each entry in the packages array and consider the presence of a unique identifier if at least one of CPE, SWID, or PURL is found. – Dependency Relationship: When present, dependency relationships between components are described in the relationships array. – Author of SBOM Data: The authoring entity is identified through the creators field under the creationInfo section. This may include the names of tools, individuals, or organizations responsible for generating the SBOM. – Timestamp: The timestamp indicating when the SBOM was created is retrieved from the created field within the creationInfo section. NTIA recommends that SBOMs include all top-level dependencies with sufficient detail to enable the recursive discovery of transitive dependencies [36]. Therefore, in this analysis, we focus primarily on the top-level dependencies for each project and verify whether each of them includes required fields such as name, supplier, versionInfo, and a unique identifier. We determine the top-level dependencies by examining the relationships array and selecting

Bhuiyan et al.

entries where the spdxElementId matches this primary identifier. Comparison of Version Coverage of Top-Level and Transitive Dependencies. Beyond just the NTIA compliance, which considers just the top-level dependencies, it is also beneficial to include version information for transitive dependencies. Transitive dependencies are known to introduce security vulnerabilities, licensing risks, or compatibility issues [68][33]. To identify transitive dependencies, we trace the dependencies from each top-level dependency to the components it depends on. After identifying both primary and transitive dependencies, we inspect the versionInfo field in their corresponding entries within the packages array to verify whether valid version information is provided. Identifying Components’ License Availability. For each SBOM, we checked whether license information was available for the top-level dependencies and transitive dependencies. Within the packages array, the licenseConcluded and licenseDeclared fields were checked for all the dependencies. We measured the share of components with available license information among toplevel dependencies, transitive dependencies, and all dependencies combined (the union of both). These dependency-level percentages were then averaged by language to determine overall license availability trends.

4.2 Do GitHub-generated SBOMs satisfy the NTIA minimum requirements? We present the NTIA compliance results for all analyzed SBOMs, summarized in Table 2, including a breakdown by element, attribute, and programming language. We observe the following: Finding 1. No GitHub SBOM is fully compliant with the NTIA minimum requirements. To be considered fully compliant, an SBOM must satisfy all defined minimum elements, and in our analysis, that was obtained by no SBOM generated by GitHub. The most critical gap is the absence of supplier information—0% of the SBOMs include supplier data for either the primary component or any top-level dependencies. Supplier information is an important information as it specifies the name of the entity that creates, defines and identifies components. For example, Guava is a Java library developed and distributed by Google; an SBOM that lists Guava as a dependency should therefore identify Google as its supplier. This universal omission of supplier data leads to 0% full compliance with the NTIA minimum requirements across all evaluated languages. Finding 2. GitHub SBOMS show strong compliance in core metadata fields. Despite the absence of supplier information, all other NTIArequired fields are widely represented and compliant across SBOMs. Every SBOM includes the name and version of the primary component, as well as their unique identifier. In addition, 100% of SBOMs provide dependency relationships for all listed components, author metadata, and timestamps. Ad-

Title Suppressed Due to Excessive Length

Table 2 Compliance of SBOMs Across Different Languages with NTIA’s Minimum Requirements Element

Attribute

C# C++

Java

JS PHP Python Rust Swift

Supplier Primary Software Name Component Version Unique ID

0% 0% 0% 0% 0% 0% 0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%

0% 0% 0% 100% 100% 100% 100% 100% 100% 100% 100% 100%

Supplier Name Version Unique ID

0% 0% 0% 0% 0% 0% 0% 100% 100% 100% 100% 100% 100% 100% 83.3% 79.5% 75.0% 93.5% 41.4% 96.2% 82.9% 100% 100% 100% 100% 100% 100% 100%

0% 0% 0% 100% 100% 100% 29.9% 91.8% 98.8% 100% 100% 100%

Dependency Relationships

100% 100% 100% 100% 100% 100% 100%

100% 100% 100%

Author Timestamp

100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%

100% 100% 100% 100% 100% 100%

All Top-Level Dependency Components All Components

SBOM Overall Compliance

ditionally, all SBOMs include the name and unique identifiers for all the toplevel dependencies. For GitHub SBOMs, all the top-level dependencies include SPDXID as well as PURL as their unique identifier. Finding 3. Dependency version coverage in projects varies significantly per programming language. The level of compliance varies substantially across projects written in different programming languages. Swift exhibits the highest coverage, with 98.8% of SBOMs providing version information for all top-level dependencies, followed closely by JavaScript (96.2%), Go (93.5%), and Rust (91.8%). In contrast, Java and Python show the lowest compliance in this area, with respectively only 41.4% and 29.9% of their SBOMs including version information for all the top-level dependencies. C, C#, C++, and PHP fall in the mid-to-high range, with coverage between 75.0% and 83.3%.

4.3 Do GitHub-generated SBOMs include version information for dependencies? We present the results of our analysis in Figure 4. We observe significant variation in the consistency with which generated SBOMs contain dependency version information across languages. It is important to note that the values in the table differ significantly from those in our analysis in RQ1. RQ1 focuses on NTIA compliance and shows the share of projects that include version information for all top-level dependencies, while in this analysis, we are interested in the average distribution of version information across projects. Finding 1. With the exception of Python and Java projects, GitHub-generated SBOMs, on average, include more than 94% of versioned dependencies. We show in Figure 4 that the presence of versioned dependencies encompasses at least 94% of a project’s dependencies

Bhuiyan et al.

Fig. 4 Average version information of dependencies per project.

across most of the evaluated programming languages. The share of versioned dependencies exceeds 99.5% in projects in Go, Rust, and Swift, showing the strongest support across the ten languages. The exception to this rule is projects in Java and Python. Java projects exhibit an average of 83% of versioned dependencies, while Python projects show only 78% of versioned dependencies. These results are inline with our results in RQ1 (Table 2), where we showed that Java and Python projects exhibited the least share of projects with version included in all top-level dependencies. C, C++, and C# projects fall between these extremes, with approximately 95% of dependencies on average specifying version information. Finding 2. All listed transitive dependencies have version information, but transitive dependencies of unversioned dependencies are omitted. Figure 4 also breaks down the analysis per top-level dependency (a.k.a. direct dependencies) and transitive dependencies. We notice that all transitive dependencies include version information across all programming languages. This apparent completeness arises from the behavior of the GitHub SBOM Tool. We examined the generated SBOMs in depth and found that when a parent dependency lacks version information, the tool omits all of its transitive dependencies from the SBOM. In these cases, only the unversioned parent dependency appears, with none of its downstream dependencies listed. Consequently, the reported 100% version coverage for transitive dependencies does not imply that all transitive dependencies in the projects are versioned; it reflects a conservative approach from the GitHub SBOM Tool, that only includes transitive dependencies of versioned parent dependencies. Finding 3. There are vast differences in the number of project components across programming languages. Similarly, as reported in previous studies [13], the number of dependencies varies substantially across programming languages. On average, JavaScript and Rust projects exhibit the highest total dependency counts, with 1143 and 558 dependencies per project, respectively, indicating a strong reliance on third-party libraries. In contrast,

Title Suppressed Due to Excessive Length

Swift and C projects show considerably lower dependency usage, averaging 77.00 and 86.62 dependencies per project. According to GitHub SBOMs, we observe distinct patterns in how programming languages structure and manage their dependency hierarchies: some ecosystems exhibit broad, direct composition, whereas others rely on deeper chains of indirect dependencies. In particular, Rust, Go, and Java projects tend to include more top-level dependencies on average than other languages, suggesting heavier reliance on direct imports. However, this pattern reverses at the transitive level. On average, JavaScript projects include over 1,024 transitive dependencies, approaching an order-of-magnitude increase relative to their direct imports, and substantially exceeding the transitive dependency counts observed in Rust, Go, and Java projects. This contrast highlights JavaScript projects’ exceptionally deep dependency trees, where relatively fewer direct components expand into extensive chains of indirect reuse. Python, PHP, C, C++, C#, and Swift projects exhibit moderate expansion, typically with two to three times more transitive dependencies than top-level components, reflecting layered but more tightly managed dependency structures.

4.4 Do GitHub-generated SBOMs provide licensing information for dependencies? We present a distribution of the percentage of dependencies with licensing information per project, broken down by programming language and dependency type (all, top-level, and transitive) in Figure 5. We opt to showcase the distributions rather than averages because, unlike the component version analysis, the data is heavily skewed. Many projects have either 0% or 100% of dependencies licensed, making the mean a poor summary of the underlying variation. Once again, we notice some contrasts related to licensing information across programming languages, and report the following observations: Finding 1. For projects written in C, C++, Go, PHP, Python, and Swift, license information is absent for the vast majority of dependencies. Figure 5 shows that, for C, C++, Go, PHP, and Swift projects, the distribution of license coverage is close to bimodal. More than half of the projects in each of these languages report 0% of their dependencies with license information, while only a small fraction achieve complete (100%) coverage. Very few projects fall in the intermediate ranges. Finding 2. SBOMs from JavaScript and Rust projects report license information for a substantially larger share of dependencies. In contrast to other ecosystems, JavaScript and Rust projects demonstrate comparatively high levels of license information reporting. In JavaScript, more than half of the projects provide license information for nearly all dependencies. Rust projects exhibit a similar trend, with nearly one-quarter achieving complete or near-complete license reporting. This observation is consistent with the findings of Wu et al. [65], who analyzed 33,710,877 packages across

Bhuiyan et al.

Language

All Dependencies

Top-Level Dependencies

Transitive Dependencies

#Projects C

750 500 250 0

C++

750 500 250 0

Java

750 500 250 0

JavaScript

750 500 250 0

PHP

750 500 250 0

Python

750 500 250 0

Rust

750 500 250 0

Swift

750 500 250 0

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

25%

50%

75% 100% 0%

25%

50%

75% 100% 0%

25%

50%

75% 100%

Fig. 5 The distribution of dependencies’ license information per project. Unlike top-level dependencies, not all generated SBOMs include transitive dependencies; thus, the count may not sum up to 1,000 per programming language.

Title Suppressed Due to Excessive Length

five package managers and reported that npm (JavaScript) and Cargo (Rust) exhibit the highest percentages license information in SBOMs among all studied ecosystems. Finding 3. C#, Java, and Python projects have varying degrees of license information. When analyzing projects written in C#, Java, and Python, we notice a more uniform distribution in the proportion of dependencies for which license information is reported. GitHub-generated SBOMs vary in the degree of license information; however, across all ecosystems, fewer than 25% of projects provide license information for all their dependencies. This suggests that most projects will have a sizeable share of components without license information in the GitHub-generated SBOMs for these programming languages. Finding 4. License information is almost always present in listed transitive dependencies. The differences in license availability across ecosystems are mainly due to how the GitHub SBOM Tool reports licenses for toplevel dependencies, as shown in Fig. 5. However, for the listed transitive dependencies, license information is almost always included, regardless of the language. As discussed in our previous analysis, GitHub SBOM Tool adopts a conservative approach and only lists transitive dependencies of parent dependencies that have a version. 5 RQ2: How consistent is the GitHub SBOM Tool compared to other SBOM generators? While RQ1 focused exclusively on evaluating the completeness of SBOMs generated by GitHub, this research question broadens the scope to assess how GitHub’s SBOM tool compares with other widely used SBOM generation tools in practice. The goal is to understand whether GitHub SBOMs provide comparable metadata quality, particularly in attributes required for effective vulnerability analysis and license compliance assessment. Accordingly, we analyze SBOMs generated by GitHub, Trivy, Syft, and the Microsoft SBOM Tool for the selected repositories. The procedures used to generate these SBOMs are described in Section 3; in this research question, we focus exclusively on analyzing and comparing the contents of the resulting SBOMs. 5.1 Approach Using the SBOMs generated for the selected repositories, we analyze differences at two levels: dependency detection and metadata completeness. We first examine variations in the number of components reported by each tool to capture differences in dependency discovery. We then assess the presence of version and license metadata at the component level, applying consistent validation criteria and aggregating results at the repository level to enable comparison across tools and programming languages.

Bhuiyan et al.

Component Count Analysis. For all the SBOMs generated by the four tools, we first examined the number of components reported in each SBOM. Due to the highly skewed distribution of component count data across SBOM tools, we employ two statistical methods, as done in previous studies [17] [28]. First, we employed the Kruskal–Wallis H-test [25], a non-parametric test used for comparing multiple independent samples. For each programming language, this test was used to assess whether the four SBOM tools differ statistically in the number of detected components across repositories. When the Kruskal–Wallis test indicated significant differences, we conducted Dunn’s post-hoc test [15] with Bonferroni correction to examine pairwise differences in component count distributions between the GitHub SBOM Tool and each of the other SBOM generation tools. Validation Criteria. We choose to focus on version and license information to assess the comparative performance of the four SBOM tools. Each metadata attribute was evaluated for presence and validity using the following criteria: – Version information was considered present if an SBOM entry contained a non-empty versionInfo or version field that was not marked as “UNKNOWN”. – License information was considered present if any of licenseDeclared, licenseConcluded, or licenseInfoFromFiles contained meaningful values other than “NOASSERTION”, “NONE”, or “UNKNOWN”. Percentage Calculation and Aggregation. For each SBOM, we computed the percentage of components that contained valid values for each metadata attribute. For every tool–language combination, these per-SBOM percentages were averaged across all valid SBOM files. This repository-centric aggregation ensures each repository contributes equally to the final results, regardless of its dependency graph size.

5.2 Results We present here the main empirical findings from our analysis, highlighting differences in component detection and metadata reporting across SBOM tools and programming languages. Finding 1. GitHub SBOM Tool detects significantly more components per project than all other SBOM generation tool, except in Go projects. Figure 6 shows clear variation in the number of components detected across the four tools. In many ecosystems, including C#, JavaScript, PHP, and Python, the GitHub SBOM Tool reports the highest average number of components. Trivy consistently reports the fewest components across most languages, while Syft and the Microsoft SBOM Tool generally fall in the middle.

Title Suppressed Due to Excessive Length

Finding 2. Go projects show consistency in the component counts across all SBOM tools. In our analysis using the Kruskal–Wallis H-test, we found no statistically significant differences in the average number of components of Go projects detected across all SBOM tools (p = 0.186). Likely due to Golang native support in listing software components, all four SBOM tools show a consistent average number of components per project. In contrast, projects from all other languagues exhibit statistically significant differences in detected component counts (p < 0.05). Finding 3. Pairwise comparisons reveal significant differences in component consistency across tools. In the nine programming languages with significant differences, we performed Dunn’s post-hoc test with Bonferroni correction to examine pairwise differences between the GitHub SBOM Tool and all the other tools. The pairwise analyses revealed the following patterns: – GitHub SBOM Tool vs Microsoft SBOM Tool: Both tools show no consistency on the number of components per project in all the remaining nine languages (p < 0.05), indicating significant differences in component detection between both tools. – GitHub SBOM Tool vs Syft: Both tools showed comparable component count distribution in Java (p = 0.466) and Rust (p = 0.601). All other languages showed significant differences in the distribution of components. – GitHub SBOM Tool vs Trivy: Only Java projects showed consistency in the number of components between both tools (p = 0.645). The remaining eight languages showed significant differences in the reported distributions of components. Surprisingly, only SBOMs generated for Go projects and Java projects showed some level of consistency across tools. Across most of our dataset, GitHub SBOM Tool shows a statistically significant difference in the reported number of components if compared to the Microsoft SBOM Tool, Syft and Trivy. Finding 4. Microsoft SBOM Tool consistently provides complete component versioning (100%), while GitHub SBOM Tool reports 94% of versioned components. We present in Figure 7 a heatmap of the average versioned component per project. We notice that the Microsoft SBOM Tool provides version information for nearly every component across all languages. The GitHub SBOM Tool follows closely, reporting version data for almost all components except in Java (85%) and Python projects (80%). Syft performs well for Go and Rust projects, where approximately 90% of components contain version information, but for projects in other languages, its coverage decreases to the 60–80% range. Trivy shows the weakest performance overall, never reaching 90% version reporting in any language and in some cases providing version data for only 15–30% of components per project. Finding 5. GitHub SBOM Tool provides more complete component license information than other tools for most programming

Bhuiyan et al. C

Microsoft SBOM Tool

1 10

1 100

10000

Component Count

100

564

101

100

10000

Component Count

10000

100

262

255

100

10000

100

Swift 5 1

278

227 100

10000 1000000

Component Count

100

10000 1000000

Component Count

Rust

Component Count

Python

1 100

104

Component Count

Trivy

Syft

86 14

PHP

466

Microsoft SBOM Tool

10000

Component Count

Java

JavaScript GitHub SBOM Tool

9 1

C++

Syft Trivy

GitHub SBOM Tool

4 10000

Component Count

100

10000

Component Count

Fig. 6 Component Count Distribution Per Repository by SBOM Tool and Language

99.2

80.8

99.8

Microsoft SBOM Tool

100.0

99.8

Syft

63.2

70.2

73.5

93.4

81.8

79.9

75.0

83.4

92.1

67.5

Trivy

17.3

40.7

26.2

84.6

57.3

55.1

27.2

49.2

72.3

45.0 Sw

st Ru

tho Py

t rip Sc Ja

C+

ift

99.5

85.0

99.8

95.5

97.0

96.8

GitHub SBOM Tool

SBOM Source

Version Reporting (%)

Language

40 60 Mean Percentage (%)

100

Fig. 7 Average percentage of versioned components per project across languages and SBOM tools.

languages. We show in Figure 8 a distribution of the license availability percentage per project, across all tools and programming languages. We notice that GitHub SBOM Tool provides a more complete license information in nine out of ten languages, showin higher averages. However, even in these cases, coverage remains far from complete: only in Rust and JavaScript projects does the GitHub SBOM Tool achieve an average of more than 80% of components’ license information. The Microsoft SBOM Tool delivers comparable performance to GitHub for JavaScript and Rust projects. For Java projects, Trivy and Syft perform similarly to the GitHub SBOM Tool. Go projects stand out as a notable exception: Trivy substantially outperforms all other tools by a wide margin.

Title Suppressed Due to Excessive Length C GitHub

16.7%

58.4%

GitHub

C++ GitHub

16.7%

Go GitHub

0.0% Microsoft

0.0% Syft

0.0% Trivy

Trivy

0 25 50 75 100 % of Components with License

JavaScript GitHub

0 25 50 75 100 % of Components with License

PHP 97.9%

11.1%

26.3%

42.8%

Trivy

0 25 50 75 100 % of Components with License

Python GitHub

Syft

Rust GitHub

0.0% Syft

0.0% Trivy

0 25 50 75 100 % of Components with License

87.6%

GitHub

0.0% Syft

0 25 50 75 100 % of Components with License

22.0%

34.5%

Swift 88.5%

0.0% Microsoft

Microsoft

38.8%

0 25 50 75 100 % of Components with License

0.0% Microsoft

Microsoft

95.1%

GitHub

Java

3.9% GitHub

0 25 50 75 100 % of Components with License

50.0%

0.0% Microsoft

0 25 50 75 100 % of Components with License

Fig. 8 Distribution of License Availability Percentage Per Project by SBOM Tool and Language

6 RQ3: How useful is the GitHub SBOM Tool compared to other SBOM generators in vulnerability tracking? Vulnerability detection is one of the most critical use cases of SBOMs. Modern software ecosystems contain large numbers of third-party libraries, and even a single outdated or vulnerable component can introduce significant security risks [48, 23]. SBOMs are intended to support this process by providing the information necessary to match components against vulnerability databases and evaluate their security status. Hence, in this RQ, we try to assess how useful GitHub SBOM Tool is in detecting vulnerabilities in a Software product and compare its performance with the three other widely used SBOM generation tools. 6.1 Approach PURL-Based Vulnerability Querying. To track software components, we rely on Package URLs (PURLs), which provide a canonical and ecosystemagnostic mechanism for uniquely identifying software components. We use OSV.dev, a distributed vulnerability database for open-source software, as our vulnerability data source [43]. OSV.dev aggregates vulnerability information from multiple authoritative upstream sources and supports vulnerability retrieval through a public API. Vulnerabilities can be queried either by specifying a component (or package)’s name and ecosystem or by providing a PURL. In our analysis, we exclusively use PURLs to ensure precise and consistent component matching. We process all SBOMs generated by the four tools and attempt to extract PURLs for every component listed in each SBOM. For each SBOM, we exam-

Bhuiyan et al.

ine the entry corresponding to every component and check for the presence of an externalRefs array, search for an item with referenceType set to purl and and extract the associated referenceLocator value, which contains the Package URL (PURL) of the component. After collecting all available PURLs within a given SBOM, we submit them together to the batch query endpoint of the osv.dev API. The API returns all known vulnerabilities associated with each queried component, and we aggregate these results to obtain the complete set of vulnerabilities corresponding to the components listed in that SBOM. Vulnerability Count Analysis. We first analyzed the total number of vulnerabilities associated with each SBOM and examined the distribution of vulnerability counts reported. Similar to the component count analysis (Section 5), we used the Kruskal–Wallis H-test and Dunn’s post-hoc test with Bonferroni correction to compare the distribution of vulnerability counts across SBOM tools.

6.2 Results Finding 1. PURL reporting is consistent in GitHub SBOM Tool and Microsoft SBOM Tool, while Syft and Trivy show inconsistency. GitHub SBOMs provides a PURL for every component across all languages (100% coverage). The Microsoft SBOM Tool performs similarly, reporting PURLs for nearly all listed components, with a negligible drop observed only for Swift (99.8%). Syft reports PURLs for a substantially smaller fraction of the listed components, and its performance varies notably by language. On average, Syft reports PURLs for approximately 60–90% of the listed components for all the languages. Trivy reports PURLs for the smallest fraction of listed components overall. For projects written in certain languages, including C (17.7%), C++ (27.1%), and PHP (27.3%), fewer than one-third of the listed components include a PURL. These results indicate that, unlike GitHub and Microsoft, Syft and especially Trivy frequently omit PURLs for a large portion of the components they list. Finding 2. PURL availability is highest for Go and Rust projects. Across all tools, Go and Rust projects show the highest PURL availability. Both GitHub and Microsoft SBOM Tool report PURLs for every component in Go and Rust projects. Syft and Trivy, despite their overall inconsistent performance, also perform relatively well for Go and Rust, achieving their highest PURL coverage in these projects. On average, Syft reports PURLs for 94.0% of components in Go projects and 91.9% in Rust projects. For Trivy, the values are 89.4% in Go and 70.9% in Rust projects. Finding 3. The number of vulnerabilities detected varies significantly by tool. As shown in Figure 10, the GitHub SBOM Tool consistently detects a higher number of vulnerabilities than the other tools across almost all programming languages. This outcome was expected, as the GitHub SBOM

Title Suppressed Due to Excessive Length

100.0

Microsoft SBOM Tool

100.0

99.8

Syft

63.2

67.6

72.2

94.0

89.8

80.4

75.7

83.3

92.2

69.7

Trivy

17.7

40.8

27.1

89.4

59.1

55.3

27.3

50.7

72.4

45.0 Sw

st Ru

tho Py

t rip va Sc Ja

C+

ift

100.0

GitHub SBOM Tool

SBOM Source

PURL Reporting (%)

Language

40 60 Mean Percentage (%)

100

Fig. 9 Average percentage of components with valid PURLs across languages and SBOM tools.

Tool generally reports more components and provides a PURL for each listed component, enabling complete vulnerability lookups via OSV.dev. Among the remaining tools, Syft and the Microsoft SBOM Tool report broadly similar vulnerability counts, though their relative differences vary by language. Syft reports more vulnerabilities in Python, C, and C++ projects, while the Microsoft SBOM Tool reports more in JavaScript and PHP projects. Trivy consistently reports the lowest number of vulnerabilities across nearly all languages. This can be attributed to its combination of listing fewer components and frequently omitting PURLs for the components it does report, which limits the effectiveness of OSV. dev-based queries. Applying the Kruskal–Wallis H-test to the vulnerability counts for each programming language, we found that all ten languages showed statistically significant differences (p < 0.05) in vulnerability detection across the four SBOM tools. Dunn’s post-hoc test with Bonferroni correction to examine pairwise differences in vulnerability detection between the GitHub SBOM Tool and each competing tool revealed the following patterns: – GitHub SBOM Tool vs Microsoft SBOM Tool: The number of vulnerabilities reported between both tools was consistent in C# (p = 0.224), JavaScript (p = 0.112), and Swift projects (p = 0.421). All other languages showed statistically significant differences. – GitHub SBOM Tool vs Syft: Both tools showed a consistent number of vulnerabilities only in C (p = 0.127), PHP (p = 0.353) and Swift projects (p = 0.944). Projects from all other languages showed a distinct number of vulnerabilities. – GitHub SBOM Tool vs Trivy: Both tools showed a consistent number of vulnerabilities in projects written in C# (p = 0.224) and PHP (p = 0.052). Tools significantly diverge in all other remaining languages.

Bhuiyan et al. C

GitHub SBOM Tool

Microsoft SBOM Tool

8 16

Syft

Trivy

10 100 1000 Vulnerability Count

JavaScript GitHub SBOM Tool Microsoft SBOM Tool Syft Trivy

PHP

C++

17 12

10 100 1000 Vulnerability Count

Python

Rust

10 100 1000 Vulnerability Count

Swift 5

10 100 1000 Vulnerability Count

Java

10 100 1000 10000 Vulnerability Count

5 5 3

10 100 1000 Vulnerability Count

10 100 Vulnerability Count

1000

Fig. 10 Vulnerability Count Distribution Per Repository by SBOM Tool and Language.

Fig. 11 Effect of presence of version information on PURL. VERSION

PURL

DETECTED VULNERABILITIES

77.0.3

pkg:pypi/[email protected]

GHSA-5rjg-fvgr-3xxf PYSEC-2025-49

pkg:pypi/setuptools

GHSA-27x4-j476-jp5f GHSA-5rjg-fvgr-3xxf GHSA-cx63-2mw6-8hw5 GHSA-r9hx-vwmv-q579 PYSEC-2013-22 PYSEC-2022-43012 PYSEC-2025-49

No Version 70.0.0

>=66.1.0

~>67.7

=78.*,>=78.1.1

Fig. 12 How version information’s availability in PURL affects detection of vulnerabilities.

While no single programming language showed consistent vulnerability counts across all tools, GitHub SBOM Tool is sporadically consistent with other tools in specific languages, usually related to C#, PHP and Swift. A caveat about the number of vulnerabilities. The precision of vulnerability detection depends on the availability of version information. Our analysis indicates that while OSV queries based on PURLs always return vulnerability

Title Suppressed Due to Excessive Length

information for a given dependency, the precision of these results critically depends on whether the PURL specifies an exact version. If the GitHub SBOM Tool does not include an exact version for a component, the resulting PURL is versionless. Figure 11 illustrates six different ways the component setuptools appears across the SBOMs. It can be observed that only when an exact version—77.0.3 in this case—is provided does the PURL reflect that version; in all other cases, where no version or a non-exact version is provided, the PURL is versionless. Submitting a versionless PURL to the osv.dev API returns all vulnerabilities associated with that component, regardless of version. For example, as shown in Figure 12, querying with the PURL pkg:pypi/[email protected] returns two vulnerabilities, enabling the software consumer to identify exactly which vulnerabilities affect their software. In contrast, querying with the versionless PURL pkg:pypi/setuptools returns six vulnerabilities, leaving the consumer uncertain about which vulnerabilities are relevant to their software.

7 Discussion In this section, we discuss four different aspects of our results. We first present an accuracy evaluation of the four SBOM tools against a manually constructed ground truth for Python, Java, and JavaScript (Section 7.1). We then compare the component agreement level between the GitHub SBOM Tool and the Microsoft SBOM Tool (Section 7.2), discuss the lack of component versioning in Python projects (Section 7.3), and examine the lack of license information across ecosystems (Section 7.4).

7.1 Accuracy Evaluation Against Ground Truth The analyses described in RQ2 measure the presence of metadata fields in generated SBOMs, but do not assess whether the reported values are correct. A tool may report an incorrect version or omit the license, undermining further compliance analysis, or fail to report components that the repository actually uses, introducing false negatives that leave real dependencies invisible to downstream consumers. To address this, we construct a ground truth dataset for Python, Java, and JavaScript, three languages whose package ecosystems provide programmatic access to authoritative dependency, version, and license metadata, enabling reliable automated ground truth. Given the time-consuming task of building projects to resolve their actual dependency tree, we randomly sampled 200 repositories per language from our full dataset, yielding 600 repositories in total. For constructing the ground truth, we resolve each repository’s dependencies programmatically from each ecosystem’s native tooling.

Bhuiyan et al.

Table 3 Component detection, version reporting accuracy, and license reporting accuracy evaluated against ground truth, macro-averaged across repositories. Best value per metric per language is shown in bold. Component Detection Precision

Component Detection Recall

Component Detection F1

Version Reporting Accuracy

License Reporting Accuracy

GitHub SBOM Tool Microsoft SBOM Tool Trivy Syft

33.7% 44.4% 20.3% 13.2%

57.3% 66.4% 32.3% 34.1%

29.0% 47.9% 19.8% 13.2%

21.6% 69.8% 27.4% 31.0%

19.7% 27.6% 1.7% 3.9%

Java

GitHub SBOM Tool Microsoft SBOM Tool Trivy Syft

46.8% 79.3% 55.7% 55.7%

38.2% 98.4% 54.9% 36.9%

35.9% 84.6% 50.5% 38.7%

53.7% 95.6% 81.9% 73.5%

28.5% 8.7% 60.2% 1.3%

JavaScript

GitHub SBOM Tool Microsoft SBOM Tool Trivy Syft

88.2% 87.9% 65.1% 63.8%

86.7% 80.3% 26.5% 40.0%

85.3% 81.1% 30.1% 41.2%

90.4% 88.8% 66.9% 69.2%

94.7% 78.5% 1.3% 25.6%

Language

Tool

Python

– For Python, following the methodology of Yu et al. [69], we install each repository’s dependencies into a fresh virtual environment using pip install and then capture the full resolved environment via pip freeze. This produces the complete set of transitive dependencies, each annotated with its exact pinned version. – For Java, we run mvn dependency:tree against each repository’s pom.xml, which triggers Maven’s dependency resolver and returns the fully resolved transitive dependency tree, including the concrete version selected for each artifact. – For JavaScript, we parse package-lock.json or yarn.lock, whichever is present in the repository, to extract the full set of installed packages together with their locked versions. During repository sampling, only repositories containing at least one of these two lockfiles were selected; repositories lacking both lockfiles were excluded and replaced by the next sampled candidate. For license ground truth, we query each ecosystem’s authoritative registry: the PyPI [46] API for Python; the npm [39] registry API for JavaScript; and the Libraries.io [58] API for Java, since Maven Central does not expose a structured REST API for license data and its POM <licenses> field is freeform text that is unsuitable for automated normalisation. We evaluate tools at the component detection level (Precision, Recall, F1 via normalized name matching) and the version accuracy and license accuracy levels (fraction of True Positives where the reported value exactly matches the ground truth, macro-averaged across repositories). For version accuracy, we require an exact match between the version reported in the SBOM and the version present in the ground truth. A tool that reports a version range or a resolved value that differs from the ground truth is considered incorrect. Table 3 presents the component detection, version accuracy, and license accuracy results evaluated against our ground truth dataset across Python, Java, and JavaScript. While RQ2 consistently positioned the GitHub SBOM

Title Suppressed Due to Excessive Length

Tool as the leading tool in terms of component count and metadata presence, the accuracy evaluation reveals a more nuanced picture. For component detection, the GitHub SBOM Tool’s higher component count in Python and Java does not translate into higher accuracy against the ground truth; the Microsoft SBOM Tool, despite reporting fewer components, achieves better precision, recall, and F1 in both ecosystems. For version accuracy, the Microsoft SBOM Tool’s near-complete version presence observed in RQ2 does carry over into correctness: it leads on version accuracy in two of the three ecosystems, confirming that it not only populates version fields consistently but also reports the right values. For license accuracy, three different tools lead in three ecosystems. In Python, the Microsoft SBOM Tool leads on all component detection metrics, achieving a Precision of 44.4%, Recall of 66.4%, and F1 of 47.9%, alongside the highest version accuracy of 69.8% and license accuracy of 27.6%. This is somewhat surprising given that RQ2’s Finding 1 (Section 5.2) identified the GitHub SBOM Tool as detecting more components than the Microsoft SBOM Tool in Python in terms of raw component counts. However, a higher component count does not imply higher accuracy: the GitHub SBOM Tool surfaces more components overall, but fewer of them match the ground truth, resulting in lower component detection performance than the Microsoft SBOM Tool. Version accuracy is low across all tools except the Microsoft SBOM Tool, ranging from 21.6% (GitHub SBOM Tool) to 31.0% (Syft). The Microsoft SBOM Tool’s relatively high version accuracy of 69.8% suggests that it resolves undeclared or range-based versions more reliably than other tools while the other tools may provide range-based versions or wrong resolved value. License accuracy is uniformly poor across all tools, with no tool exceeding 27.6%. In Java, the Microsoft SBOM Tool dominates component detection with a Precision of 79.3%, Recall of 98.4%, and F1 of 84.6%, and achieves the highest version accuracy at 95.6%. RQ2’s Finding 4 showed that Microsoft SBOM Tool provides version information for every detected component across nearly all ecosystems, and the 95.6% version accuracy confirms that these reported versions are also almost always correct for Java projects. The GitHub SBOM Tool performs considerably weaker in component detection (F1 of 35.9%) and version accuracy (53.7%), consistent with our earlier observation in RQ1’s Finding 2 (Section 4) that GitHub SBOMs omit transitive dependencies of unversioned parent components. Notably, the Microsoft SBOM Tool’s dominant component detection performance does not extend to license accuracy, where it achieves only 8.7% compared to GitHub SBOM Tool’s 28.5% or Trivy’s 60.2%. In JavaScript, the GitHub SBOM Tool leads across all metrics, achieving a Precision of 88.2%, Recall of 86.7%, F1 of 85.3% for component detection, version accuracy of 90.4%, and license accuracy of 94.7%. The strong version accuracy of the GitHub SBOM Tool in JavaScript suggests that it benefits from the availability of lockfiles, which provide exact pinned versions and remove the need to resolve range-based declarations. The Microsoft SBOM Tool is competitive across component detection and version accuracy (81.1%

Bhuiyan et al.

and 88.8% respectively) but falls more behind on license accuracy (78.5%). The license accuracy results for both tools are consistent with RQ2’s Finding 5 (Section 5.2), which identified JavaScript as one of only two languages where the GitHub SBOM Tool achieves above 80% license presence and Microsoft SBOM Tool also reports similar performance for JavaScript projects. Our ground truth analysis confirms that the reported licenses are not only present but also largely correct. Trivy and Syft show sharply lower performance in component detection, particularly on recall (26.5% and 40.0%), confirming that they detect a selective subset of the full dependency tree in this ecosystem.

7.2 GitHub SBOM vs MS SBOM Generation tool Among the SBOM generation tools evaluated, both the Microsoft SBOM Tool and the GitHub SBOM Tool demonstrated reliable performance in providing component version and license information (RQ2) and PURL information (RQ3). However, high completeness scores do not preclude the possibility that the two tools are detecting substantially different sets of dependencies. To examine whether the tools also agree on the components they report, we perform a component-level agreement analysis. For the component-level comparison, each component’s PURL is normalized by removing the version from the PURL, causing multiple versions of the same component to collapse into a single entry; we refer to these as unique components. For the versioned comparison, each entry is keyed by the pair (versionless PURL, versionInfo), such that the same component appearing with different versions is treated as distinct entries; we refer to these as unique versioned components. Packages lacking version information are excluded from the versioned comparison. Table 4 shows the percentage of components that are uniquely reported in GitHub SBOMs (GH), reported in both GitHub and Microsoft SBOMs (GH ∩ MS), and uniquely reported to Microsoft (MS). In this analysis, we first summed all three component categories per project and computed the percentage of the aggregated sum across all unique components found in both tools’ SBOMs. Overall, in most languages GitHub SBOM Tool and Microsoft SBOM Tool tend to agree on the majority of unique components. The highest agreement is observed in Go (76.8%), JavaScript (75.0%), and Rust (72.8%) projects, suggesting strong consistency. At the same time, GitHub tends to identify more components in 8 out of 10 languages, with Java and Rust projects being the only exceptions. In contrast, PHP exhibits the lowest overlap (43.0%), where more than half of the components (50.5%) are reported exclusively by GitHub, indicating substantial divergence between the tools. When accounting for versioned components, i.e., whether both GitHub SBOM Tool and Microsoft SBOM Tool report the same component and their versions, the agreement decreases. For example, in Go projects, agreement drops from 76.8% (components) to 32.6% (versioned components), with both

Title Suppressed Due to Excessive Length

Table 4 Agreement level between GitHub SBOM Tool (GH) and the Microsoft SBOM Tool (MS). We highlight in light grey values above 25% and in dark grey values above 50%. Languages

Agreement Level Unique Components Unique Versioned Components GH GH∩MS MS GH GH∩MS MS

C C# C++ Go Java JavaScript PHP Python Rust Swift

28.3% 38.2% 27.5% 13.0% 17.6% 15.1% 50.5% 24.2% 13.2% 20.5%

52.5% 52.7% 53.7% 76.8% 49.2% 75.0% 43.0% 57.9% 72.8% 66.1%

19.1% 9.1% 18.8% 10.3% 33.1% 9.9% 6.5% 17.9% 14.0% 13.3%

33.4% 42.4% 33.4% 34.0% 22.8% 22.4% 51.8% 29.6% 29.1% 23.2%

43.9% 43.3% 41.7% 32.6% 36.1% 62.4% 36.4% 45.9% 44.6% 60.0%

22.8% 14.3% 24.9% 33.4% 41.1% 15.2% 11.8% 24.5% 26.3% 16.8%

tools reporting roughly one-third of versioned components exclusively. A similar decline is observed in Java and C++ projects, suggesting that version resolution and normalization strategies differ significantly. Java, in particular, shows a notable asymmetry, with Microsoft reporting 41.1% of versioned components exclusively compared to 22.8% for GitHub. These results indicate that while the tools often agree on the presence of dependencies at the component level, discrepancies become more pronounced when version information is incorporated. 7.3 Potential causes for lack of versioning of Python dependency components. From the results of RQ2, we observe that GitHub SBOM Tool provides low support for component versioning in Python projects, with an average of 80% of components per project being versioned. To understand the potential causes, we automatically analyzed the requirements.txt files of all Python projects in our dataset. Python allows developers to import and use a library without specifying it in any metadata file, as long as it is installed in the runtime environment. Furthermore, Python projects offer multiple mechanisms for specifying dependencies, including requirements.txt, setup.py, pyproject.toml, and Pipfile, among others — though our analysis focused on requirements.txt files, which are the most widely used convention for specifying installation requirements. Among the 9,575 PyPI library instances missing version information in GitHub SBOM Tool-generated SBOMs, we identified distinct causes. – Lack of version specifiers. In 5,961 cases (62.26%), the libraries were present in requirements.txt files but declared without any version specifier, which directly propagates to the SBOM as an unversioned entry. – Absent libraries in the manifest file. In the remaining 3,614 cases (37.74%), the libraries were absent from requirements.txt entirely, attributable to one of three scenarios: the dependency may be declared in an alternative metadata file not covered by our analysis; it may be a transitive

Bhuiyan et al.

Table 5 License Availability Across Package Managers Based on Version Specification. Package Manager

pypi cargo githubactions maven nuget npm golang gem composer pub swift github deb

Total Dependencies 82,191 353,825 58,953 114,587 43,206 2,708,803 128,770 25,048 41,970 3,333 3,854 208 589

Exact Version Dependencies Licensed % 54,647 319,231 58,906 74,150 31,809 2,647,719 128,770 24,258 30,041 2,872 3,854 208 589

86.20% 98.71% 0.00% 48.21% 66.56% 99.18% 0.00% 97.34% 41.76% 0.00% 0.00% 0.00% 0.00%

Non-Exact Version Dependencies Licensed % 12,679 34,594 47 226 9,556 61,070 0 790 11,501 461 0 0 0

0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%

No Version Dependencies Licensed % 14,865 0 0 40,211 1,841 14 0 0 428 0 0 0 0

0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%

dependency resolved by GitHub from the full dependency tree; or it may not be declared anywhere at all, with GitHub SBOM Tool surfacing it in the SBOM through static analysis of import statements in the source code. These findings suggest that the version-adherence problem in GitHub SBOMs is primarily due to loose dependency-declaration practices in the Python ecosystem rather than a limitation of the SBOM generation tool itself. 7.4 Why is licensing information so seldom reported in SBOMs? Our results show that component license information is seldom reported in SBOMs, across all tools. This may indicate that poor license coverage is not just a tool limitation, but likely reflects a limitation on how license metadata is resolved. We break the analysis of the license into two orthogonal factors: 1) dependency versioning practices and 2) the related package manager ecosystem. To this aim, we classify every dependency exported across all SBOMs analyzed based on their reported version into three categories: – Exact version. We classify a dependency as having an exact version if the version is expressed in the SBOM as a unique version, without ranges or semantic versioning, a.k.a., dependency pinning. – Non-exact version. We classify a dependency’s version as Non-exact if it is expressed as a range, a wildcard, or using semantic versioning (semver) operators. – No version. Dependencies where no version is specified in the SBOM. Table 5 presents license availability across package managers, broken down by component version strategy [23]. We observe a clear pattern across all package managers: dependencies reported with an exact version are far more likely to include license information, whereas those with missing or non-exact versions consistently lack versions. We also observe that exact versioning alone does not guarantee license availability. For dependencies managed by the package managers swift, pub, deb, golang, github, and githubactions, the tool

Title Suppressed Due to Excessive Length

Table 6 Explanations for lack of license information Language

# Unlicensed

Non-Exact/ No Version (E1)

GH Actions (E2)

Golang (E3)

Pub (E4)

Swift (E5)

GitHub (E6)

Deb (E7)

Unaccounted

C C# C++ Go Java JavaScript PHP Python Rust Swift

11360 30050 19344 135288 96736 44342 38607 40260 60916 7025

3101 (27.3%) 13953 (46.4%) 7891 (40.8%) 4135 (3.1%) 47195 (48.8%) 28766 (64.9%) 14569 (37.7%) 23634 (58.7%) 44691 (73.4%) 348 (5.0%)

4779 (42.1%) 4660 (15.5%) 6584 (34.0%) 9713 (7.2%) 5805 (6.0%) 4219 (9.5%) 4582 (11.9%) 7423 (18.4%) 8909 (14.6%) 2232 (31.8%)

1869 (16.5%) 182 (0.6%) 1728 (8.9%) 119099 (88.0%) 1868 (1.9%) 1573 (3.5%) 485 (1.3%) 988 (2.5%) 899 (1.5%) 79 (1.1%)

637 (5.6%) 92 (0.3%) 412 (2.1%) 140 (0.1%) 1022 (1.1%) 0 (0.0%) 0 (0.0%) 337 (0.8%) 18 (0.0%) 214 (3.0%)

15 (0.1%) 13 (0.0%) 72 (0.4%) 8 (0.0%) 26 (0.0%) 16 (0.0%) 0 (0.0%) 2 (0.0%) 32 (0.1%) 3670 (52.2%)

0 (0.0%) 16 (0.1%) 191 (1.0%) 0 (0.0%) 1 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%)

0 (0.0%) 0 (0.0%) 220 (1.1%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 369 (0.9%) 0 (0.0%) 0 (0.0%)

959 (8.4%) 11134 (37.1%) 2246 (11.6%) 2193 (1.6%) 40819 (42.2%) 9768 (22.0%) 18971 (49.1%) 7507 (18.6%) 6367 (10.5%) 482 (6.9%)

fails to retrieve license information for all dependencies, regardless of the versioning strategy. Based on this, we examined all unlicensed dependencies per language and attributed each to one of the following probable causes: – Explanation-1 (E1): Dependency is found in the SBOM having a nonexact version or having no version, e.g., dependencies declared using semantic versioning. – Explanation-2 (E2): Dependency is listed in the SBOM with a pkg: githubactions type, indicating it is a GitHub Action referenced within the project’s CI/CD pipeline. – Explanation-3 (E3): Dependency appears in the SBOM with a pkg: golang type, indicating it is a Golang component. – Explanation-4 (E4): Dependency is listed in the SBOM with a pkg:pub type, indicating it is a Pub component. – Explanation-5 (E5): Dependency is found in the SBOM with a pkg: swift type, indicating it is a Swift component. – Explanation-6 (E6): Dependency appears in the SBOM with a pkg: github type, indicating it is a component sourced directly from a GitHub repository rather than a formal registry. – Explanation-7 (E7): Dependency is listed in the SBOM with a pkg:deb type, indicating it is a Debian system-level component. Table 6 presents the result of our analysis across all ten languages. Whenever possible, each dependency was categorized in one of the explanations. Dependencies that could not be attributed to any of the identified explanations are reported as Unaccounted. The most pervasive cause of missing license information is a non-exact or absent version specification (E1). In 8 out of 10 languages, E1 is the single largest contributor to unlicensed dependencies. Among dependencies that do have an exact version, missing licenses are explained by membership in package managers for which the GitHub SBOM Tool does not retrieve license information (E2–E7). The degree to which these causes account for the remaining missing licenses varies substantially across languages. For Go, Golang dependencies (E3) alone account for 88.0% of all unlicensed dependencies, leaving only 1.6% unaccounted for. Swift is similarly well-explained, with Swift dependencies (E5) and GitHub Actions (E2) together covering 84.0% of missing licenses. In contrast, Java and PHP prove harder to explain: even after accounting for all identified causes, 42.2% and 49.1% of their unlicensed depen-

Bhuiyan et al.

dencies, respectively, remain unaccounted for — meaning these dependencies carry exact version information and belong to supported package managers, yet the GitHub SBOM Tool still fails to retrieve their license metadata. This points to gaps in the upstream registries the tool relies upon, and represents an important avenue for future improvement. 8 Related Works In this section, we discuss the related works. 8.1 Empirical Studies on SBOM. Empirical research on SBOM adoption has provided useful insights into the practices and challenges observed across industry and open-source ecosystems. Xia et al. conducted an empirical study based on interviews with practitioners and a survey of professionals to examine current SBOM practices, available tool support, and major concerns surrounding SBOM usage [66]. Their findings indicate limited adoption in practice. Even among projects that generate SBOMs, there is no clear consensus on what information to include, despite official recommendations and guidelines, underscoring the need for clearer expectations and stronger standardization. These observations are consistent with evidence reported in the Linux Foundation’s SBOM readiness survey, which analyzed responses from 412 organizations worldwide [22]. The survey revealed notable gaps in organizational familiarity with SBOMs, as well as uncertainty regarding SBOM production and consumption strategies. In addition, many respondents expressed skepticism about whether SBOM requirements are being broadly adopted across the software industry, raising concerns about alignment with emerging regulatory and policy initiatives. Beyond adoption awareness, Xia et al. further identified significant limitations in existing SBOM tooling, particularly for SBOM consumption and vulnerability handling, underscoring the need for more robust, interoperable, and enterprise-ready solutions [66]. The authors also emphasized the importance of validation and verification mechanisms to ensure the reliability and trustworthiness of SBOM data, especially given the risk of incomplete or manipulated metadata. Together, these findings highlight the need for continued improvements in SBOM tooling and practices to support meaningful adoption. Nocera et al. focused specifically on open-source projects hosted on GitHub, uncovering a growing trend in SBOM adoption [38]. Despite this increase, the overall uptake remains limited, particularly in smaller projects. Their comparison of SPDX and CycloneDX formats highlighted that while SPDX is widely adopted, CycloneDX offers superior support for security-related features, which is a critical consideration for projects prioritizing security. One of the closest related works is the work of Yu et al., who compared the quality of SBOM generation across GitHub Dependency Graph, Syft, Trivy,

Title Suppressed Due to Excessive Length

Table 7 Comparison of our study with the two most closely related empirical SBOM studies. ✗ denotes not addressed. Dimension

Yu et al. [69] DSN 2024

Wang et al. [63] ACM TOSEM 2026

This Work Under submission

Study scale

7,876 repos; 9 languages; 4 tools ✗

3,287 repos; 3 languages; 6 tools extended NTIA field set

version only; pinned vs. unpinned component counts + Jaccard similarity Python only

version, license, PURL; field-level per tool triple-factor matching; field-level consistency Python only; 100 repos

10,000 repos; 10 languages; 4 tools all 7 NTIA minimum elements, per language version, license, PURL; top-level & transitive, per tool & language counts, version, license; Kruskal-Wallis + Dunn per language Python, Java, JS; 200 repos each

✗

parser limitations & metadata file constraints

standard ambiguity, scope definition

SBOM compliance assessment Metadata completeness analysis Inter-tool comparison Ground-truth accuracy evaluation Vulnerability tracking utility

Root-cause analysis of missing metadata

PURL-based via OSV.dev; effect of version precision on detection, per tool & language 7 causes at package manager level

and Microsoft SBOM Tool [69]. The study covered 7,876 open-source repositories spanning nine programming languages. The authors compared the tools’ outputs by examining the number of detected packages, measuring pairwise overlap using Jaccard similarity over (name, version) dependency sets, and quantifying the presence of duplicate packages within individual SBOMs. Although this work is similar to ours, their analysis focused primarily on how different tools vary in reporting the number of dependencies in SBOMs, whereas we focused on how the tools differ in reporting key metadata, such as component version, licensing, and compliance with NTIA requirements. Another close work to ours is Wang et al. [63]. They present an empirical study of 3,287 repositories across four languages (C, C++, Java, and Python), focusing on three dimensions: structural compliance with SBOM standards, inter-tool consistency, and information accuracy. Consistent with our results, they found inadequate compliance with policy requirements, poor inter-tool consistency, and low license reporting rates. Our study is complementary in multiple ways. We cover more programming languages, which enables us to draw parallels across other ecosystems. We also include an in-depth investigation into the potential root causes of missing license information across ecosystems and evaluate the practical utility of SBOMs for automated vulnerability tracking. The work of Wang et al. [63] was published recently, and was conducted concurrently with our study. We believe that both studies offer complementary perspectives on the state of SBOM tooling. Table 7 summarises the key dimensions along which our study differs from the two most directly related works, [69] and [63]. The table highlights that while all three studies share a concern for SBOM quality and inter-tool comparison, each addresses a distinct set of research questions, and our work uniquely combines the broadest language coverage, a focus on the GitHub-native SBOM Tool, end-to-end vulnerability tracking evaluation, and a systematic root-cause analysis of missing metadata at the package-manager level.

Bhuiyan et al.

8.2 Compliance with NTIA Requirements. Compliance with NTIA requirements is a pivotal aspect of SBOM implementation, and several studies have explored the associated challenges and benefits. Zahan et al. identified the top five benefits and challenges of aligning SBOMs with NTIA guidelines [70]. Their findings suggest that while SBOMs significantly enhance transparency and security, the complexity of achieving compliance poses a substantial hurdle. Similarly, Torres et al. conducted a study through which multiple SBOM generating tools were assessed according to NTIA requirements to see how compliant they are with these latter [61]. Only 1% of SBOMs coming from the assessed Docker images are fully NTIA compliant. Nocera et al. analyzed 119 SBOMs—89 using CycloneDX and 30 using SPDX—spanning 84 projects and 22 owners [37]. In their assessment, the authors noted that the analyzed SPDX SBOMs contained all NTIA minimum data fields at least once, with the supplier name and dependency relationship appearing in only a single SBOM. Based on this, they concluded that 3% (1/30 × 100) of the SPDX SBOMs satisfied all NTIA minimum data fields.

8.3 Challenges in SBOM Implementation. Research into the challenges of SBOM implementation has identified a range of technical and organizational obstacles. Stalnaker et al. detailed 12 major challenges in creating and using SBOMs, including tool deficiencies, domainspecific challenges, and difficulties in integrating SBOMs into existing workflows [53]. These findings are echoed by Torres-Arias et al., who argue that the development of quality measurement mechanisms for SBOMs is crucial for enhancing their effectiveness [61]. The study emphasizes that without robust mechanisms to assess SBOM quality, their potential benefits in security and transparency may not be fully realized. Dalia et al. published a paper through which they conducted an analysis of the challenges holding back SBOM adoption [12]. It further conducted a comparative analysis between multiple SBOM generation tools based on the features considered, based on the challenges initially stated in the paper. Bi et al. proposed a study through which the first comprehensive classification of SBOM-relevant issues and potential solutions has been provided, along with an identification of the different phases of the SBOM lifecycle and their characteristics [5]. In this classification, issues were correlated with various stages of the SBOM lifecycle, offering a structured analysis that serves as a guide for developers. This analysis aids in understanding the challenges associated with SBOMs and offers recommendations for effectively integrating SBOMs to address development issues in real-world scenarios. Additionally, the study identifies gaps in the existing production and usage of SBOMs, highlighting shortcomings in current practices. Based on these insights, the authors suggest future research directions aimed at im-

Title Suppressed Due to Excessive Length

proving SBOM quality and adoption, providing a roadmap for advancing the field.

8.4 Impact of SBOMs on Software Security and Transparency. SBOMs are increasingly recognized for their role in improving software supply chain security and transparency. Xia et al. explored the integration of blockchain technology into SBOMs as a means of enhancing security by providing a tamper-resistant mechanism for SBOM sharing [67]. This study suggests that blockchain can significantly improve the trustworthiness of SBOMs, particularly in environments where security is paramount. Similarly, Carmody et al. examined the potential of SBOMs in the medical technology supply chain, highlighting their ability to improve transparency and trust among stakeholders [7]. Sharma et al. introduced SBOM.exe, a tool designed to detect the malicious usage of dynamic features in Java by verifying the binary integrity of an application’s dependencies at runtime [51].

9 Threats to Validity Generalization of the results. Our study’s findings may not generalize to all software projects or SBOM tools. We generated 10,000 SBOMs across 10 programming languages using the GitHub SBOM tool and the three other popular tools: the Microsoft SBOM, Syft and Trivy. While this covers a variety of languages and tooling approaches, it still represents only a subset of the broader software ecosystem. Projects outside our selection—especially those in different domains or using different dependency management practices—might show different patterns. Moreover, we only considered open-source projects hosted on GitHub. Proprietary software or projects hosted elsewhere may have different dependency characteristics, so caution is needed when extending our conclusions to those contexts. Future studies including a wider range of languages, tools, and project types are needed to confirm whether our observations hold more generally. Internal threat to validity. Potential mismatches in SBOMs could arise if different tools analyze different snapshots or branches of a repository, or if their internal generation mechanisms differ. To control for this, Microsoft SBOM Tool, Trivy, and Syft were executed on locally cloned repositories, using a snapshot of each repository’s default branch to ensure identical source code, dependency manifests, and project structures. In contrast, GitHub SBOMs were obtained via the GitHub REST API; although these are also generated from the default branch, the internal mechanisms remain opaque and may differ from those of the other tools. It is also important to note that the Microsoft SBOM Tool did not successfully generate SBOMs for 254 out of 10,000 projects (2.54%). These failures were distributed across multiple programming

Bhuiyan et al.

languages, indicating that they were not concentrated in a single ecosystem. While this represents a small proportion of the overall dataset, the excluded repositories may slightly affect the completeness of the comparison. Nevertheless, the large remaining sample provides broad coverage across languages and helps mitigate the impact of these isolated tool limitations. Reproducibility limitations. Reproducibility in our study differs across tools due to the nature of how SBOMs are generated. The GitHub REST API does not support generating SBOMs for historical states of a repository, making it impossible to recreate the same GitHub-generated SBOMs at a later time. However, our replication package includes all GitHub-generated SBOMs, which can be used directly for reproducibility purposes. In contrast, for Microsoft SBOM Tool, Trivy, and Syft, SBOMs can be regenerated from the same locally cloned repository snapshot, ensuring consistent results across repeated analyses. Each generated SBOM records its creation timestamp, further supporting reproducibility for these three tools. Conclusion validity. Throughout this study, our research questions are framed primarily around programming languages, as we collected repositories labeled under ten different languages on GitHub. However, as discussed in Section 7.4, real-world projects are rarely confined to a single language: most repositories are polyglot in nature, comprising components written in multiple languages and managed by different package managers [60, 27]. This means that the SBOMs we analyze for a given language may contain packages drawn from entirely different ecosystems, and the results we attribute to a language may in part reflect the behavior of those other ecosystems rather than the primary one. Practitioners and researchers should therefore interpret our per-language conclusions with this caveat in mind, as they represent the experience of repositories labeled as a given language, not of purely single-language projects.

10 Conclusion This study examined the quality of automatically generated SBOMs at scale, using GitHub’s native SBOMs as a representative case and comparing them with widely used third-party tools. Our results show that while automated SBOM generation has become easier with the availability of multiple tools, important gaps remain that limit its effectiveness for security and compliancedriven use cases. In particular, the absence of supplier information, the incomplete and uneven version reporting introduces blind spots that propagate to transitive dependencies and directly weaken vulnerability and license analysis. Our findings demonstrate that missing or non-exact versions force downstream tools to either over-approximate vulnerabilities or ignore affected components entirely. The comparison across SBOM tools further indicates that no single approach currently offers comprehensive coverage across languages and metadata dimensions. GitHub’s SBOMs provide broad dependency visibility and strong

Title Suppressed Due to Excessive Length

identifier support, while the Microsoft SBOM Tool prioritizes version completeness at the cost of coverage. The variability observed across ecosystems suggests that SBOM quality is shaped as much by language-specific tooling and conventions as by the SBOM generator itself. Overall, these results suggest that automated SBOMs are best viewed as a baseline rather than a complete solution. Improving supplier attribution, version propagation, and license extraction will be critical for SBOMs to fully support large-scale vulnerability management and supply-chain risk assessment. Our findings highlight concrete areas where SBOM tooling and standards can evolve to better align with their intended role in securing modern software supply chains.

Declarations Funding We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC)[funding reference number RGPIN2023-05163]. Ethical Approval Not applicable. Informed Consent Not applicable. Author Contributions Conceptualization: Kawsar Ahmed Bhuiyan, Diego Elias Costa. Methodology: Kawsar Ahmed Bhuiyan, Mohamed Bilel Besbes, Rachna Raj, Diego Elias Costa. Data Collection: Kawsar Ahmed Bhuiyan, Adam Al Assil. Analysis: Kawsar Ahmed Bhuiyan, Mohamed Bilel Besbes. Writing - original draft preparation: Kawsar Ahmed Bhuiyan, Adam Al Assil. Writing - review and editing: Mohamed Bilel Besbes, Rachna Raj, Diego Elias Costa. Supervision: Diego Elias Costa. Data Availability Statement The data and code used in this study are available at https://doi.org/10.5281/zenodo.18883005. Conflict of Interest Not applicable. Clinical Trial Number Not applicable.

References 1. Anchore. Syft: Sbom generation tool. https://anchore.com/opensource/syft/, 2025. Accessed: 2025-11-25. 2. Anchore. Syft: Supported sources. https://github.com/anchore/syft/wiki/support ed-sources, 2025. Accessed: 2025-11-25. 3. Aqua Security. Trivy documentation: Repository scanning. https://trivy.dev/docs /latest/target/repository/, 2025. Accessed: 2025-11-25. 4. Aqua Security. Trivy: Open source vulnerability and sbom scanner. https://trivy.de v/, 2025. Accessed: 2025-11-25. 5. T. Bi, B. Xia, Z. Xing, Q. Lu, and L. Zhu. On the way to sboms: Investigating design issues and solutions in practice. ACM Trans. Softw. Eng. Methodol., 33(6), June 2024.

Bhuiyan et al.

6. H. Borges, A. Hora, and M. T. Valente. Understanding the factors that impact the popularity of github repositories. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 334–344, 2016. 7. S. Carmody, A. Coravos, G. Fahs, A. Hatch, J. Medina, B. Woods, and J. Corman. Building resilient medical technology supply chains with a software bill of materials. NPJ Digital Medicine, 4, 2021. 8. A. Chaora, N. Ensmenger, and L. J. Camp. Discourse, challenges, and prospects around the adoption and dissemination of software bills of materials (sboms), 2023. 9. ClearlyDefined. ClearlyDefined: Helping foss projects be more clearly defined. https: //clearlydefined.io/, 2025. Accessed: 2025. 10. Cybersecurity and Infrastructure Security Agency. SBOM FAQ 2024. Technical report, U.S. Department of Homeland Security, July 2024. Accessed: 2026-02-10. 11. O. Dabic, E. Aghajani, and G. Bavota. Sampling projects in github for msr studies. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 560–564, 2021. 12. G. Dalia, C. A. Visaggio, A. Di Sorbo, and G. Canfora. Sbom ouverture: What we need and what we have. In Proceedings of the 19th International Conference on Availability, Reliability and Security, ARES ’24, New York, NY, USA, 2024. Association for Computing Machinery. 13. A. Decan, T. Mens, and P. Grosjean. An empirical comparison of dependency network evolution in seven software packaging ecosystems. Empirical Softw. Engg., 24(1):381–416, Feb. 2019. 14. A. Decan, T. Mens, P. R. Mazrae, and M. Golzadeh. On the use of github actions in software development repositories. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 235–245, 2022. 15. O. J. Dunn. Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64, Mar. 1961. 16. W. Enck and L. Williams. Top five challenges in software supply chain security: Observations from 30 industry and government organizations. IEEE Security & Privacy, 20(2):96–100, 2022. 17. E. Frankford, T. Antensteiner, M. Vierhauser, C. Sauerwein, V. Wallner, I. Groher, R. Plösch, and R. Breu. A survey on feedback types in automated programming assessment systems. ACM Trans. Comput. Educ., 26(1), Dec. 2025. 18. GitHub. Octoverse 2024: The most popular programming languages. https://gith ub.blog/news-insights/octoverse/octoverse-2024/, 2024. Published: 2024-10-29; updated: 2025-10-28; Accessed: 2025-12-24. 19. GitHub. Exporting a software bill of materials for your repository. https://docs.git hub.com/en/code-security/supply-chain-security/understanding-your-softwar e-supply-chain/exporting-a-software-bill-of-materials-for-your-repository, 2025. Accessed: 2025-07-03. 20. GitHub. Rest api: Sboms. https://docs.github.com/en/rest/dependency-graph/sbo ms?apiVersion=2022-11-28, 2025. Accessed: 2025-07-10. 21. GitHub. What is an sbom (software bill of materials)? https://github.com/resou rces/articles/what-is-an-sbom-software-bill-of-materials, 2025. Accessed: 2026-02-10; Published: September 5, 2025. 22. S. Hendrick. The state of software bill of materials (sbom) and cybersecurity readiness. Research report, The Linux Foundation, 2022. Foreword by Jim Zemlin. 23. A. Javan Jafari, D. E. Costa, R. Abdalkareem, E. Shihab, and N. Tsantalis. Dependency smells in javascript projects. IEEE Transactions on Software Engineering, 48(10):3790– 3807, 2022. 24. E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. Germán, and D. Damian. The promises and perils of mining github. In Proceedings of the 11th Working Conference on Mining Software Repositories (MSR). ACM/IEEE, 2014. 25. W. H. Kruskal and W. A. Wallis. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260):583–621, 1952. 26. P. Ladisa, H. Plate, M. Martinez, and O. Barais. SoK: Taxonomy of Attacks on OpenSource Software Supply Chains . In 2023 IEEE Symposium on Security and Privacy (SP), pages 1509–1526, Los Alamitos, CA, USA, May 2023. IEEE Computer Society.

Title Suppressed Due to Excessive Length

27. W. Li, L. Li, and H. Cai. How are multilingual systems constructed: Characterizing language use and selection in open-source multilingual software. ACM Transactions on Software Engineering and Methodology, 33(3), 2023. 28. J. Liu, Q. Huang, X. Xia, et al. An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks. Empirical Software Engineering, 26(1):16, 2021. 29. F. Massacci and I. Pashchenko. Technical leverage in a software ecosystem: Development opportunities and security risks. In Proceedings of the 43rd International Conference on Software Engineering, ICSE ’21, page 1386–1397. IEEE Press, 2021. 30. A. Mastropaolo, M. Ciniselli, L. Pascarella, R. Tufano, E. Aghajani, and G. Bavota. Towards summarizing code snippets using pre-trained transformers. In Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, ICPC ’24, page 1–12, New York, NY, USA, 2024. Association for Computing Machinery. 31. Microsoft. Microsoft sbom tool. https://github.com/microsoft/sbom-tool, 2025. Accessed: 2025-11-25. 32. Microsoft. Sbom tool: Download and installation. https://github.com/microsoft /sbom- tool?tab=readme- ov- file#download- and- installation, 2025. Accessed: 2025-11-25. 33. A. M. Mir, M. Keshani, and S. Proksch. On the effect of transitivity and granularity on vulnerability propagation in the maven ecosystem. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 201– 211, 2023. 34. National Institute of Standards and Technology. Executive order 14028, improving the nation’s cybersecurity, 2022. Accessed: 2025-07-04. 35. National Telecommunications and Information Administration. Software bill of materials (sbom) overview. U.S. Department of Commerce, NTIA, Aug. 2020. PDF available at https://www.ntia.gov/sites/default/files/publications/sbom_overview_2020 0818_0.pdf, accessed 2025-07-03. 36. National Telecommunications and Information Administration. The minimum elements for a software bill of materials (sbom). Technical report, U.S. Department of Commerce, July 2021. 37. S. Nocera, S. Romano, M. Di Penta, R. Francese, and G. Scanniello. On the adoption of software bill of materials in open-source software projects. Journal of Systems and Software, 230:112540, 2025. 38. S. Nocera, S. Romano, M. D. Penta, R. Francese, and G. Scanniello. Software bill of materials adoption: A mining study from github. 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 39–49, 2023. 39. npm, Inc. npm. https://www.npmjs.com/, 2026. Accessed: 2026-06-01. 40. F. W. G. NTIA Multistakeholder Process on Software Component Transparency. Framing software component transparency: Establishing a common software bill of material (sbom). Technical report, Nov. 2019. 41. NTIA Multistakeholder Process on Software Component Transparency Standards and Formats Working Group. Survey of existing sbom formats and standards. Technical report, National Telecommunications and Information Administration (NTIA), 2021. 42. M. Ohm, H. Plate, A. Sykosch, and M. Meier. Backstabber’s knife collection: A review of open source software supply chain attacks. In Detection of Intrusions and Malware, and Vulnerability Assessment: 17th International Conference, DIMVA 2020, Lisbon, Portugal, June 24–26, 2020, Proceedings, page 23–43, Berlin, Heidelberg, 2020. Springer-Verlag. 43. OSV Project Maintainers. Osv.dev – open source vulnerabilities database. https: //osv.dev/, 2025. Accessed: 2025-11-24. 44. OX Security. Top 5 sbom tools for securing the software supply chain. https://www. ox.security/blog/sbom-tools/, 2025. Accessed: 2026-01-21. 45. M. D. Papamichail, T. Diamantopoulos, V. Matsoukas, C. Athanasiadis, and A. L. Symeonidis. Towards extracting the role and behavior of contributors in open-source projects. In Proceedings of the 14th International Conference on Software Technologies (ICSOFT 2019), pages 536–543, 2019. 46. Python Software Foundation. Python package index (pypi). https://pypi.org/, 2026. Accessed: 2026-06-01.

Bhuiyan et al.

47. J. Romeo, M. Raglianti, C. Nagy, and M. Lanza. Uml is back. or is it? investigating the past, present, and future of uml in open source software. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 2342–2354, 2025. 48. H. Samaana, D. E. Costa, A. Abdellatif, and E. Shihab. Opportunities and security risks of technical leverage: A replication study on the npm ecosystem. Empirical Software Engineering, 30(4), Apr. 2025. 49. SEART Research Group. Seart github search (ghs). https://seart-ghs.si.usi.ch/, 2021. Accessed: 2025-12-24. 50. S. Sham. The top 11 open-source sbom tools. https://www.wiz.io/academy/applica tion-security/top-open-source-sbom-tools, Dec 2025. Accessed 2026-01-21. 51. A. Sharma, M. Wittlinger, B. Baudry, and M. Monperrus. Sbom.exe: Countering dynamic code injection based on software bill of materials in java, 2024. 52. Stack Overflow. Stack overflow developer survey 2025: Technology. https://survey.s tackoverflow.co/2025/technology, 2025. Accessed: 2025-12-24. 53. T. Stalnaker, N. Wintersgill, O. Chaparro, M. D. Penta, D. M. German, and D. Poshyvanyk. Boms away! inside the minds of stakeholders: A comprehensive study of bills of materials for software systems. ArXiv, abs/2309.12206, 2023. 54. The Linux Foundation. What is an sbom? https://www.linuxfoundation.org/blog/b log/what-is-an-sbom, June 2021. Accessed: 2026-02-10. 55. The Linux Foundation. Spdx 2.3 document overview. https://spdx.dev/learn/overv iew/spdx-2-3-document/, 2025. Accessed: 2025-07-08. 56. The Linux Foundation. Spdx overview. https://spdx.dev/about/overview/, 2025. Accessed: 2025-07-03. 57. The White House. Executive order on improving the nation’s cybersecurity, May 2021. Accessed: 2025-07-04. 58. Tidelift, Inc. Libraries.io. https://libraries.io/, 2026. Accessed: 2026-06-01. 59. TIOBE Software. Tiobe programming community index. https://www.tiobe.com/ti obe-index/, 2025. Accessed: 2025-12-24. 60. F. Tomassetti and M. Torchiano. An empirical assessment of polyglot-ism in GitHub. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE). ACM, 2014. 61. S. Torres-Arias, D. Geer, and J. S. Meyers. A viewpoint on knowing software: Bill of materials quality when you see it. IEEE Security & Privacy, 21:50–54, 2023. 62. E. O. o. t. P. United States. Improving the nation’s cybersecurity. Executive Order No. 14028, 86 Fed. Reg. 26633–26647, May 2021. 63. C. Wang, J. Wu, H. Lyu, X. Ling, T. Luo, Y. Wu, and C. Zhao. A large scale empirical analysis on the adherence gap between standards and tools in sbom. ACM Trans. Softw. Eng. Methodol., Jan. 2026. 64. Y. Wang, B. Chen, K. Huang, B. Shi, C. Xu, X. Peng, Y. Wu, and Y. Liu. An empirical study of usages, updates and risks of third-party libraries in java projects. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 35– 45, 2020. 65. J. Wu, L. Bao, X. Yang, X. Xia, and X. Hu. A large-scale empirical study of open source license usage: Practices and challenges. In Proceedings of the 21st International Conference on Mining Software Repositories, MSR ’24, page 595–606, New York, NY, USA, 2024. Association for Computing Machinery. 66. B. Xia, T. Bi, Z. Xing, Q. Lu, and L. Zhu. An empirical study on software bill of materials: Where we stand and the road ahead. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2630–2642, 2023. 67. B. Xia, D. Zhang, Y. Liu, Q. Lu, Z. Xing, and L. Zhu. Trust in software supply chains: Blockchain-enabled sbom and the aibom future. In Proceedings of the 2024 ACM/IEEE 4th International Workshop on Engineering and Cybersecurity of Critical Systems (EnCyCriS) and 2024 IEEE/ACM Second International Workshop on Software Vulnerability, EnCyCriS/SVM ’24, page 12–19, New York, NY, USA, 2024. Association for Computing Machinery. 68. W. Xu, H. He, K. Gao, and M. Zhou. Understanding and remediating open-source license incompatibilities in the pypi ecosystem. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 178–190, 2023.

Title Suppressed Due to Excessive Length

69. S. Yu, W. Song, X. Hu, and H. Yin. On the correctness of metadata-based sbom generation: A differential analysis approach. In 2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pages 29–36, 2024. 70. N. Zahan, E. Lin, M. Tamanna, W. Enck, L. A. Williams, D. Balzarotti, W. Enck, T. Holz, and A. Stavrou. Software bills of materials are required. are we there yet? IEEE Security & Privacy, 21:82–88, 2023.

Record · ID 343548 · SHA-256 482b466e89134b7e
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.