A Multi-Surface Consistency Audit of Software Citation Metadata

D OES R ESEARCH S OFTWARE AGREE WITH I TSELF ? A M ULTI -S URFACE C ONSISTENCY AUDIT OF S OFTWARE C ITATION M ETADATA

arXiv:2608.17159v1 [cs.SE] 17 Aug 2026

A P REPRINT Pengyin Shan National Center for Supercomputing Applications University of Illinois Urbana-Champaign Urbana, IL 61820 pengyins@illinois.edu

August 19, 2026

A BSTRACT Research software projects describe themselves in many places at once: citation files in the repository, archive deposits, DOI registry records, package registries, and README text. We treat the software as the underlying object and these machine-readable self-descriptions as its surfaces: the points where people and automated systems read what the project declares about the software. Citation guidance, indexing services, and automated agents may read a different subset of these surfaces, so disagreement between them can silently fragment credit and provenance. This paper asks a simple question that has not been measured directly: when a project’s own metadata surfaces are compared with each other, how often do they agree? We audited 117 open-source research software projects, comprising an 87-project high-performance computing and quantum computing corpus and a 30-project registered baseline drawn from the JOSS and pyOpenSci accepted-package lists, across up to seven machinereadable surfaces per project. Using a four-level verdict rubric across six metadata fields, with 98.5% hand-verified verdict precision on a 338-row stratified sample, we found that 52 of the 62 projects exposing at least two comparable surfaces (83.9%) contain at least one core-field conflict, a result that is insensitive to the fuzzy-matching threshold. Half of hand-adjudicated cross-surface conflicts trace to a single mechanism: surfaces describing the software’s paper rather than the software itself. Among projects whose CITATION.cff includes a preferred citation, 28 of 32 route citations to a record that disagrees with the software’s own metadata. The author lists and titles disagree the most, and the registry surfaces are the least aligned. We release the audit pipeline as an importable library, the corpus, the registered sampling protocol, all raw snapshots, and the complete verification log. Keywords software citation · metadata consistency · research software · CITATION.cff · provenance · scholarly infrastructure

1

Introduction

The software citation infrastructure has largely solved the problem of where a project can declare its citation metadata. CITATION.cff [1], codemeta.json [2], archived deposits with DOIs [3], package registry metadata, and README citation sections are all established, machine-readable, and widely promoted. What this infrastructure does not guarantee is that these surfaces say the same thing. Each could be written at a different time, by a different process, for a different consumer: a CITATION.cff is edited by software developers or maintainers, a Zenodo record may be minted automatically from a GitHub release, a PyPI record is generated at the packaging time, and a README BibTeX block is often pasted directly from a paper’s landing page and never updated. Consumers of citation metadata, whether human authors following a "how to cite" section, automated indexers resolving a DOI, or Large Language Models (LLMs) assembling citations, each read one surface and trust it. If the surfaces

A Multi-Surface Consistency Audit of Software Citation Metadata

A P REPRINT

disagree, which record a citation carries depends on which surface the consumer happened to read. The consequences are familiar from software citation research: split citation counts, misattributed credit, version ambiguity, and papersoftware conflation [4–6]. What has been missing is a direct measurement of how often a project’s own self-descriptions disagree, across surfaces, at scale, with verified instrument precision. This paper contributes that measurement. We framed the problem as a supply-side audit: we frame what projects declare across multiple machine-readable surfaces they expose, and score pairwise consistency. Our contributions: • An audit instrument. An open pipeline (harvest, normalize, compare, report) covering seven surfaces: CITATION.cff (root and preferred-citation), codemeta.json, .zenodo.json, the resolved DOI record, PyPI and npm registry records, and README citation content. The comparison rubric has four verdict levels over six fields, with all thresholds documented and released. • A measured corpus with a registered baseline. 87 HPC and quantum computing projects from a prior, DOI-anchored corpus [7], plus a 30-project baseline drawn from the accepted-package lists of JOSS and pyOpenSci under a sampling protocol fixed and timestamped before candidate acceptance. • Verified findings. 83.9% of projects with at least two comparable surfaces disagree with themselves on at least one core field, with verdict precision hand-verified at 98.5% over a stratified 338-row sample and results insensitive to the matching threshold. • A mechanism taxonomy. Half of adjudicated cross-surface conflicts trace to paper-versus-software conflation, and the remainder cluster into archive staleness, registry ghost metadata, and identity ambiguities, each illustrated with named, snapshot-dated examples.

2

Related Work

The FORCE11 software citation principles established software as a citable research product and specified what a citation must identify: the artifact actually used, through its authors, title, version, date, and a unique persistent identifier [4]. The FAIR principles for research software carry the same position into metadata practice, tying findability and reuse to rich, machine-readable description of the software itself [8]. Reviewing the implementation two years after the principles, Katz and Chue Hong reported that the infrastructure had arrived, with over 50,000 DOIs issued for software at the time, while challenges remained in developer practice and in publisher systems retaining the required metadata [6]. Five of the six fields our rubric compares (title, authors, version, year, and DOI) are the fields this line of work designates as citation-critical; the sixth, license, is carried by every machine-readable surface we audit. Our question is not whether projects declare these fields but whether the surfaces declaring them agree. Declaration infrastructure followed the principles. The Citation File Format gives maintainers a plain-text citation file in the repository root [1], and GitHub renders that file as the "cite this repository" widget on the repository landing page, so a single YAML file now reaches every visitor. CodeMeta provides a JSON-LD exchange schema with crosswalks between registry and archive vocabularies [2], and archive DOI records are registered against the DataCite metadata schema [3]. Adoption of this infrastructure is measured at registry scale: El Hounsri and Garijo analyzed research software metadata adoption across the software registries of five European Open Science clusters and a multi-disciplinary registry and found recommended practice unevenly followed [9]. Adoption studies of this kind have established whether a surface is present and well-formed, but none has asked whether the surfaces a project has adopted describe the same object in the same way, which is the measurement this paper adds. Where metadata quality is studied, it is studied one surface at a time. On the paper side, Howison and Bullard examined how software is mentioned in the biology literature and found mentions that were informal and incomplete, lacking the version and identifier information needed to find the artifact used [5]. On the registry side, Bommarito and Bommarito characterized package metadata across 178,592 PyPI packages [10], and Gao et al. measured that existing tools can recover source-repository information for at most 70.5% of PyPI releases because release metadata omits the repository or names the wrong one, and built PyRadar to retrieve and validate those links [11]. The newest large-scale datasets aggregate in the same direction: OpenDORS compiles license, version, and language metadata for over 120,000 openly referenced research software repositories, one record per project [12]. Each of these efforts measures the presence or quality of a single surface across many projects, or merges a project’s surfaces into one record. To our knowledge, no prior study compares a project’s own self-description surfaces against each other, so the rate at which research software disagrees with itself has not been measured. That distinction also separates this audit from adoption analyses such as [9] as counting surfaces and cross-checking them are different measurements, and only the first exists in the literature. A separate line of work documents the paper standing in for the software. The Journal of Open Source Software aims to create a peer-reviewed, citable paper, so that software work can be credited through existing publication infrastructure [13]; which citation carries is left to authors and citing practice. Corpus studies show how the proxy pattern emerges 2

A Multi-Surface Consistency Audit of Software Citation Metadata

A P REPRINT

in that practice. The Softcite dataset, built from expert annotation of 4,971 biomedical and economics publications, found that software is mentioned informally far more often than it is formally cited [14], and a follow-up study of the CORD-19 corpus found that even where projects request a specific citation, identifying the canonical record among a project’s several candidate sources is difficult [15]. These studies measure conflation where it lands in the citing literature. Our mechanism analysis in Section 4.4 measures it one step earlier: in the machine-readable metadata projects themselves publish, where half of verified cross-surface conflicts trace to a surface describing the paper rather than the software. Supply-chain security frameworks audit adjacent layers of the same projects. OpenSSF Scorecard scores repository and maintenance practices automatically [16]; SLSA defines graded levels of build provenance, and since the specification is community-maintained without a peer-reviewed primary description, we reference the qualitative study of its deployment challenges by Tamanna et al. [17]; Sigstore provides signing infrastructure for released artifacts [18]. These frameworks address whether code and artifacts are what they claim to be. None checks whether a project’s self-descriptions agree with each other, so the audit reported here is complementary to them and overlaps none of their controls.

3

Methods

3.1

Corpus

The corpus contains 117 projects in three strata. The supercomputing stratum (labeled sc26 in the released data files; 87 projects; 44 HPC, 43 quantum computing) is exported verbatim from the tagged v1.0.0 release of our prior provenance-audit corpus [7], where the derivation from the tagged release keeps the corpus identity claim checkable. The baseline strata (15 JOSS, 15 pyOpenSci) exist to test whether findings from infrastructure-heavy HPC/Quantum software generalize to community-reviewed research software, and were drawn as follows. Sampling frames were retrieved on 2026-08-12 from the JOSS published-papers API (3,662 records after retrieval) and the pyOpenSci packages registry (70 records). Corresponding software repositories were required to be GitHub-hosted, since all six eligibility surfaces are probed through the GitHub API. Non-GitHub candidates were excluded at sampling under rule R5 (JOSS: 185, pyOpenSci: 4, hosts reported in the exclusion log). Four further pre-registered rules applied in order: R1 corpus overlap with the 87 (7 exclusions), R2 dual-listing resolved to the pyOpenSci stratum (32), R3 JOSS version papers deduplicated to the most recent (22), R4 missing repository URL (0). Domain bins were assigned from a committed keyword map (first match wins; disciplinary keywords precede generic ones). Candidates with no keyword matched were retained in an explicit "unmapped" bin that participates in the stratified draw rather than being excluded, so the draw does not depend on the coverage of a hand-written map. The registered unmapped share was 33% (JOSS) and 2% (pyOpenSci). One design correction was made before the protocol was registered: an earlier rule that refused to finalize while unmapped candidates appeared in the draw window was unsatisfiable under round-robin interleaving, since the unmapped bin is guaranteed window slots by construction, and it was replaced by unmapped-bin participation plus a 50% frame-level ceiling. A seeded, domain-stratified draw (seed 202608121338, round-robin over domain bins, oversample 30 per stratum) produced ranked candidate windows. The sampling protocol, ranked candidate lists, and exclusion log were committed (baseline-sampler v0.1.0) and deposited under restricted access with a DOI before any eligibility probing (10.5281/zenodo.21909720; the deposit timestamp precedes the acceptance log timestamp). Eligibility required at least 2 of six machine-readable surfaces, probed per candidate: CITATION.cff, codemeta.json, .zenodo.json in the repository root, a README citation surface (citation section or BibTeX block; the registered rule), a discoverable DOI (from CFF or the README citation section, not resolved at probe time) and a verified registry listing (a PyPI or npm package matching the repository name and linking back to the repository). Acceptance proceeded strictly in rank order to 15 per stratum: JOSS accepted ranks 2-18 (3 skips for insufficient surfaces), pyOpenSci ranks 1-19 (4 skips). One drawn candidate’s declared repository did not resolve at probe time and was never reached by acceptance. 15 of the 30 pyOpenSci window candidates are dual-listed with JOSS and therefore carry a JOSS paper DOI, so the DOI-by-construction confound noted below extends partially into that stratum. Hand-curated concept DOIs were filled for the supercomputing stratum only (20 projects), drawn from the prior corpus’s distribution-metadata records; both baseline strata rely entirely on automatic discovery from project surfaces, keeping the baseline availability numbers free of curation effects. Registry package names were detected uniformly for all 117 projects by name-part lookup verified by repository back-link and were hand-reviewed. Two detections were removed as packages not controlled by the project. 3

A Multi-Surface Consistency Audit of Software Citation Metadata

3.2

A P REPRINT

Surfaces

For each project we harvest up to seven surfaces: cff (CITATION.cff root), cff_preferred (its preferredcitation block, extracted as a pseudo-surface so intra-file disagreement is measurable), codemeta (codemeta.json), zenodo_json (.zenodo.json), doi_record (the DOI’s registered metadata, resolved via DataCite [3] with Crossref fallback; DOI priority order: corpus-curated concept DOI, then CFF-declared, then README citation section, with provenance recorded), pypi and npm (registry records), and readme (a heuristic parse of the README’s citation section, BibTeX block, and DOIs). GitHub’s "cite this repository" widget is generated from CITATION.cff and is covered by the cff surface. The README surface is defined as a README file in the repository root; GitHub additionally renders docs/ and .github/ READMEs on the landing page, which this audit does not credit (one corpus project is affected). Every API response is snapshotted with a retrieval timestamp; all snapshots were retrieved on 2026-08-14 and are released in the data deposit. 3.3

Normalization

Each surface is normalized into one record shape: title, authors (family, given, ORCID), version, year, DOI, license. Extraction rules are released with the pipeline, where three deserve note: BibTeX in READMEs is extracted with brace-matching rather than regular expressions, after live testing showed regex extraction silently dropping fields across common forms; DOI strings are sanitized (resolver-URL prefixes, badge-image suffixes, trailing punctuation) and validated against the DOI pattern before resolution; DataCite creator records frequently carry the full name in familyName with no givenName so the normalizer uses record-supplied name parts as-is and falls back to name-splitting only when the record supplies neither part. 3.4

Comparison rubric

For every pair of surfaces present, each of the six fields receives one of four verdicts. exact: normalized values identical. minor: clearly the same referent with superficial variation (title token-sort fuzzy >= 90; the same author set reordered or fuzzy-matched; year differing by exactly 1; version differing only by a v prefix). conflict: both present and materially different. missing: at least one side absent. Zenodo concept-versus-version DOI pairs are flagged for hand adjudication rather than auto-resolved. All thresholds live in one released module. Headline results are insensitive to the fuzzy-match threshold: at 85 and 95 the share of projects whose surfaces contain at least one core-field conflict is unchanged (83.9%, with identical per-stratum rates), while field-level agreement shifts modestly (authors 0.459-0.511 around the reported 0.481; title 0.436-0.442; median per-project pairwise agreement 0.50-0.55). Threshold-sensitivity outputs are released alongside the results. 3.5

Verification protocol and instrument disclosure

We hand-verified the instrument’s verdicts against the harvested snapshots rather than the live web, since precision measures whether the pipeline judged correctly given the evidence it retrieved. The verification set comprised every conflict verdict, every minor verdict, every intra-file conflict between CFF root and preferred-citation metadata, and a seeded random sample of 30 exact verdicts. For each judgment, we re-derived the verdict by hand from the recorded value pair; identical value pairs recurring across surface pairs within a project were adjudicated once and the judgment propagated to each covered comparison row. Raw snapshots were consulted where a recorded value appeared anomalous. Rows flagged as possible Zenodo concept-versus-version identifier pairs were additionally adjudicated by resolving both identifiers. Verified precision was 98.5% over 338 comparison rows (275 deduplicated judgments): conflict 276/281 (98.2%), minor 26/26, exact 31/31. The five disagreeing rows classify into two mechanisms, both documented heuristic limits: threshold boundary (2 rows; an author-entity name variant scoring just below the fuzzy threshold) and parse limitations (3 rows; an organizational author split at an embedded "and", and a LaTeX-escaped accented character blocking one name match). The complete verification log is released. Instrument defects found and corrected before the reported measurements, in the honest-reporting style this protocol requires: two normalization defect classes were identified in pre-run live testing (YAML date serialization in CITATION.cff snapshots; .zenodo.json license objects producing spurious license conflicts); three DOI-string hygiene classes were identified in first-run review (resolver-URL prefixes, badge-image suffixes, non-DOI identifier values); and hand verification identified one systematic defect (DataCite creator records carrying full names in familyName caused token duplication and depressed author matching), which was corrected and the affected comparisons re-scored and re-verified. The instrument’s stage history (v0.2.0 pre-run; v0.2.1 first-run corrections; v0.2.2 verification-corrected 4

A Multi-Surface Consistency Audit of Software Citation Metadata

A P REPRINT

Table 1: Availability of self-description surfaces across the corpus (n = 117; all projects resolvable at snapshot time). Surface

Availability a

README (root-level) Stable releases CITATION.cffb Registry listing (PyPI or npm)c Archival DOI (doi_record) SECURITY.md .zenodo.jsond codemeta.json

116/117 (99%) 111/117 (95%) 48/117 (41%) 46/117 (39%) 20/117 (17%) 16/117 (14%) 10/117 (9%) 6/117 (5%)

a

One project (preCICE) keeps its README under docs/; under the root-only rule it is counted absent (see Section 3.2). b By stratum: 26/87 supercomputing, 10/15 JOSS, 12/15 pyOpenSci. c One project is npm-listed; it is also PyPI-listed. d By stratum: 3/87 supercomputing, 1/15 JOSS, 6/15 pyOpenSci.

Table 2: Pairwise agreement by field across comparable surface pairs, with and without registry-involved pairs. Registry exclusion is shown only for the two fields where registry pairs change the rate materially. All comparable pairs Field

n

Agreement

Year DOI License Authors Version Title

50 61 47 135 47 165

.780 .770 .766 .481 .447 .436

Excluding registry pairs n

Agreement

98

.612

90

.644

Of 93 title conflicts, 61 involve a registry surface.

final) is documented above and in the repository README; the public repository history was flattened for release, and v0.2.2 produced every number reported here.

4

Results

4.1

Surface availability

All 117 repositories resolved at snapshot time. Table 1 reports surface availability. The human-facing substrate is near-universal: README in 116 of 117 projects and stable releases in 95%. The machine-readable citation surfaces are minority practice, led by CITATION.cff at 41% and thinning to codemeta.json at 5%. 62 of 117 projects (53%) expose at least two comparable surfaces; the consistency results below concern those 62. That only half the corpus can be cross-checked at all is itself a finding: for the other half, whatever the single surface says is uncheckable by construction. 4.2

Pairwise consistency

Across 1,104 field comparisons (505 comparable, 599 with at least one side missing), 52 of 62 projects (83.9%) contain at least one core-field conflict (title, authors, version, year, or DOI), and median per-project pairwise agreement is 0.50. By field, agreement among comparable pairs is: year 0.78, DOI 0.77, license 0.77, authors 0.48, version 0.45, title 0.44. Figure 1 shows the composition behind these rates: of 1,104 field comparisons, 599 could not be made at all because at least one surface omits the field, so the agreement rates in Table 2 describe only the comparable remainder. The registry surfaces drive much of the title and authors disagreement: 61 of 93 title conflicts involve PyPI or npm, largely the structural gap between a registry’s package name and a descriptive title ("mpi4py" versus "MPI for Python"). Excluding registry pairs, title agreement rises from 0.44 to 0.64 and authors from 0.48 to 0.61. Figure 2 locates the disagreement: pairs involving a registry surface concentrate the low agreement values, while the repository-native pairs (cff with codemeta, codemeta with .zenodo.json) agree most. We report both because both are real: a human reader distinguishes a package name from a title, but a metadata consumer joining records across surfaces sees a mismatch. 5

A Multi-Surface Consistency Audit of Software Citation Metadata

A P REPRINT

Figure 1: Verdict composition per field across all 1,104 surface-pair field comparisons. Gray indicates the field was not comparable because at least one surface omits it; Table 2 reports agreement among the comparable remainder. Table 3: Consistency by stratum among projects with at least two comparable surfaces. Agreement is pooled pairwise agreement: exact plus minor verdicts over comparable pairs within the stratum. Projects

Comparable pairs

Agreement

≥1 core-field conflict

supercomputing JOSS pyOpenSci

36 11 15

236 97 172

.559 .649 .494

86.1% 63.6% 93.3%

Overall

62

505

.554

83.9%

Stratum

The median per-project pairwise agreement across the 62 projects is .500.

By stratum, the phenomenon is not an HPC idiosyncrasy: core-field conflict affects 86.1% of supercomputing projects (31/36 with >= 2 surfaces), 63.6% of JOSS projects (7/11), and 93.3% of pyOpenSci projects (14/15). Two cautions attach to the baseline comparison: the JOSS stratum’s DOI field is partially DOI-by-construction (every JOSS paper carries a DOI, and 15 of 30 pyOpenSci window candidates are dual-listed), and per-stratum denominators are small (11-36 projects), so we report the direction (community-reviewed packages are not cleaner; pyOpenSci projects expose more surfaces and correspondingly more disagreement) without significance claims. One tautology is excluded from interpretation: when the resolved DOI was discovered from a project’s own CFF or README, the DOI-field agreement between the discovering surface and the DOI record is expected; informative DOI comparisons are those between non-source surfaces, including the four adjudicated concept-versus-version pairs. 4.3

The project’s own citation file disagrees with itself

32 projects’ CITATION.cff files include a preferred-citation block. In 28 of 32 (87.5%), following the preferred-citation yields a record that conflicts with the software’s own metadata on at least one field (56 conflicting field comparisons; 39 exact, 1 minor). This is partly the CFF specification working as designed, since preferred-citation exists precisely to route citation to a different object, usually a paper. The measured fact is about the consequence: for seven out of eight such projects, the record a compliant citation tool emits and the record the software’s other surfaces describe are different objects, and which one a consumer lands on depends on tooling, not on the project’s intent. 6

A Multi-Surface Consistency Audit of Software Citation Metadata

A P REPRINT

Figure 2: Pairwise agreement (exact plus minor over comparable) by surface pair. Cell denominators vary widely; extreme values, including 0.00 and 1.00, can rest on few comparable pairs, and per-cell counts are in the released results data; the corpus contains a single npm-listed project. 4.4

A mechanism taxonomy of verified conflicts

Hand adjudication of every conflict produced a mechanism classification; the four recurring mechanisms, with named, snapshot-dated examples (retrieved 2026-08-14): Paper-versus-software conflation (86 of 171 cross-surface conflict judgments, 50%). Surfaces describe the software’s paper rather than the software. Two sub-patterns. Divergent: dace’s CITATION.cff root titles the software ("DaCe - Data Centric Parallel Programming") while its README BibTeX carries the conference paper’s title and its five authors, against 28 in the CFF. Consistent-but-paper-anchored: thread-pool’s CFF root itself declares the SoftwareX paper’s DOI and title, so its surfaces agree with each other while all describing the paper; consistency is not correctness of referent. Archive staleness. DOI’d archive records lag the distributed release: felupe’s Zenodo record says v9.4.0 while PyPI serves 10.1.0; cuda-quantum shows the same mechanism at minimum distance (0.15.0 versus 0.15.1). The rubric counts each non-equal pair once regardless of distance. Registry ghost metadata. Registry records exist that the project does not control or no longer describes: our review removed two back-link-verified packages (a community npm build and a third-party PyPI wrapper) as not projectcontrolled; registry author fields are frequently entity strings or empty, and no registry surface in the corpus carries an ORCID (0/25 sampled authors on PyPI). 7

A Multi-Surface Consistency Audit of Software Citation Metadata

A P REPRINT

Identity ambiguity. Entity and person names collide across surfaces: cirq’s surfaces disagree only on "Cirq Developers" versus "The Cirq Developers"; mfem’s CFF root credits "MFEM Team" while its archive record credits two named individuals and its preferred citation seventeen. By design, the by-design routing of Section 4.3 (mfem-style preferred-citation) is tabulated separately from crosssurface conflation, since the former is specification-compliant behavior with a fragmenting consequence and the latter is unintended divergence. 4.5

ORCID coverage

Where surfaces carry authors, ORCID coverage varies from 62% of authors in .zenodo.json and 54% in CFF to 39% in resolved DOI records and 0% in registry records and README BibTeX, so the surfaces most likely to be read by packaging tooling carry the least persistent-identifier information.

5

Discussion

Three implications follow from the supply side alone. First, single-surface consumers inherit whichever record their surface carries; at the measured disagreement rates, two compliant consumers reading different surfaces of the same project will disagree about its citation roughly half the time at the field level. Second, the paper-versus-software mechanism dominating verified conflicts (50%) means the fragmentation is not primarily sloppiness: it is the unresolved question of what the citable object is, propagating into machine-readable metadata, consistent with the credit incentives documented in software citation research [4, 6], and it will not be fixed by adoption campaigns for any single format. Third, agreement infrastructure is absent: no existing supply-chain or metadata framework checks cross-surface consistency ([16–18] audit build and release integrity, not self-description coherence), and the near-universal surfaces (README, releases) are precisely the ones with the least structure. The audit engine is released as an importable library whose per-project primitive (harvest, normalize, compare, report for one repository) is designed for downstream reuse in release-time integrity checks and citation and provenance checker tools; the corpus-scale study reported here is one instantiation. The demand side, how automated consumers weight these surfaces when they disagree, is measurable with the same instrument as ground truth and is deliberately out of scope here.

6

Limitations

The corpus is GitHub-hosted by criterion and skewed toward HPC/QC by design, with small baseline strata. README parsing is heuristic; verified precision (98.5%) bounds but does not eliminate instrument error, and the released log classifies every known disagreement. The registry surface covers PyPI and npm only, so R and Julia packages face a five-surface ceiling, and their registry practice is unmeasured. Registry name-versus-title conflicts are structural and reported both included and excluded. The eligibility filter conditions the baseline on having >= 2 surfaces, so baseline agreement rates describe metadata-active projects, not the registries at large. Snapshots date to a single retrieval window; projects change. Concept-versus-version DOI adjudication covered flagged Zenodo pairs only. Verdict counts weight all conflicts equally regardless of severity.

7

Conclusion

Measured across their own machine-readable surfaces, five of six research software projects that can be cross-checked disagree with themselves on at least one citation-critical field, and the single largest verified mechanism is the paper standing in for the software. The instrument, corpus, registered protocol, snapshots, and verification log are released so the measurement can be repeated, extended to other ecosystems, and used as ground truth for the demand-side question of what metadata consumers do when the surfaces disagree. Extending the corpus beyond GitHub-hosted projects and additional registries, and re-measuring the same projects over time and assessing mitigation in which a single authoritative record generates the other surfaces, is planned future work on the same instrument.

8

Data and Code Availability

Pipeline: https://github.com/pengyin-shan/rda-audit-pipeline (Apache-2.0). Measurements were produced with v0.2.2; the archived release v0.2.3 adds only license, citation, and README files, with measurement code identical 8

A Multi-Surface Consistency Audit of Software Citation Metadata

A P REPRINT

(10.5281/zenodo.21969695). Corpus, raw snapshots (retrieval-timestamped), results, verification log, and sensitivity outputs: 10.5281/zenodo.21969769, CC-BY-4.0. Sampling protocol, ranked candidate lists, and exclusion log: 10.5281/zenodo.21909720 (restricted until publication; opened on posting); the same files are public in the sampler repository. Sampler: baseline-sampler v0.1.0 (registered draw commit 64b979e), https://github.com/pengyinshan/baseline-sampler. Prior corpus: 10.5281/zenodo.21443211.

9

Acknowledgments

The author thanks Daniel S. Katz for comments on a draft. Portions of the manuscript text were drafted with writing assistance from Claude (Anthropic); the author reviewed and verified all content, and all data collection, measurement, and verification were performed by the author. The corpus derives from work conducted during the author’s 2025-2026 Trusted CI Fellowship with Trusted CI, the NSF Cybersecurity Center of Excellence; the fellowship provided no financial support for this work. The author thanks the National Center for Supercomputing Applications (NCSA), University of Illinois Urbana-Champaign, for a supportive professional environment; this work was conducted in the author’s individual capacity and received no dedicated funding. The author declares no competing interests.

References [1] Stephan Druskat, Jurriaan H. Spaaks, Neil Chue Hong, Robert Haines, James Baker, Spencer Bliven, Egon Willighagen, David Pérez-Suárez, and Alexander Konovalov. Citation file format, August 2021. URL https: //doi.org/10.5281/zenodo.5171937. [2] Matthew B. Jones, Carl Boettiger, Abby Cabunoc Mayes, Arfon Smith, Peter Slaughter, Kyle Niemeyer, Yolanda Gil, Martin Fenner, Krzysztof Nowak, Mark Hahnel, Luke Coy, Alice Allen, Mercè Crosas, Ashley Sands, Neil Chue Hong, Patricia Cruse, Daniel S. Katz, and Carole Goble. CodeMeta: an exchange schema for software metadata, 2017. URL https://doi.org/10.5063/schema/codemeta-2.0. Version 2.0. [3] DataCite Metadata Working Group. DataCite metadata schema for the publication and citation of research data and other research outputs, v4.5, 2024. URL https://doi.org/10.14454/g8e5-6293. [4] Arfon M. Smith, Daniel S. Katz, Kyle E. Niemeyer, and FORCE11 Software Citation Working Group. Software citation principles. PeerJ Computer Science, 2:e86, 2016. doi:10.7717/peerj-cs.86. [5] James Howison and Julia Bullard. Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature. Journal of the Association for Information Science and Technology, 67(9):2137–2155, 2016. doi:10.1002/asi.23538. [6] Daniel S. Katz and Neil P. Chue Hong. Software citation in theory and practice. In James H. Davenport, Manuel Kauers, George Labahn, and Josef Urban, editors, Mathematical Software – ICMS 2018, volume 10931 of Lecture Notes in Computer Science, pages 289–296, Cham, 2018. Springer. doi:10.1007/978-3-319-96418-8_34. [7] Pengyin Shan. Channel-based provenance signals for 87 HPC and quantum-computing open-source projects, 2026. URL https://doi.org/10.5281/zenodo.21443211. Version 1.0.0, dataset. [8] Michelle Barker, Neil P. Chue Hong, Daniel S. Katz, Anna-Lena Lamprecht, Carlos Martinez-Ortiz, Fotis Psomopoulos, Jennifer Harrow, Leyla Jael Castro, Morane Gruenpeter, Paula Andrea Martinez, and Tom Honeyman. Introducing the FAIR principles for research software. Scientific Data, 9(1):622, 2022. doi:10.1038/s41597-02201710-x. [9] Anas El Hounsri and Daniel Garijo. Good practice versus reality: A landscape analysis of research software metadata adoption in european open science clusters. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), pages 116–128, 2025. doi:10.1109/MSR66628.2025.00028. [10] Ethan Bommarito and Michael J. Bommarito. An empirical analysis of the Python Package Index (PyPI), 2019. [11] Kai Gao, Weiwei Xu, Wenhao Yang, and Minghui Zhou. PyRadar: Towards automatically retrieving and validating source code repository information for PyPI packages. Proceedings of the ACM on Software Engineering, 1(FSE): 2608–2631, 2024. doi:10.1145/3660822. Article 115. [12] Stephan Druskat and Lars Grunske. OpenDORS: A dataset of openly referenced open research software, 2025. URL https://doi.org/10.48550/arXiv.2512.01570. arXiv:2512.01570. [13] Daniel S. Katz, Kyle E. Niemeyer, and Arfon M. Smith. Publish your software: Introducing the journal of open source software (JOSS). Computing in Science & Engineering, 20(3):84–88, 2018. doi:10.1109/MCSE.2018.03221930. 9

A Multi-Surface Consistency Audit of Software Citation Metadata

A P REPRINT

[14] Caifan Du, Johanna Cohoon, Patrice Lopez, and James Howison. Softcite dataset: A dataset of software mentions in biomedical and economic research publications. Journal of the Association for Information Science and Technology, 72(7):870–884, 2021. doi:10.1002/asi.24454. [15] Caifan Du, Johanna Cohoon, Patrice Lopez, and James Howison. Understanding progress in software citation: a study of software citation in the CORD-19 corpus. PeerJ Computer Science, 8:e1022, 2022. doi:10.7717/peerjcs.1022. [16] Nusrat Zahan, Parth Kanakiya, Brian Hambleton, Shohanuzzaman Shohan, and Laurie Williams. OpenSSF scorecard: On the path toward ecosystem-wide automated security metrics. IEEE Security & Privacy, 21(6): 76–88, 2023. doi:10.1109/MSEC.2023.3279773. [17] Mahzabin Tamanna, Sivana Hamer, Mindy Tran, Sascha Fahl, Yasemin Acar, and Laurie Williams. Analyzing challenges in deployment of the SLSA framework for software supply chain security, 2024. URL https: //doi.org/10.48550/arXiv.2409.05014. arXiv:2409.05014. [18] Zachary Newman, John Speed Meyers, and Santiago Torres-Arias. Sigstore: Software signing for everybody. In Proc. 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2022. doi:10.1145/3548606.3560596.

10

Related documents

• Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering#120615
• Citation Traceability for Web-Native AI Research Workflows#173509
• Type-IV Code Clone Detection via Layer-Wise Non-Contrastive Representation Learning#1123705
• Type-IV Code Clone Detection via Layer-Wise Non-Contrastive Representation Learning#919387
• Loreley: Repository-Scale Program Evolution with Quality-Diversity Search#471310