ARTICLE
Building a research-software catalog with a coding agent: from hackathon prototype to public deployment Kazuyoshi Yoshimia , Satoshi Terasakia , Gotai Yamadaa
arXiv:2609.04711v1 [cs.SE] 4 Sep 2026
a
Institute for Solid State Physics, The University of Tokyo, Kashiwa, Japan ARTICLE HISTORY Compiled September 7, 2026 ABSTRACT Generative AI and coding agents can accelerate research software development, but they also increase the need for efficient software discovery and maintenance. We developed a repository catalog during a three-day hackathon and subsequently examined the engineering required to make it suitable for public deployment, including adversarial review, data-quality checks, browser-level validation, and publication safeguards. We then explored whether the lessons learned from this prototype could be transferred to a much larger, human-curated portal, through a retrieval agent under development for MateriApps that combines curated portal metadata, external documentation, vector search, and local language-model generation. Implementation with coding agents was rapid, but achieving reliable operation required substantial additional engineering: the most consequential problems were not crashes but silent failures that produced plausible yet incomplete or incorrect outputs, arising from incomplete data acquisition, misleading assessments, and retrieval or preprocessing failures. These observations suggest that AI-assisted software portals require explicit validation, monitoring, and repeated review, and that curated metadata and maintained documentation remain essential. The MateriApps work is exploratory and remains under active development, so the observations reported for it are preliminary; a comparable combination of curated metadata, automatically collected documentation, and retrieval-based assistance may nevertheless be useful for extending other research-software portals. GRAPHICAL ABSTRACT CONTACT Kazuyoshi Yoshimi. Email: [email protected]
1 HACKATHON PROTOTYPE
Coding agent (Claude Code)
2
VALIDATION FOR PUBLIC DEPLOYMENT
the system kept running and produced plausible but incomplete or incorrect output
Incomplete data acquisition
Retrieval / preprocessing failures
Misleading assessments
caught only by explicit validation 4-stage pipeline
export-web static site
Applying the same lessons to a larger, human-curated portal
MateriApps 336 applications ~45,000 passages Curated metadata human-curated Deterministic narrowing
fetch build-data
UNDER ACTIVE DEVELOPMENT
NOT CRASHES - SILENT FAILURES
・3-day hackathon ・68 repositories
Exploratory extension
3
Adversarial review
Data-quality checks
Browser validation
Publication safeguards
Publishable — with explicit validation and monitoring
Rapid implementation
≠
Document retrieval (vector search) Local language-model human-curated
Public-ready system
KEYWORDS research software; software catalog; retrieval-augmented generation; generative AI; scientific portals; computational materials science
1. Introduction Generative AI and coding agents are lowering the cost of developing research software. Since large language models were shown to generate working code from naturallanguage descriptions[1], AI coding assistants such as GitHub Copilot[2] have been integrated into everyday software-development workflows, and tools and extensions that previously required substantial programming effort can now be produced within short development cycles. Empirical studies of these tools report genuine gains in development speed alongside recurring defects in the generated code, including quality and security weaknesses that are not evident from the output alone[3–5]. This acceleration, however, creates a corresponding challenge: software may be developed and updated faster than information about it can be collected, maintained, and made discoverable to potential users. This problem is particularly relevant to continuing software development programs. The Project for Advancement of Software Usability in Materials Science (PASUMS)[6], for example, produces new software packages and functional enhancements each year. MateriApps[7, 8], maintained by the Institute for Solid State Physics, provides a community portal through which such computational materials science software can be discovered. As of 31 July 2026 it listed 336 applications across 11 categories. Its descriptions, classifications, and evaluations have largely been created and maintained 2
through human editorial effort. As the number of software packages and updates increases, however, manually keeping this information current becomes increasingly difficult. The challenge is therefore not simply to create a software portal, but to maintain one as the underlying software ecosystem evolves. Findability is the first of the FAIR principles as they have been adapted to research software[9], and it depends on descriptive metadata of the kind that software citation practice also relies upon[10]. Registries and archives address parts of this need in other communities: bio.tools documents bioinformatics resources through sustained community curation[11], while Software Heritage preserves and identifies source code itself[12]. In all such systems, however, new applications must be registered, and existing entries must be updated when repositories, documentation, supported environments, or functionality change. If software development accelerates while curation remains predominantly manual, the gap between available software and discoverable software will continue to widen. Generative AI contributes to this problem by accelerating software production, but it may also provide part of the solution. In data-driven materials science, its use in research and education has begun to be examined systematically, including the tasks for which current models are and are not dependable[13]. Software discovery and portal maintenance are a natural extension of that question: repository metadata and documentation can potentially be collected, summarized, classified, and searched automatically. This study therefore asks whether the technologies that accelerate research software development can also reduce the recurring effort required to maintain the portals through which that software is disseminated. This paper presents an experience report originating from a three-day hackathon held in June 2026[14], whose theme was the construction of materials databases using AI agents. During the hackathon, we developed a catalog site that automatically collected information from research software repositories and provided a natural-language interface using a locally hosted language model. The initial system contained 68 catalog records. No further repositories were made public between the hackathon and the 1 September 2026 snapshot analyzed here, so the same 68 records are reported throughout this paper. We then examined the additional engineering and validation required to transform this prototype into a system that could be deployed and maintained for use by others. After the hackathon and the subsequent development of the catalog, we began exploring whether the lessons learned from this small prototype could be transferred to an established portal of much larger scale. MateriApps was selected as the test case because it is human-curated and because its documentation corpus is far larger and more heterogeneous than that of the catalog. Information collected from the portal and from the external documentation linked by its entries yielded approximately 49,500 indexed passages associated with the 336 applications listed on 31 July 2026, in the corpus examined here, whose crawl was completed on 3 August 2026. To retrieve information from this collection, we assembled a staged workflow combining manually curated portal metadata, deterministic narrowing, document retrieval, and language-model-based response generation. This retrieval agent under development for MateriApps is exploratory and remains under active development; we therefore present it as a larger-scale case study rather than as a second completed system. The two settings address complementary aspects of portal maintenance. The repository catalog supports the automatic incorporation of newly developed and updated software, including outcomes generated through continuing projects such as PASUMS. The MateriApps case study indicates how automation can build upon, rather than re3
place, existing editorial work: manually assigned categories and tags remain important for narrowing broad user queries to relevant groups of applications. This study makes three main contributions. First, it documents the transition from a generative-AI-assisted hackathon prototype to a deployable research software portal. Second, it identifies failure modes in which automated processing produces plausible but incorrect results without explicit execution errors, a class of behavior related to but distinct from the hallucination phenomena documented for language generation itself[15]. Third, it illustrates how automated collection, human-curated metadata, document retrieval, and language-model generation can play complementary roles in software discovery. The remainder of this paper is organized as follows. Section 2 describes the catalog developed during the hackathon, and Section 3 discusses the engineering and validation required to make it suitable for public deployment. Section 4 describes the exploratory application of these lessons to MateriApps. Section 5 discusses the implications for software portal maintenance, system reliability, and human–AI collaboration in software curation, together with the limitations of the study, and Section 6 concludes.
2. What we built first: the catalog 2.1. Catalog pipeline and readiness assessment The catalog is generated through a four-stage pipeline (Figure 1). JSON files are used as the interfaces between stages, allowing each stage to be executed, inspected, and tested independently. The fetch stage collects repository and project metadata through the GitHub and GitLab APIs. It handles pagination and rate limits, and records basic repository metadata, the root-level file listing, and the README content. The build-data stage normalizes these records, merges manually maintained metadata—including taglines, categories, intended users, use cases, and DOIs—applies the publication policy, and computes a readiness score. The export-web stage extracts only the records approved for public release. Finally, a static-site generator produces a searchable and filterable website that can be deployed through GitHub Pages, GitLab Pages, or another static hosting service. The pipeline is executed on a daily schedule by a GitHub Actions workflow, so that repository metadata, readiness scores, and the published site are regenerated without manual intervention; the same workflow can also be triggered manually. Each scheduled run rebuilds the catalog from the provider APIs and redeploys the site only after the publication checks described in Section 3.2 have passed. The figures in this paper are taken from one deployment of the resulting catalog[16]. The same implementation generated three site profiles from separate configuration files at the revision described here. Repository sources, publication policies, branding, and presentation settings are specified per deployment rather than embedded in the application code. This separation improved reuse, although it also introduced configuration-related security concerns discussed in Section 3.3. One of these deployments is the catalog for the Project for Advancement of Software Usability in Materials Science (PASUMS)[6], a continuing software-development and enhancement project at the Institute for Solid State Physics (ISSP)[17]. The harvesting, readiness assessment, publication filtering, and site-generation mechanisms are shared across the deployments; the principal differences are the configuration, reposi-
4
tory source, and publication policy. The category taxonomy is an exception, because the current taxonomy reflects the activities and terminology of ISSP and would need to be adapted for use by another institution. The PASUMS deployment therefore provides a practical case for examining whether a repository catalog can reduce the recurring effort required to maintain an institutional software portal. The pipeline output is published as a browsable catalog site, available in both English and Japanese, with browse-by-purpose entry points, category counts, and keyword search (Figure 2). Each repository is assigned a readiness score from 0 to 100 based on seven binary signals: the presence of a README (20 points), license information (20), examples (15), tests (15), documentation (10), citation information (10), and a tagged release (10). The score is presented together with a checklist (Figure 3) so that maintainers can identify which artifacts are missing rather than seeing only an aggregate value. The same signals are also summarized as two star ratings, Openness and Richness. Openness is derived from the README, license, citation information, and release status, and reflects whether an external user can identify, use, and cite the software. Richness awards one star for each of documentation, examples, and tests, and is therefore shown on a 0–3 scale (Figure 3); it reflects whether the software is explained, demonstrated, and checked. These indicators are intended as transparent summaries of observable repository artifacts rather than as evaluations of scientific quality. Across the 68 entries public on 1 September 2026, these artifacts are unevenly available: README files and licenses are almost universal, whereas examples, citation information, and tagged releases are not (Figure 4). 2.2. Single-stage question answering The initial catalog also includes a lightweight question-answering function implemented as an exploratory feature rather than as part of the published service. It is deliberately disabled in public deployments: the public-build mode removes the feature, and a deployment check prevents publication of any build in which it remains reachable. The function therefore operates only in a private local environment using a locally hosted language model. A query is tokenized into Latin-script word sequences and, for Japanese and Chinese text, overlapping character bigrams. Catalog records are ranked solely by weighted token overlap across their fields; the language model is used only after retrieval, with the highest-ranked records supplied as context through Ollama[18]. The weights reflect the catalog’s primary use case of software discovery. Short, structured fields receive greater weight than the README body, with manually curated fields such as the display name, tagline, and use cases assigned the highest weights. Repository names and primary programming languages also receive relatively high weights, whereas the README has the lowest weight among the twelve indexed fields. This reduces the influence of long-form repository text relative to concise metadata prepared specifically for software discovery. Retrieval is restricted to the public dataset, so internal repository information is never included in the context passed to the model. A Python implementation supports command-line use, while a TypeScript implementation performs retrieval in the browser. In both cases, the intended configuration uses a locally hosted model so that neither queries nor retrieved context need to be sent to an external service. This first implementation uses single-stage token-overlap retrieval, without query
5
decomposition, learned re-ranking, or answer verification. Its limitations became apparent even over the 68 public records. For example, a query for variational Monte Carlo applications also returned software implementing ordinary Monte Carlo methods, because token overlap alone cannot reliably distinguish a specific method from the broader family to which it belongs. This was acceptable for an exploratory feature that was not exposed to external users. Nevertheless, it made the underlying limitation clear: because retrieval-augmented generation relies on retrieved documents as context for generation[19, 20], retrieval quality becomes critical when questions concern specific software capabilities rather than named software. Section 4 therefore examines how the same limitation reappeared when a related approach was applied to a substantially larger corpus. Before that, however, Section 3 describes the engineering and review required to make the catalog itself suitable for public deployment.
3. Making the catalog publishable The catalog was developed using Claude Code[21], a command-line coding agent. Over the eight days from the first commit to the completion of the initial development and review cycle, the repository accumulated 54 commits. The development history can be broadly divided into two phases. The first was a construction phase focused primarily on feature implementation. The second was an adversarial review phase, during which most commits addressed a numbered issue. In total, 25 issues were filed. We organize this section around these issues rather than the sequence of features added during construction. The issues provide a more informative view of the work required to transform a functioning prototype into a system suitable for public deployment. They fall into three groups: the development process used with the coding agent, the integrity of the published data, and the reliability and security of the published site. 3.1. Working with the coding agent 3.1.1. Separating implementation from adversarial review. During the construction phase, Claude Code[21] was used as the primary coding agent to implement new features. At the end of this phase, Claude Code was instructed to invoke Codex[22] in a fresh context to perform an adversarial review of the repository. Codex was asked to identify potential failure modes and to report each finding using a common structure consisting of a description of the problem, a concrete failure scenario, and a proposed correction. The resulting issues were then addressed individually. After these corrections had been applied, the repository was subjected to a further adversarial review. This second pass identified additional problems in both the data pipeline and the front end, which were addressed in two further commits. The workflow therefore did not treat review as a single terminal step; instead, implementation, independent inspection, correction, and renewed inspection formed an iterative cycle. Although this procedure does not establish reviewer independence in a formal sense, separating implementation and review across different agents and fresh conversational contexts helped expose assumptions that had not been examined during construction. In this case, rapid implementation accounted for only part of the agent-assisted development effort, while repeated attempts to identify failure modes constituted a 6
substantial fraction of the total engineering work. 3.1.2. Verifying claims against the running system. Claims that a change had been correctly implemented were verified against the deployed or locally running application rather than accepted on the basis of code inspection alone. Commit records therefore include browser-level verification notes, such as confirming that switching the interface language preserves the active filters, or that a relative date is calculated against the current date rather than the site-build date. The latter exposed a failure mode specific to static-site generation. Relative timestamps had been calculated at build time and embedded in the generated pages, causing a continuously deployed site to display increasingly stale values. The footer also labeled the site-build time as the time at which the underlying dataset had been updated. Both outputs were syntactically valid and visually plausible, but represented the wrong quantities. This experience highlighted a distinction between verifying source-code changes and verifying observable system behavior. For generated sites in particular, correctness may depend on when and where a value is computed, not merely on whether the calculation itself is implemented correctly. 3.2. Maintaining the integrity of published data The most consequential issues concerned the catalog data. Because the system assigns readiness indicators to repositories, it publishes an assessment rather than merely reproducing source metadata. Incorrect data presented as a valid assessment can therefore be more misleading than data that are explicitly marked as unavailable. 3.2.1. Distinguishing unavailable data from negative evidence. Several early failure modes produced plausible but incorrect outputs without raising exceptions. For example, if a README request failed because of rate limiting or a server-side error, the repository was recorded as having no README and received a lower readiness score. This was indistinguishable from a repository that genuinely lacked a README. The pipeline could therefore complete successfully and publish an incorrect assessment even when data acquisition had failed. Authentication failures created a similar problem: if one data source could not be accessed, the output could still pass validation as long as records from other sources remained. A failed run could also overwrite the last known-good dataset with partial output. In addition, API requests and rate-limit waits were not properly bounded, allowing an unresponsive external service to occupy a continuous-integration job for an extended period. The revised pipeline treats acquisition failures as explicit data-quality events. Enrichment failures are counted, and in the deployment configuration used in this study, a strict run fails if at least five requests are attempted and more than 20% of them fail. This threshold is an operational choice for the present system rather than a general standard. Strict mode is enabled in continuous integration, whereas local runs may continue while clearly marking the resulting data as degraded. These changes led to three operational principles: acquisition failures should not be silently converted into negative values; deployment should stop when data degradation exceeds an explicit threshold; and a failed run should never overwrite the last knowngood dataset. 7
3.2.2. Verifying privacy constraints before deployment. A central requirement of the system is that information from private repositories must not be exposed through the public catalog. This requirement is checked at deployment through three mechanisms: a leak check over the exported public dataset, a guard against unexpectedly empty output, and an inspection of the generated site. Deployment proceeds only if all checks pass in public-build mode, which excludes local-only routes. Adversarial review identified cases in which this requirement could be violated even when individual components behaved correctly in isolation. The most instructive was that operations on internal records could indirectly modify records that had already been validated for publication, because the public and internal datasets initially shared the same record objects. Slug deduplication over the internal dataset, for example, could rewrite a slug that had already passed the leak check, so that the presence of a private record changed a public URL. The review also found that deployment could proceed without running the test suite, and that the example workflow in the README omitted both the leak checks and public-build mode. The workflows were therefore revised so that tests run on every push and pull request and are also required before deployment. A separate risk was that internal documentation could be added accidentally to the public repository because it was not covered by the version-control exclusion rules. The repository configuration was revised accordingly. These findings show that privacy requirements should be verified explicitly at the publication boundary rather than assumed from the correctness of individual components. The deployment checks themselves must also be tested and included in the documented publication procedure. 3.2.3. Validating rule-based indicators against repository data. The readiness indicators are based on simple rules inferred from repository structure, metadata, and documentation. During development, several rules that appeared reasonable in isolation produced incorrect results when applied to real repositories. For example, test detection initially relied on the presence of a tests/ directory, but was later extended to inspect test configuration files, README sections, and continuousintegration settings. Inspection of the generated catalog revealed both false positives and false negatives. A project name containing -test could be mistaken for evidence of a test suite, while a C project with a large shell-based test harness could be classified as a shell project. Similar adjustments were required for license and citation detection to account for repository-specific filenames and wording. These examples show that simple rule-based indicators can be useful, but must be validated against the repositories to which they are applied. Many misclassifications were difficult to identify from the rules alone but became obvious once the results were inspected in the rendered catalog. 3.3. Maintaining the reliability of the published site 3.3.1. Testing the interface in real browsers The front end was evaluated not only through code inspection but also against accessibility guidelines[23] and actual browser behavior. This revealed defects that were difficult to identify from the source code alone. For example, several filter controls were 8
assigned the same identifier because their identifiers were generated from Japanese labels. This caused accessibility problems and, in some cases, made a label activate the wrong input. Browser testing also revealed that frequent URL updates could exceed Safari’s history-update limits, while client-side rendering could leave the main catalog page temporarily blank until JavaScript loaded. These cases show that an implementation that appears correct at the source-code level can still fail when combined with browser-specific constraints and accessibility requirements. 3.3.2. Avoiding unnecessary data in the browser. Another issue concerned the JavaScript bundle sent to the browser. A filter component imported helper functions from a module that also loaded the complete catalog data. As a result, the full catalog, including README text, was unnecessarily included in the client-side bundle. The helper functions were moved to a separate module with no dependency on the catalog data. This reduced the relevant JavaScript bundle from 455 kB to 16 kB. This case shows that code organization affects not only maintainability but also the amount of data transferred to and exposed in the browser. 3.3.3. Validating configuration values. To support multiple catalog deployments, site-specific settings such as colors and watermark images were moved into configuration files. Because some of these values were inserted into CSS, invalid or unexpected values could affect the generated page in unintended ways. The revised implementation validates color values and percentencodes image paths before inserting them into CSS. These checks are implemented as pure functions and covered by automated tests. This case shows that configuration values should be treated as inputs that require validation rather than as inherently trusted data. Taken together, these issues show that constructing a functioning prototype was only part of the development process. Reliable publication also required browser-level testing, careful control of the data sent to the client, and validation of configuration values. The most consequential problems were often not crashes, but cases in which the site continued to operate while producing incomplete, misleading, or unsafe results. The following section examines how related challenges appeared when these lessons were applied to the much larger and more heterogeneous MateriApps corpus.
4. Applying the approach to a larger existing portal Following the hackathon and the subsequent development of the repository catalog, we began exploring whether the lessons learned from this small prototype could be transferred to a larger, existing software portal. For this purpose, we applied a related retrieval workflow to MateriApps[7], a human-curated portal for materials-science software. Including the external documentation linked from its entries, the corpus examined here—its crawl completed on 3 August 2026, covering the 336 applications listed on 31 July 2026, with both the Japanese and English portal pages indexed— comprises 49,485 indexed passages, and is therefore far larger and substantially more heterogeneous than the 68-record catalog of Section 2. This work is still exploratory 9
and remains under active development. Nevertheless, the larger setting has already exposed limitations that did not appear in the initial catalog. Figure 5 summarizes the retrieval workflow used in the current MateriApps prototype and the main failure modes that emerged as the corpus was scaled up. 4.1. Retrieval workflow and routing The MateriApps corpus was organized into three levels: concise portal metadata, detailed application pages, and external documentation. Retrieval combines these sources in stages. Curated metadata is first used, when possible, to identify candidate applications. Passages retrieved from the identified candidate applications are then pooled with those returned by unrestricted vector search across all applications, using multilingual sentence embeddings[24]. A cross-encoder[25], built on the BGEM3 multilingual encoder[26], then re-ranks the pooled passages before they are passed to a locally hosted language model. Early experiments suggest that these stages address different types of retrieval failure, but also that the conditions under which each is beneficial are narrower than initially expected. Dictionary routing is effective when the question explicitly contains an application name, but cannot contribute when no such name is detected. On a set of paraphrased keyword questions with the gold answers unchanged, the Japanese dictionary never fired, and routed retrieval was therefore identical to unrestricted vector search. In English, all seven dictionary activations were false positives, causing the routing step to discard documents that unrestricted vector search had retrieved. Cross-encoder re-ranking performed best by mean reciprocal rank, among the modes that do not first restrict the candidate set of applications, on these questions because it re-orders candidates returned by unrestricted vector search rather than committing to a restricted candidate set at an earlier stage. Table 1 reports these measurements. The question sets hold the editorially assigned gold applications fixed (34 Japanese and 33 English keyword questions, one per keyword shared by two or more applications) and differ only in wording: the template form embeds the keyword string verbatim, whereas the paraphrased form describes the research situation without it. With the template wording, dictionary routing is near-perfect; with the paraphrase, it is indistinguishable from unrestricted vector search. 4.2. Scaling of approximate retrieval The larger corpus has also exposed problems beyond the choice of retrieval model. As the index grew, the hierarchical navigable small-world index[27] used for approximate nearest-neighbor search failed to retrieve some passages that were nearby in embedding space when used with its default search setting. Increasing the parameter controlling the breadth of graph traversal recovered much of the lost performance. The effect was first observed when the Japanese-only index grew from about 20,000 to about 73,000 passages, where top-5 hit rate for unrestricted vector search fell from 1.000 to 0.950 and returned to 0.990 once the search-breadth parameter (ef search) was raised from the library default of 100 to 500. Table 2 repeats the comparison on the cleaned bilingual index examined in this paper; the loss at the default setting persists and is concentrated in questions about individual applications, both factual ones (license, developer, environment) and usage-related ones.
10
4.3. Corpus construction and hidden failure modes Corpus construction introduced additional difficulties: a crawler could collect disproportionate amounts of repeated material from a single site, while aggressive deduplication could remove valid documentation together with boilerplate. One such failure was particularly instructive. Two packages had been registered twice in the portal, causing the crawler to collect their official documentation once under each registration. A cleaning rule then discarded any passage occurring under more than one registration, on the assumption that text shared between applications was boilerplate rather than documentation specific to a single application. Because the rule removed every copy rather than retaining one, the documentation of such a package disappeared entirely. For one of the two, the 22 installation pages present in the crawled data were reduced to none in the index, leaving only 17 passages, compared with 183 produced by the corrected pipeline. Aggregate benchmark results did not reveal this defect because a usage question was counted as answered whenever any passage from the correct application was retrieved, and the package’s surviving README passage was sufficient to satisfy that criterion. The defect became visible only when a real user asked how to install the package: retrieval correctly narrowed the search to that package, but the relevant installation documentation was no longer present in the index. These observations should therefore be regarded as early findings from the retrieval agent under development for MateriApps, rather than as a final evaluation of the system. They nevertheless reinforce a lesson already visible in the smaller catalog: a system can remain operational and produce plausible results even when important information has been lost. Ongoing development will focus on retrieval quality, validation of corpus construction, monitoring of application-level coverage, and extending the leakage-controlled evaluation framework from retrieval performance to generation quality, which has so far been measured only for the English paraphrased keyword questions and not yet for Japanese or for usage questions.
5. Discussion 5.1. The continuing value of editorial curation This study began from the concern that manual curation may not keep pace with the growing volume of research software. Our observations do not resolve that broader question, but they suggest that existing editorial metadata can remain useful when combined with automated retrieval. In the retrieval agent under development for MateriApps, manually assigned application names can be used directly to restrict the search space before vector retrieval, and this step requires neither a language model nor an additional learned classifier. Its benefit, however, is confined to questions in which the curated string appears verbatim; where it does not, the same mechanism contributes nothing over unrestricted retrieval. Curated metadata therefore guides retrieval for users who already know what to call what they are looking for, and the harder case remains open. A similar pattern appears in the repository catalog, where manually prepared fields such as software names, taglines, and use cases receive greater retrieval weight than automatically harvested README text. These observations suggest that automated retrieval and generative AI are better viewed as extensions of editorial work than as replacements for it. Curated metadata 11
provides a compact representation of domain knowledge that can guide automated systems before they search larger and less structured collections of documentation. 5.2. Documentation sets a ceiling on automated discovery Retrieval quality is ultimately limited by the information available in the source corpus. In the corpus examined here, whose crawl was completed on 3 August 2026, the crawler obtained no external documentation for 62 of the 336 MateriApps applications: one entry lists no official site or manual, and for the remaining 61 no page could be retrieved or retained. For these applications, improvements in retrieval architecture cannot recover information that is absent from the available sources. The repository catalog points to a related limitation. Its readiness indicators measure artifacts such as a README, license, examples, tests, citation information, documentation, and tagged releases. These are also among the materials both human users and automated systems need to understand what software does, how it should be used, and how it should be cited. Preparing research software for machine-assisted discovery therefore does not require a fundamentally new documentation practice. Rather, it reinforces established practices for making software understandable and usable: clear descriptions, explicit licensing, reproducible examples, citation guidance, and identifiable releases benefit both human readers and automated systems. 5.3. From prototype to sustainable operation The catalog and the retrieval agent examined here were developed by a small group closely involved in their maintenance. A more demanding test is whether an institutionally maintained service can remain accurate over time without continuous supervision by its original developers. The PASUMS catalog[17] is an initial step in this direction. It shows that the same generator can be applied to a different repository organization with its own branding, data sources, and publication policy. However, it has not yet operated long enough to determine whether automated generation actually reduces long-term maintenance effort rather than shifting that effort from writing records to reviewing them. This distinction matters because many of the failures identified in this study were silent: the system continued to operate while producing incomplete or misleading results. This resembles the broader problem of data and configuration failures propagating silently through machine-learning systems[28, 29]. Long-running services therefore require automated checks for changes in coverage, failed data acquisition, and other forms of degraded output rather than relying on routine inspection by the original developers. 5.4. Limitations and future work This work is an experience report based on the repository catalog and the retrieval agent under development for MateriApps, both built by a small group of developers. Its conclusions should therefore be interpreted within that scope. In particular, we did not perform a controlled comparison between agent-assisted and conventional software development. We therefore cannot determine which of the observed defects were specific to coding-agent use and which would also arise in conventional develop12
ment. The preliminary retrieval evaluation relies on automatically generated questions rather than a human-authored gold standard, and such questions cannot fully represent real user information needs. Using a local language model as a judge[30] to assess generation quality introduces further assumptions and possible biases. The real-user query that revealed missing installation documentation further shows that aggregate benchmark performance does not guarantee application-level corpus completeness. Two extensions are particularly relevant for future work. First, automatically derived repository-readiness indicators could be compared with MateriApps editorial assessments to examine where machine-readable evidence agrees with or differs from expert judgment. Second, the retrieval agent could be connected to executable environments such as MateriApps LIVE![31], which is one possible route from software discovery to reproducible, human-supervised computational workflows. While the catalog continues to be maintained, the retrieval agent under development for MateriApps remains exploratory, and the observations reported for it are correspondingly preliminary. The catalog and corpus quantities reported here—the record, application, passage, and documentation counts—describe the two systems as they stood at their respective snapshot dates: the catalog on 1 September 2026, the MateriApps portal listing on 31 July 2026, and the crawled corpus on 3 August 2026. Both continue to change, so these values should be read as a dated snapshot rather than as stable properties of either system. The complete per-record export of the 1 September catalog build, together with the derived tables behind Figure 4, and the MateriApps corpus behind the retrieval case study (portal metadata, crawled documentation pages, and the vector index) are archived in the ISSP Data Repository[32] (see Data availability).
6. Conclusion We investigated whether generative-AI technologies that accelerate research software development can also support its discovery and maintenance. The central system in this study is a repository catalog developed during a three-day hackathon and subsequently hardened for public deployment. We also explored whether the lessons from this prototype transfer to a larger, human-curated portal through a retrieval agent under development for MateriApps; this work remains exploratory, and its observations are preliminary. Coding agents enabled rapid implementation, but reliable publication required repeated review and validation. The most consequential failures were not crashes, but silent failures that produced plausible yet incomplete or incorrect outputs. Addressing them required separating implementation from adversarial review, surfacing acquisition and processing failures explicitly, testing publication safeguards, and checking system behavior against the running application rather than source code alone. Our observations also indicate that automated retrieval builds upon rather than removes the need for editorial curation and maintained documentation. Curated metadata helped guide retrieval for users who already knew what to call what they were looking for, while missing documentation could not be recovered by retrieval architecture alone. The PASUMS deployment provides an early institutional example, and similar combinations of curated metadata, automatically collected documentation, and retrieval-based assistance may be applicable to other research-software and research-data portals such as MatDaCs[33]. Connecting software discovery to exe13
cutable environments such as MateriApps LIVE![31] is one possible further direction. Taken together, these observations suggest that generative AI is most useful in research-software infrastructure when it extends, rather than replaces, existing technical and editorial work. This work provides one concrete case study of where generative AI can and cannot be relied upon in research-software infrastructure for data-driven materials science [13]. The central challenge is not simply to generate more software or more answers, but to ensure that software, metadata, documentation, and automated outputs remain verifiable, maintainable, and useful to the wider research community.
Acknowledgements The initial version of the catalog described in this paper was developed at DxMT AIMHack 2026 [14]. We thank the organizers and the other participants for discussions during and after the event. G.Y. used the ARIM-mdx data system [34] in this work. This work was supported by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan, through the Data Creation and Utilization-Type Material Research and Development Project (JPJ010337) and the project “Developing a Research Data Ecosystem for the Promotion of Data-Driven Science”. K.Y. and S.T. were additionally supported by the Japan Science and Technology Agency (JST) through the Moonshot Research and Development Program, Grant Number JPMJMS24A3, for the exploratory work on applying this approach to a larger researchsoftware portal, reported in Section 4, and for the preparation of this manuscript.
Disclosure statement No potential conflict of interest was reported by the author(s).
Data availability The generated catalog[16] and its institutional deployment for ISSP PASUMS[17] are publicly accessible, as is MateriApps[7], the portal for which the retrieval agent is being developed. These are live services that continue to change, so they do not by themselves reproduce the dated snapshots analyzed here. The data underlying this article are archived in the ISSP Data Repository[32]. The archive contains, first, the complete per-record export of the 1 September 2026 catalog build analyzed in Section 2.1 (68 repositories, with the seven readiness signals and the readiness score of each record), the derived signal table and the aggregate values of Figure 4, provenance metadata of the generating build, and a validation script that checks all reported values; and second, the corpus behind the MateriApps retrieval case study of Section 4: the portal metadata of the 336 applications listed on 31 July 2026, the crawled documentation pages (crawl completed on 3 August 2026), the 49,485-passage vector index built from them, and supplementary generation-quality evaluation runs. The catalog data and evaluation runs are released under a CC BY 4.0 license; the crawled documentation pages contain text from third-party websites and manuals, which remains under the rights of its owners and is included solely to allow verification of the reported results.
14
The source code of the catalog generator and of the retrieval agent under development for MateriApps, together with the question sets and per-question retrieval results behind Tables 1 and 2, is not publicly released at present, but is available from the corresponding author on reasonable request.
References [1] Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code Arxiv:2107.03374. Available from: https://arxiv.org/abs/2107.03374. [2] GitHub, Inc. GitHub Copilot https://github.com/features/copilot. Accessed 31 July 2026. [3] Yetiştiren B, Özsoy I, Ayerdem M, et al. Evaluating the code quality of AI-assisted code generation tools: An empirical study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT Arxiv:2304.10778. Available from: https://arxiv.org/abs/2304.10778. [4] Pearce H, Ahmad B, Tan B, et al. Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions. In: 2022 IEEE Symposium on Security and Privacy (SP); 2022. p. 754–768. Available from: https://doi.org/10.1109/SP46214.2022. 9833571. [5] Zhang B, Liang P, Zhou X, et al. Practices and challenges of using GitHub Copilot: An empirical study. In: Proceedings of the 35th International Conference on Software Engineering and Knowledge Engineering (SEKE); 2023. p. 124–129. Available from: https://doi.org/10.18293/SEKE2023-077. [6] Yoshimi K, Motoyama Y, Aoyama T, et al. Project for advancement of software usability in materials science. Science and Technology of Advanced Materials: Methods. 2025; 5(1):2564055. doi: 10.1080/27660400.2025.2564055. [7] MateriApps. MateriApps: A portal site of materials science simulation https://ma.issp. u-tokyo.ac.jp/. Accessed 31 July 2026. [8] Konishi Y, Igarashi R, Kasamatsu S, et al. MateriApps – a portal site of materials science simulation. Proceedings of Computational Science Workshop 2014 (CSW2014), JPS Conf Proc. 2015;5:011007. doi: 10.7566/JPSCP.5.011007. [9] Barker M, Chue Hong NP, Katz DS, et al. Introducing the FAIR principles for research software. Scientific Data. 2022;9(1):622. doi: 10.1038/s41597-022-01710-x. [10] Smith AM, Katz DS, Niemeyer KE, et al. Software citation principles. PeerJ Computer Science. 2016;2:e86. doi: 10.7717/peerj-cs.86. [11] Ison J, Rapacki K, Ménager H, et al. Tools and data services registry: a community effort to document bioinformatics resources. Nucleic Acids Research. 2016;44(D1):D38–D47. doi: 10.1093/nar/gkv1116. [12] Di Cosmo R, Zacchiroli S. Software heritage: Why and how to preserve software source code. In: iPRES 2017: 14th International Conference on Digital Preservation; Kyoto, Japan; 2017. p. 1–10. Available from: https://hal.science/hal-01590958. [13] Misawa T, Koizumi A, Tamura R, et al. Exploring utilization of generative AI for research and education in data-driven materials science. Science and Technology of Advanced Materials: Methods. 2025;5(1):2535956. doi: 10.1080/27660400.2025.2535956. [14] Data Creation and Utilization-Type Materials Research and Development Project (DxMT). DxMT AIMHack 2026: Construction of materials databases using AI agents https://dxmt.nims.go.jp/news/4090. Held 24–26 June 2026, Gotemba, Shizuoka, Japan. In Japanese. Accessed 31 July 2026. [15] Ji Z, Lee N, Frieske R, et al. Survey of hallucination in natural language generation. ACM Computing Surveys. 2023;55(12):1–38. doi: 10.1145/3571730. [16] Research software catalog https://k-yoshimi.github.io/repo-catalog/en/. Generated catalog site. Accessed 31 July 2026. [17] Project for Advancement of Software Usability in Materials Science (PASUMS), Insti-
15
tute for Solid State Physics, The University of Tokyo. ISSP PASUMS catalog https: //issp-center-dev.github.io/software-catalog/en/. Institutional deployment of the catalog generator. Accessed 31 July 2026. [18] Ollama. Ollama https://ollama.com/. Local language-model runtime. Accessed 31 July 2026. [19] Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems; Vol. 33. Curran Associates, Inc.; 2020. p. 9459–9474. Available from: https://proceedings.neurips. cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf. [20] Gao Y, Xiong Y, Gao X, et al. Retrieval-augmented generation for large language models: A survey Arxiv:2312.10997. Available from: https://arxiv.org/abs/2312.10997. [21] Anthropic. Claude Code https://claude.com/product/claude-code. Command-line coding agent. Accessed 31 July 2026. [22] OpenAI. Codex CLI https://developers.openai.com/codex/cli. Command-line coding agent. Accessed 19 August 2026. [23] World Wide Web Consortium (W3C). Web content accessibility guidelines (WCAG) 2.2 https://www.w3.org/TR/WCAG22/. W3C Recommendation, 12 December 2024. Accessed 19 August 2026. [24] Wang L, Yang N, Huang X, et al. Multilingual E5 text embeddings: A technical report Arxiv:2402.05672. Available from: https://arxiv.org/abs/2402.05672. [25] Nogueira R, Cho K. Passage re-ranking with BERT Arxiv:1901.04085. Available from: https://arxiv.org/abs/1901.04085. [26] Chen J, Xiao S, Zhang P, et al. M3-Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation Arxiv:2402.03216. Available from: https://arxiv.org/abs/2402.03216. [27] Malkov YA, Yashunin DA. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020;42(4):824–836. doi: 10.1109/TPAMI.2018.2889473. [28] Sculley D, Holt G, Golovin D, et al. Hidden technical debt in machine learning systems. In: Advances in Neural Information Processing Systems; Vol. 28. Curran Associates, Inc.; 2015. p. 2503–2511. Available from: https://papers.nips.cc/paper/ 5656-hidden-technical-debt-in-machine-learning-systems. [29] Sambasivan N, Kapania S, Highfill H, et al. “everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems; 2021. p. 1–15. Available from: https://doi.org/10.1145/3411764.3445518. [30] Zheng L, Chiang WL, Sheng Y, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In: Advances in Neural Information Processing Systems: Datasets and Benchmarks Track; Vol. 36; 2023. Available from: https://arxiv.org/abs/2306.05685. [31] Motoyama Y, Yoshimi K, Kato T, et al. MateriApps LIVE! and MateriApps Installer: Environment for starting and scaling up materials science simulations. SoftwareX. 2022; 20:101210. doi: 10.1016/j.softx.2022.101210. [32] Yoshimi K. Snapshot data for “building a research-software catalog with a coding agent: From hackathon prototype to public deployment” Issp data repository, the university of tokyo. Dataset, version 1.0.0: per-record export and Figure 4 values of the 1 September 2026 catalog build, and the MateriApps retrieval corpus (crawl completed 3 August 2026). https://isspns-gitlab.issp.u-tokyo.ac.jp/k-yoshimi/rsc-agent. [33] Data Creation and Utilization-Type Materials Research and Development Project (DxMT). MatDaCs (MaterialsDataCommons): A portal site supporting data-driven research https://mat-dacs.dxmt.nims.go.jp/en/. Accessed 19 August 2026. [34] Hanai M, Ishikawa R, Kawamura M, et al. ARIM-mdx Data System: Towards a nationwide data platform for materials science. In: Proceedings of 2024 IEEE International Conference on Big Data (BigData); 2024. p. 2326–2333. Available from: https: //doi.org/10.1109/BigData62323.2024.10825674.
16
fetch
build-data
export-web
build site
GitHub / GitLab API metadata, file listing, README
normalise, merge curated metadata, score readiness
public dataset only
static, searchable, bilingual
JSON between every stage: each stage re-runnable and testable on its own
Figure 1. The catalog-generation pipeline. JSON interfaces allow the output of each stage to be inspected, tested, and regenerated independently. They also allow the last known-good output of an earlier stage to be retained when a later stage fails, as discussed in Section 3.2.
17
(a) Landing page with browse-by-purpose entry points, category counts, and a keyword search interface.
(b) Catalog listing with keyword search and multicriteria filtering, including filters based on missing repository artifacts.
Figure 2. The generated catalog site[16]. The interface is available in both English and Japanese. The counts in panel (a) are calculated from the same 68 entries analyzed in Figure 4. Two additional profiles, including the PASUMS catalog[17], are generated by the same implementation using different configurations, repository sources, and publication policies.
18
Figure 3. Readiness information shown for a repository containing all seven artifacts. The checklist reports all seven readiness signals, whereas the Richness rating shown here awards one star for each of documentation, examples, and tests; the remaining four signals contribute to the separate Openness rating. Repository activity is displayed separately and is not included in the readiness score.
19
Which artefacts are present (68 repositories)
Score distribution 63
README
15.0
57
LICENSE
12.5
repositories
43
tests 36
documentation examples
33
tagged release
32
10.0 7.5 5.0
openness richness
24
citation info
2.5 0.0
0
25
50
75
100%
0
25
50
75
100
readiness score (out of 100)
Figure 4. Availability of documentation and release artifacts across the 68 catalog entries public on 1 September 2026 and shown in Figure 2. README files and licenses are common, whereas examples, citation information, and tagged releases are less consistently available. The information that an automated assistant can provide is necessarily limited by the artifacts exposed by each repository, an issue revisited in Section 5.2.
20
MateriApps corpus Portal metadata concise
Routing errors false positives / no activation
Metadata routing when possible
Application pages detailed External documentation linked
Failure modes at scale
Retrieval pipeline
Pooling candidate passages
Cross-encoder re-ranking
Unrestricted vector search multilingual embeddings
Local LLM answer generation
~45,000 passages
Approximate-search misses HNSW recall loss
Corpus construction duplicates / over-deduplication
Evaluation blind spot missing content can stay hidden
336 applications
Key lesson Scaling exposes failures in routing, approximate search, corpus construction, and evaluation—not only in the retrieval model.
Figure 5. Overview of the retrieval workflow applied to MateriApps and the main failure modes exposed at larger scale. Curated metadata routing and unrestricted vector search provide complementary candidate passages, which are pooled and re-ranked by a cross-encoder before answer generation with a locally hosted language model. Scaling revealed limitations in routing, approximate nearest-neighbor search, corpus construction, and evaluation.
21
Table 1. Retrieval on keyword questions over the MateriApps corpus (hit@10 / mean reciprocal rank, top10 passages, questions asked and answered in one language). Template questions contain the keyword string; paraphrased questions keep the same gold applications but avoid it. For the keyword Ising Model (six gold applications), for example, the template question is “Which apps are related to Ising Model?” and the paraphrased question is “Which apps can simulate lattices of binary up-or-down spins coupled to their nearest neighbours to study magnetic ordering?”; for Dynamical Mean Field Theory (DMFT) (twelve gold applications) the paraphrase is “For materials where local electron repulsion dominates, which solvers approximate the lattice by one interacting site embedded in a dynamically determined bath?”. The Japanese sets are constructed in the same way. Values are for the routing dictionary after the false-positive fix; before it, English routed retrieval on the paraphrased questions scored 0.515 / 0.252 (seven activations, all on questions that did not concern the matched application). A two-stage hierarchical variant not shown here, which first selects candidate applications by embedding similarity and then searches only within them, reached 0.676 / 0.355 (Japanese) and 0.606 / 0.434 (English) on the paraphrased questions.
Japanese (n = 34) Retrieval mode Unrestricted vector search Dictionary routing Cross-encoder re-ranking Routing + re-ranking (workflow)
English (n = 33)
template
paraphrased
template
paraphrased
0.912 / 0.700 1.000 / 1.000 1.000 / 0.880 1.000 / 0.914
0.647 / 0.332 0.647 / 0.332 0.618 / 0.457 0.618 / 0.457
0.970 / 0.700 1.000 / 1.000 0.939 / 0.889 1.000 / 0.985
0.667 / 0.291 0.667 / 0.291 0.636 / 0.417 0.636 / 0.417
22
Table 2. Unrestricted vector search on the 49,485-passage MateriApps index at the two HNSW search-breadth settings. Template questions of eight kinds, up to 25 per kind (197 Japanese, 200 English); the value 500 is the setting used throughout this paper.
Japanese (n = 197)
English (n = 200)
ef search
hit@5
hit@10
MRR
hit@5
hit@10
MRR
100 (library default) 500
0.914 0.949
0.924 0.959
0.815 0.849
0.880 0.920
0.885 0.930
0.798 0.836
23