AgentTorrent: A Dataset of Context and Harness Engineering Artifacts on GitHub

AgentTorrent: A Dataset of Context and Harness Engineering Artifacts on GitHub Sebastian Baltes∗ , Seyedmoein Mohsenimofidi∗ , Levi Böhme† , Vahram Nersesyan∗ , Muhammad Auwal Abubakar‡ , Jai Lal Lulla§ , Matthias Galster‡ , Christoph Treude§ ∗ Heidelberg University, Heidelberg, Germany ‡ University of Bamberg, Bamberg, Germany

† University of Bayreuth, Bayreuth, Germany

§ Singapore Management University, Singapore, Singapore

sebastian.baltes@uni-heidelberg.de, s.mohsenimofidi@uni-heidelberg.de, levi.boehme@uni-bayreuth.de, vahram.nersesyan@stud.uni-heidelberg.de,

arXiv:2611.15436v1 [cs.SE] 11 Nov 2026

muhammad.abubakar@uni-bamberg.de, jailal.l.2025@phdcs.smu.edu.sg, mgalster@ieee.org, ctreude@smu.edu.sg

Abstract—Over the past months, developers have moved from formulating detailed prompts to engineering the context that agentic coding tools read and to configuring the underlying harnesses. For example, developers commit context files, skills, hooks, and MCP configurations to their repositories, and these files control which instructions a harness reads and which actions it can take. Developers may also commit the specs, plans, and task files that agentic coding tools and spec-driven development frameworks write for agents to implement. The AgentTorrent dataset records these files for six proprietary and open-source agentic coding tools in 10,944 engineered software projects on GitHub. It contains 62,374 configuration artifacts, the files that these artifacts import, 27,593 agentic planning documents, and more than 1.2 million commits that credit an AI tool. All heuristics used for data collection are grounded in tool documentation or derived from GitHub repositories, and we evaluate their detection accuracy with each collection. We use open-weight models to classify whether GitHub repositories host engineered software projects. Unlike most datasets, AgentTorrent links configuration artifacts, imported files, agentic planning documents, and commits that credit an AI tool within the same repositories. Configuration artifacts, planning documents, and credited commits occur together in 388 repositories. For example, our dataset allows researchers to study whether a tool is credited in commits that add instructions for that same tool and how instruction files import and contradict each other. Our data collection pipeline supports regular updates and automatically checks vendor documentation and reports unknown artifacts and agentic coding tools. The dataset and the pipeline are open-source and are archived on Zenodo. Index Terms—agentic coding tools, context engineering, harness engineering, mining software repositories, dataset

I. I NTRODUCTION Agentic coding tools such as Claude Code, Codex, and Cursor can implement software changes with a high degree of autonomy [1, 2]. To guide their work, developers engineer the context that an agentic coding tool reads and configure the harness that governs its actions [3]. Developers frequently commit artifacts to their repositories that instruct and control agent harnesses through different configuration mechanisms [2]. Such mechanisms include context files, skills, and hooks. We refer to each instance of a configuration mechanism, such as a CLAUDE.md file or a skill folder, as a configuration artifact. The instructions that an agentic coding tool reads can extend beyond individual configuration artifacts. A context file can import other files or refer to an agentic planning document,

such as a specification, plan, or task list that spec-driven development frameworks produce before implementation. Studies of context files have examined how projects instruct agentic coding tools [4, 5], but related datasets focus on individual artifacts and use different repository samples and collection dates (Table I). Therefore, researchers cannot simply merge the datasets to study the adoption of agentic software engineering across repositories and artifact types. AgentTorrent brings this evidence together for 10,944 engineered software projects on GitHub. The dataset covers Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini, and OpenCode. It contains configuration artifacts across ten configuration mechanisms, their imported files, agentic planning documents, and commits that credit an agentic coding tool. For example, AgentTorrent enables researchers to study whether planned tasks are consistent with repository instructions, permissions, and checks, and whether agentic coding tools coauthor the instructions and controls for subsequent work. AgentTorrent succeeds our previous AgentConfigs dataset [6, 7] published in March 2026, for which we collected 15,591 configuration artifacts for five tools in 4,738 repositories. The new dataset contains 62,374 configuration artifacts, an increase by a factor of approximately four. We added OpenCode because our data suggested that it was the most popular tool on GitHub that we had not yet targeted. Our new dataset also adds plugins and language servers as configuration mechanisms, resolves file imports, adds agentic planning documents, and includes commits that credit AI tools other than those we target. Each detection heuristic now cites supporting vendor documentation or examples from GitHub repositories, and our repository classification now uses open-weight models and is complemented by human validation. II. R ELATED DATASETS AgentTorrent links configuration artifacts, imported files, agentic planning documents, and credited commits for the same repositories, supporting studies of how tools contribute to the instructions and plans for subsequent work. Table I lists related datasets and how repositories were selected. The authors of SpecMine, for example, acknowledge that searching by file name mixes “serious projects with tutorials, template clones, and one-off experiments”. Our pipeline

TABLE I R ELATED DATASETS AND THEIR REPOSITORY SELECTION CRITERIA .

commit a1503db

added

“Scaffold the agentic layout. . . ”

Dataset

Content

Selection of repositories

3,797,117 SKILL.md files 470,795 specs files, 98,574 Kiro files GHAgentFiles [10] 142K+ file histories: context files, skills, subagents Zarzycki [11] root AGENTS.md or CLAUDE.md in 6,006 of 22,854 repositories MCP-Eco [12] 9,200 MCP declarations, 78,302 candidate servers Robbes et al. [1] configuration files, authors, branches, or labels: 28,418 of 128,018 projects Khosravani and configuration files, commit Mockus [13] messages, authors, bots: 180M+ repositories AIDev [14] 932,791 pull requests; five agentic coding tools AgentConfigs [6] 15,591 configuration artifacts, 148,519 commits

code search by file name code search by file names

GitSkills [8] SpecMine [9]

AgentTorrent

62,374 configuration artifacts, 10,662 imports, 27,593 agentic planning documents, 1.2M+ credited commits

GHS, 1+ year old, not a fork, 300+ commits, active in the last year 1,000+ stars, pushed in the last 90 days, not a fork or archived, 13 languages code search for MCP servers, dir., registries GHS, no forks, 5,000+ LOC, 100+ commits, active in the last 3 months World of Code without fork copies GitHub search query per agentic coding tool GHS, filters, engineered software projects GHS, 10+ stars, OSI license, ten languages; forks/archives excluded; contribution/activity filters; eng. software projects

supplements activity thresholds with a model that checks project purpose and engineering practices (Section IV-A). Selecting repositories independently of configuration also provides a denominator for prevalence: 10,944 of the 40,901 repositories in our sample contain a detected configuration artifact. Among the datasets, AgentTorrent is the only one that records configuration artifacts, imports, agentic planning documents, and credited commits for the same repositories. Interpreting artifact frequency and commit counts requires checking both whether detected records are correct (precision) and whether relevant records were missed (recall). Robbes et al., Khosravani and Mockus, and the authors of GHAgentFiles estimated detection precision from labeled samples; Khosravani and Mockus state that “absolute recall is unknown”. We evaluated both precision and recall, using repository file lists and complete commit logs to search beyond detected files and commits (Section IV-E). III. DATA D ESCRIPTION AND R ESEARCH U SE We provide our dataset as CSV files that can be joined by repository name (Table II). repos.csv describes the full sampling frame of 46,503 repositories. It contains GitHub metadata and repository IDs, which allow researchers to follow repositories across renames. It also records whether each repository hosts an engineered software project and the reasons for that classification. Classification fields are blank for 1,355 repositories, mainly because their README files were not in English (Section IV-A). A configured repository is a sampled engineered software project with at least one detected configuration artifact. Of these repositories, 4,124 (37.7%)

credits

ad

CLAUDE.md context file imports

de d

AGENTS.md context file

Claude Code

refers to

credits

commit 7f2e345

added

“Install spec-kit as the. . . ”

.specify/memory/ constitution.md planning document

Fig. 1. Motivating example: Linked files nornir-automation/nornir. The reference document appears in the collected AGENTS.md file.

and commits in to the planning

TABLE II CSV FILES IN AgentTorrent, WITH ARTIFACT AND REPOSITORY COUNTS . File

Content

Count Repos.

repos.csv context_files.csv skills.csv rules.csv settings.csv subagents.csv commands.csv mcp.csv plugins.csv hooks.csv lsp.csv

metadata and labels e.g., CLAUDE.md skill folders rule files settings files subagent definitions custom commands MCP configurations plugin manifests hook configurations LSP configurations

46,503 46,503 25,353 10,026 23,585 3,218 3,483 680 3,532 2,246 2,713 651 1,436 281 885 584 751 291 619 498 17 16

commits.csv other_commits.csv imports.csv planning-documents.csv

credited commits credited commits imports and targets specs, plans, tasks

1,281,628 72,527 10,662 27,593

9,254 2,323 2,170 417

contain at least two configuration mechanisms. Among the 417 repositories with planning documents, 388 (93.0%) also contain commits crediting one of the six target tools. For these repositories, repos.csv indicates the tools and configuration mechanisms detected. The ten mechanism CSVs record the path, creation date, commit count, and first and last commit hashes of each configuration artifact. Shared configuration artifacts, such as AGENTS.md, have any in the tool column because multiple tools can read them. The 2,820 repositories with only shared configuration artifacts cannot be attributed to a particular tool from their configuration alone. For example, researchers can investigate whether tools contribute instructions for the same or other tools. Commits introducing 23.8% of context files and 36.1% of planning documents credit a tool. In Figure 1, Claude Code receives credit in the commits adding two linked context files and a Spec Kit planning document. Researchers can also study instruction dependencies and contradictions: 1,551 imports in 176 repositories connect files beyond the initial instruction file. File dates and commit histories also support trend analyses within a release. Settings and hooks support studies of permitted commands and checks on agent actions. We also collect GitHub Agentic

1 Sampling: Filter GHS repositories (n = 46,503). Compare successive retrievals. 2 Classification: Identify engineered software projects (n = 40,901). A second model labels a sample; the first author reviews disagreements. 3 Collection: Collect files, imports, and credited commits (n = 10,944 configured repositories). Check unknown paths and credits; evaluate detection accuracy. 4 Packaging: Package CSVs, files, heuristics, and provenance records. Check consistency and agreement with the current heuristics. 5 Analysis: Compute figures and tables. Run each script twice to check reproducibility.

Fig. 2. The five stages of the AgentTorrent data collection pipeline. The gray text describes how the output of each stage is checked.

Workflows from 123 repositories, exposing tasks delegated through continuous integration. When comparing contributions, researchers can separate accepted review suggestions from other credits: Of 6,360 commits crediting Gemini, 5,261 credit it solely for an accepted review suggestion. For Copilot, this applies to 39,047 of 184,386 credited commits. We archive our dataset on Zenodo [15] with collected and imported files, all heuristics, sampling and collection records, and a README defining every column. The DOI, open formats, GitHub repository IDs, and provenance records support reuse in accordance with the FAIR principles [16]. AgentTorrent is licensed under CC BY 4.0 and the pipeline [17] under MIT; collected files and commit messages retain the licenses of their repositories. Our website lets researchers download and browse the dataset and includes a Python usage example: https://se-uhd.de/agenttorrent/. IV. DATASET C ONSTRUCTION Our pipeline automates the five stages listed in Figure 2. Rerunning it requires GitHub access tokens and an OpenRouter API key, but then operates autonomously. A. Sampling and Classification To select active repositories with multiple contributors, we queried SEART GitHub Search (GHS) [18] for non-fork repositories with a license, at least two contributors and one pull request, created at least 18 months before 2026-10-22, and with a commit in the preceding six months. GHS indexes repositories with at least 10 stars. The contribution and recency requirements address the risks of personal and inactive projects identified by Kalliamvakou et al. [19]. The pipeline added metadata from GitHub’s API and removed unavailable, private, archived, disabled, locked, and forked repositories. Further filters retained OSI-approved software licenses and the ten most common primary languages, excluded educational repositories, and required at least the median number of commits and watchers (358 commits and 5 watchers). We refer to these 46,503 repositories as our sampling frame. The classification stage of our pipeline assessed which repositories host an engineered software project [20]. We

prompted the open-weight model DeepSeek V4.1 Flash with each repository’s name, description, README, issue and pull request activity, GitHub metadata, and file outline. The model could request summaries of up to five linked pages through the read_linked_page tool exposed by an MCP server that we developed. The model evaluated exclusion criteria, such as teaching material or templates, with quoted evidence and checked for at least two engineering practices, such as testing or contributor documentation. The pipeline assigned false when an exclusion criterion applied and unclear when a criterion or the engineering evidence remained uncertain; true required that no exclusion criterion applied and that evidence supported at least two engineering practices. After removing copies with shared head commits, the pipeline retained 40,901 repositories as the sample, labeled 4,222 false and 25 unclear, and excluded 1,355 without published labels, mainly because their README files were not in English. To avoid repeated model requests when updating the dataset, our pipeline reuses a classification when the README, description, model, and classification parameters are unchanged. Keeping the original model responses and request settings lets us reconstruct the published labels without rerunning the model. We archive the prompts, classification checklist, and exclusion criteria with the pipeline on Zenodo [17]. To evaluate the repository labels assigned by DeepSeek V4.1 Flash, GLM-5.3 independently labeled a random sample drawn from the repositories that the first model had labeled, using the same prompt and tools. We sized the evaluation sample for a target margin of error of 3% at 95% confidence in the estimated agreement rate. We excluded repositories from earlier calibration and label evaluations because we had used those evaluations to refine the prompts and metadata supplied to the models. The models agreed on 1,018 of 1,039 repositories (98.0%, 95% confidence interval (CI) [96.9%, 98.7%], Cohen’s κ = 0.87). The first author reviewed the 21 disagreements and agreed with DeepSeek V4.1 Flash on 11 repositories, GLM-5.3 on 8, and neither model on 2. This review identified 10 errors in the labels assigned by DeepSeek V4.1 Flash within the evaluation sample. Additional errors may remain in that evaluation sample because the first author did not review repositories on which both models agreed. The pipeline repository records the two model labels and the first author’s decision for each disagreement. We use feedback from these reviews to continually improve the classification prompts and the metadata supplied to the models. B. Configuration, Imports, and Agentic Planning Documents A file counts as a configuration artifact when the vendor of an agentic coding tool documents it as repository-level configuration. Our path heuristics encode where and how an agentic coding tool loads configuration. For example, Cursor reads .mdc files in .cursor/rules and ignores README.md files in that directory. Each of our 358 path heuristics cites supporting documentation or source code from the vendor and records the date of verification. From 2026-11-04 to 2026-11-06, we cloned the default branches of the repositories

in our sample and copied the files matched by our path heuristics to make their contents available for analysis. We followed file renames when reading commit histories to identify the commits that added or last changed each file. Retained commit logs let us apply revised attribution heuristics without cloning repositories again (Section IV-D). We also recorded paths that resemble configuration but that agentic coding tools do not read, such as .claude/memory, with notes explaining why each path does not count as configuration. All 2,343 heuristics are stored in CSV files, and automated tests require a source for each heuristic. A file import references another file that the tool loads into its context, such as @docs/style.md in a CLAUDE.md. Imported files can import further files. Our pipeline follows these chains according to each tool’s loading rules, including Claude Code’s limit of four successive imports per chain. We compared the imports identified by our parsers with those returned by vendor implementations on 74 test texts for Claude Code and 66 for Gemini CLI. We also collected planning documents, including specifications, plans, and task lists, at paths documented by 73 frameworks and tools (e.g., Spec Kit and OpenSpec), using 523 heuristics. C. Commit Attribution Commits credit tools through author or committer identities and message text, including co-author trailers. We distinguish tool identities, accepted review suggestions, credits inserted by tools, and user conventions. We used 431 heuristics to identify commits crediting the six target tools and 402 to identify commits crediting additional tools. Vendor documentation supports 17 of these heuristics. We derived the remaining heuristics from example commits in GitHub repositories, including credits identified by the checks described in Section IV-D. D. Regular Updates and Automatic Checks We plan quarterly releases, starting updates with new GHS retrievals and reusing unchanged labels and files. Our weekly documentation watch checks 438 vendor pages and opens issues in our GitHub repository when those pages change. We review affected heuristics with Claude Code. None of the related datasets describes such a documentation check. While building AgentTorrent, we reviewed paths in configuration directories and commit credits that none of our heuristics matched. This review revealed additional configuration locations, such as .claude/workflows, and undocumented ways of crediting tools. We used these findings to extend our heuristics and will repeat the checks for future releases. For the collection run published in this release, repositoryderived heuristics detected 312,453 of 1,281,628 credited commits that documentation-based patterns would miss. Before publication, revisions added 22,371 commits, removed 275, and identified 87 additional tools. We reapplied the revised heuristics to that run’s retained commit logs without cloning the repositories again. Automated checks verified agreement between tool flags and configuration records, consistent collected commits per repository, and message credits matching the revised heuristics.

E. Evaluation We evaluate detection of configuration artifacts, imports, planning documents, and credited commits. For this release, Claude Code agents using Claude Opus 5.5 labeled samples stratified by tool, mechanism, or heuristic against vendor documentation. Separate agents using the same model relabeled 20% of each sample without seeing initial labels, disagreeing on 7 of 650 items. Other agents resolved disagreements and uncertain labels using the explanations and evidence accompanying the detecting heuristics. Weighting by the number of detections in each stratum, we estimated precision for detected configuration artifacts at 99.87% (95% CI [94.4%, 100%], 806 sampled detections) and precision for credited commits of the six tools at 99.96% (CI [96.6%, 100%], 360 sampled commits). Precision for detected file imports was 99.98% (CI [90.5%, 100%], 157 sampled imports); precision for agentic planning documents was 99.8% (CI [93.3%, 100%], 366 sampled documents). Our heuristics correctly recorded the type of credit, such as an accepted review suggestion, in 571 of 574 judgments. We reviewed repository file lists for configuration missed by our heuristics. In 100 repositories with detections, this review identified 386 configuration files or directories; our heuristics found 383 of them. Among 200 repositories without detections, one contained an undetected configuration artifact. For all 29,955 repositories without detections, the extrapolated 95% confidence interval extends up to 833 repositories with missed artifacts. We also sampled 1,000 commits uniformly from complete logs: 19 credited the six tools, and our heuristics found 18. With only 19 credited commits, the fraction detected cannot be estimated precisely. V. L IMITATIONS AND I MPROVEMENTS Detected configuration artifacts in AgentTorrent do not establish tool use, and AI tools credited in commits do not quantify contributions or capture uncredited usage. Our data collection is limited to default branches of active, licensed, engineered projects in ten languages. Changes in sampling and detection between releases can change total counts even if tool adoption remains unchanged. Our detection labels are not independent because the same model labeled and resolved samples using heuristics developed with Claude Code. Checks of unknown paths can miss shared configuration locations. In future collections, we plan to assess each path heuristic’s precision and date when artifacts first appeared at their current paths. Following renames can currently report earlier dates from before a file became configuration. VI. C ONCLUSION AgentTorrent links configuration artifacts, imported files, planning documents, and credited commits for 10,944 projects, allowing researchers, for example, to identify tools credited for adding instruction files and examine conflicts between planned tasks, instructions, and permissions. We plan quarterly releases and will continuously improve our heuristics, repository classification approach, and tool coverage.

R EFERENCES [1] R. Robbes et al., “Agentic much? Adoption of coding agents on GitHub,” ACM Transactions on Software Engineering and Methodology, vol. 35, no. 11, 2026. [Online]. Available: https://doi.org/10.1145/3822180 [2] M. Galster et al., “Configuring agentic AI coding tools: An exploratory study,” in Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware ’26). Montreal, QC, Canada: ACM, 2026, pp. 11–20. [Online]. Available: https://doi.org/10.1145/ 3805760.3814887 [3] J. L. Lulla et al., “Loop engineering: Building blocks, adoption, and impact,” 2026. [Online]. Available: https://arxiv.org/abs/2608.21884 [4] S. Mohsenimofidi, M. Galster, C. Treude, and S. Baltes, “Context engineering for AI agents in open-source software,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26). ACM, 2026, pp. 194–198. [Online]. Available: https://doi.org/10.1145/3793302.3793350 [5] W. Chatlatanagulchai et al., “Agent READMEs: An empirical study of context files for agentic coding,” ACM Transactions on Software Engineering and Methodology, vol. 35, no. 11, pp. 1–38, 2026. [Online]. Available: https://doi.org/10.1145/3840295 [6] M. Galster et al., “A dataset of agentic AI coding tool configurations,” in Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware ’26). Montreal, QC, Canada: ACM, 2026, pp. 314–322. [Online]. Available: https://doi.org/10.1145/3805760.3814922 [7] S. Baltes et al., “A dataset of agentic AI coding tool configurations,” May 2026, dataset. [Online]. Available: https://doi.org/10.5281/zenodo.20090356 [8] G. Destefanis, D. Graziotin, M. Vaccargiu, and M. Ortu, “GitSkills: A dataset of agent skills on GitHub,” 2026. [Online]. Available: https://arxiv.org/abs/2608.10906 [9] S. Agarwal et al., “Spec-driven development as prescribed and as practiced,” 2026. [Online]. Available: https://arxiv.org/abs/2608.25202 [10] G. Cardoen, T. Mens, and A. Decan, “GHAgentFiles: A dataset of coding agent file histories in GitHub repositories,” in 2026 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2026, pp. 754–758. [Online]. Available: https: //doi.org/10.1109/ICSME67971.2026.00080

[11] P. Zarzycki, “What open-source projects tell their coding agents: A census of AGENTS.md and CLAUDE.md,” 2026. [Online]. Available: https://arxiv.org/abs/2610. 31715 [12] K. Tounoussi, M. Sayagh, A. Ouni, and Y. Kashiwa, “MCP-Eco: A linked dataset of MCP servers, consumers, and development activity,” 2026. [Online]. Available: https://arxiv.org/abs/2611.06897 [13] A. Khosravani and A. Mockus, “Detecting AI coding agents in open source: A validated multi-method census of 180 million repositories,” 2026. [Online]. Available: https://arxiv.org/abs/2606.24429 [14] H. Li, H. Zhang, and A. E. Hassan, “AIDev: Studying AI coding agents on GitHub,” in Proceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26). ACM, 2026, pp. 1029–1033. [Online]. Available: https://doi.org/10.1145/ 3793302.3797249 [15] S. Baltes et al., “AgentTorrent: A dataset of context and harness engineering artifacts on GitHub,” Nov. 2026, dataset. [Online]. Available: https://doi.org/10. 5281/zenodo.24058027 [16] M. D. Wilkinson et al., “The FAIR guiding principles for scientific data management and stewardship,” Scientific Data, vol. 3, p. 160018, 2016. [Online]. Available: https://doi.org/10.1038/sdata.2016.18 [17] S. Baltes et al., “AgentTorrent: A dataset of context and harness engineering artifacts on GitHub (Pipeline),” Nov. 2026, software. [Online]. Available: https://doi.org/ 10.5281/zenodo.24058029 [18] O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in GitHub for MSR studies,” in 18th IEEE/ACM International Conference on Mining Software Repositories, MSR 2021, Madrid, Spain, May 17-19, 2021. Madrid, Spain: IEEE, 2021, pp. 560–564. [Online]. Available: https://doi.org/10.1109/MSR52588. 2021.00074 [19] E. Kalliamvakou et al., “The promises and perils of mining GitHub,” in 11th Working Conference on Mining Software Repositories, MSR 2014, Proceedings, May 31 - June 1, 2014, Hyderabad, India. Hyderabad, India: ACM, 2014, pp. 92–101. [Online]. Available: https://doi.org/10.1145/2597073.2597074 [20] N. Munaiah, S. Kroh, C. Cabrey, and M. Nagappan, “Curating GitHub for engineered software projects,” Empir. Softw. Eng., vol. 22, no. 6, pp. 3219– 3253, 2017. [Online]. Available: https://doi.org/10.1007/ s10664-017-9512-6

Related documents

• AgentTorrent: A Dataset of Context and Harness Engineering Artifacts on GitHub#1659018
• Acquiring and Verifying Repository Norms for Coding Agents#1217132
• Acquiring and Verifying Repository Norms for Coding Agents#1210358
• Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?#1141485
• SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents#10441