SWHForest: A Structural Curation and Search Layer for Large-Scale Code Corpora

SWHForest: A Structural Curation and Search Layer for Large-Scale Code Corpora Federico Pennino† , Andrea Gurioli† , Maurizio Gabbrielli† , and Paolo Ferragina‡

arXiv:2611.11439v1 [cs.SE] 8 Nov 2026

†

Department of Computer Science and Engineering, University of Bologna, Italy {federico.pennino2, andrea.gurioli5, maurizio.gabbrielli}@unibo.it ‡ Sant’Anna School of Advanced Studies, Pisa, Italy paolo.ferragina@santannapisa.it

Abstract—Pre-training code language models relies on massive code corpora usually streamed over the internet. However, researchers rarely need a multi-terabyte archive in its entirety; instead, their workflow centers on curated subsets—selecting welldocumented files, filtering out repetitive boilerplate, auditing for benchmark contamination, or retrieving specific code patterns. Yet, because modern corpora are typically published as file pointers and basic attributes rather than source text, no general structural layer exists to select which files to download: curating a subset requires downloading and parsing millions of files first just to discard most of them. We present SWHF OREST, an open-source framework and public dataset that provides this missing structural layer for any pointer-based corpus without redistributing source code. A single pass over the archive parses each file into an AST and immediately discards the source text, keeping only its structural summary. Queried locally or exposed by archive providers as a server-side filtering API, SWHF OREST selects matching files first—via SQL or index intersection—and downloads only surviving candidates. Across the ten largest languages of Stack-Edu (52.1M unique files, 341M definitions), SWHF OREST answers curation queries in milliseconds while cutting downloads by 46%–72%, reducing live structural searches from hours of per-language streaming to seconds or minutes.

I. I NTRODUCTION Modern code language models are commonly trained on large code corpora collected from publicly available sources on the internet [1], [2]. Increasingly, large-scale datasets decouple corpus metadata from raw file storage: rather than shipping multi-terabyte archives of source text (as in The Stack v1 [1]), modern releases such as The Stack v2 [2] and Stack-Edu [3] distribute lightweight tables of content-addressed file identifiers (blob_id) paired with basic metadata (repository, path, byte length, license, quality score), while actual file contents (blobs) reside in remote object storage such as the Software Heritage Archive (SWH) [4] or cloud buckets. Distributing pointers rather than source text avoids duplicating terabytes of code while respecting upstream opt-outs and preserving historical provenance [5], [6], [7]. However, when the goal is data curation or analytics rather than downloading the entire archive, pointer-only distribution creates a severe bottleneck: pointer-based corpora lack a structural layer to select files by content and syntax before downloading them. Suppose a team wants to filter for documented files with at least five definitions, strip out repetitive Java/PHP getters, audit for benchmark contamination, or locate specific constructs (e.g., defer inside a Go loop or transmute

1. Corpus IDs File IDs (blob_id) + metadata (no code)

2. Hash Sample SHA-1 prefix filter (52.1M unique files)

3. Stream & Parse Stream remote blobs; Tree-sitter discards text

Precomputed Structural Layer (37.7 GB Parquet, 0 B source code) catalog.parquet: 17 syntactic metrics per file (functions, classes, docstrings, depth) functions & shapes: byte spans + rename-invariant AST fingerprints (341M defs) structure-*.parquet: search index mapping AST building blocks → file ID lists

(a) Building the structural layer (one-time pass; zero source code stored) 1. Code Pattern Find functions: def train(): ... .fit()

2. Break into Pieces Every match must have: • def named "train" • call to ".fit" • def enclosing a call

If Pattern Has 1 Piece (Exact) e.g. except{pass} or eval() Index list is exact → 0 files fetched

3. Intersect File IDs Keep only files present in all piece lists (skips 99.5% median)

If Pattern Combines Pieces e.g. def train(): ... .fit() Fetch 0.53% median & verify (ast-grep)

(b) Curating and querying locally before downloading files Fig. 1. Architecture and workflow of SWHF OREST. (a) Our pipeline streams and parses remote files once (instantiated on 52.1M Stack-Edu/SWH files), releasing compact Parquet tables and a structural index with zero source code. (b) Users filter and query the local index first, downloading only matching candidate files.

inside Rust unsafe). Because the metadata table records only byte size, answering any of these questions requires downloading and parsing millions of remote files: even on a 128-core server, streaming and parsing 52.1M files takes 7.1 hours. Existing infrastructures such as World of Code [8] and the SWH Graph [9], [10] index commit histories and provenance rather than intra-file syntax, while query platforms such as Boa [11] and CodeQL [12] must first ingest raw source code into custom databases. We designed SWHF OREST to provide this missing layer for any pointer-based or remote code corpus. Keyed by blob_id, its precomputed Parquet tables and inverted AST index join directly with upstream metadata, letting users filter files via SQL or intersect AST building-block lists locally and fetch only surviving candidates. Our contributions are:

A Storage-Agnostic Structural Layer and Tool: An open-source framework and compact AST feature index (integrated with ast-grep [13] across ten languages) that resolves single-constraint patterns from the local index without downloading files and prunes complex AST searches to a 0.53% median with zero false negatives. • A 52.1M-File Multi-Language Public Catalog: A public dataset covering a uniform 40% sample of the ten largest languages in Stack-Edu (52.1M files across 16.8M repository–language pairs), with per-file syntactic metrics, byte spans for 341M definitions (functions, methods, classes), and 64-bit structural shape hashes. • Reproducible Views and Ten-Language Evaluation: A portable JSON manifest format for citing and rebuilding filtered subsets, evaluated across all ten languages on SQL curation, boilerplate/contamination detection, and structural search.1

•

TABLE I O UR PUBLIC SWHF OREST RELEASE ACROSS THE TEN LARGEST LANGUAGES OF S TACK -E DU (40% UNIFORM SAMPLE OF 200 B–200 K B FILES ). F ILES : UNIQUE FILES ( B L O B _ I D ); D EFS : FUNCTIONS , METHODS , AND CLASSES ; D OCS : FILES WITH DOC COMMENTS (G O HAS NO DISTINCT SYNTAX ); E RR : FILES WITH SYNTAX ERROR NODES ( MAINLY UNEXPANDED C/C++ MACROS ); TABLES /I NDEX : DISK SIZES . Language

Files (M)

Defs Docs (M) (%)

Java 17.43 135.6 Python 9.48 43.4 C++ 6.45 40.0 JavaScript 5.17 39.5 C# 4.56 30.5 PHP 3.90 20.8 C 2.27 8.4 TypeScript 1.67 12.5 Go 0.75 4.8 Rust 0.45 5.1 Total

52.1

341

Err Tables Index (%) (GB) (GB)

41 0.7 23 1.4 15 9.6 13 2.1 24 1.3 50 1.1 15 12.1 24 0.5 – 0.4 35 2.9

6.15 2.82 2.25 2.04 1.65 1.17 0.64 0.66 0.26 0.23

7.52 3.16 2.71 1.77 1.81 0.86 0.74 0.66 0.26 0.29

17.9

19.8

II. S YSTEM OVERVIEW AND DATA M ODEL Reproducible Views. To share curated subsets without redisFig. 1 shows how SWHF OREST decouples structural curation from raw file storage. The framework requires only a tributing code, SWHF OREST exports a view: a JSON manifest content hash (blob_id) per file and a pluggable reader for recording the filter query, tool and grammar versions, sorted local disks, cloud buckets, or SWH. At build time (Fig. 1a), blob_id list, and SHA-256 checksum. Each language ships the pipeline reads file IDs from the manifest, streams each file view_release.json for the 40% sample; custom views are through Tree-sitter [14], and discards the source text after writ- cited by checksum and re-fetched on demand. ing its syntactic metrics, definition fingerprints (shape), and III. S TRUCTURAL I NDEXING AND P UBLIC R ELEASE AST feature posting lists into Parquet (catalog, functions, Rename-Invariant Structural Fingerprints (shape). To deshapes, structure-*). At query time (Fig. 1b), users either tect repeated code templates and renamed clones withdownload the tables and index for their target languages locally out storing source text, SWHF OREST strips all identi(37.7 GB in total; Table I) or, when hosted by an archive fier names and literal values (ignoring identifier reuse) provider, query a filtering API that returns matching blob_ids from each definition’s syntax tree, keeping only gramdirectly, fetching only the surviving source files. mar node types (e.g., identifier or return). It then Dataset Tables. Each language release (Table I) comprises hashes the subtree bottom-up into a 64-bit Merkle fingerfive Parquet tables linked by blob_id (the 40-hex Git SHA-1): print [15] from each node’s grammar type and child hashes, • rows.parquet maps each unique blob_id in the sample storing the root hash as shape in functions.parquet to a compact integer file ID (row) and upstream metadata. and shapes.parquet. Thus, def get_x(self): return • catalog.parquet has one row per file, joining represenself.x and def get_y(o): return o.y share a shape tative upstream metadata (repository, path, license, score) (matching Type-2 clones [16], retaining comments). with 17 syntactic metrics: size (bytes, lines), tree structure Structural Search Index. Text search engines often rely on (AST nodes, leaves, depth, syntax errors), documentation trigram indexes to match literal substrings [17], [18], which (comments, docstrings, comment ratio), and code elements cannot filter purely structural patterns such as nested loops (definitions, distinct shapes, imports, strings, identifiers). or empty exception handlers. At the other extreme, merging • functions.parquet lists all 341M definitions shared subtrees into a deduplicated AST DAG [19], [20] on a (function, method, or class) with their blob_id, 100k-file sample required 4.1× the compressed text size (vs. byte offsets, AST node count, docstring flag, and 64-bit ∼0.4× for our index) and kept every literal token. Instead, shape (Section III), allowing functions to be sliced from drawing on feature indexes for tree and graph databases [21], fetched files without re-parsing. [22], [23], SWHF OREST decomposes each file’s AST into • shapes.parquet groups definitions sharing an identical three simple building blocks (Fig. 1b): shape within a language, recording cluster size, mean 1) Parent–child links (E): a grammar construct and the AST size, and a sample byte span. syntactic role of its direct child (e.g., an if statement • structure-<k>.parquet stores the inverted structural and its condition or body, ignoring punctuation); index (in 2M-file shards), mapping each AST feature to 2) Literals in context (T): a literal token with its enclosing the sorted list of file IDs containing it. construct and role (e.g., "eval" as the target of a call); 3) Nesting pairs (A): an ancestor and a descendant construct 1 Code: https://github.com/halykoss/swhforest; data: https://huggingface.co/ datasets/halykoss/swhforest (Zenodo DOI pending). anywhere inside it (e.g., a for loop enclosing await).

Java PHP Rust C# C++ Go JavaScript Python TypeScript C

24.9%

33.8%

16.8% 14.4% 13.7% 12.6% 12.5% In large clusters (100k sample): • 73% rename variables/literals 11.5% (missed by exact text dedup) 8.9%

parsed files (11% rejected by one parser). Across the other nine languages, auditing 110 random definitions and 300 files per language confirmed 100% precision and 99.6% recall (7 missed definitions in 2,700 files, 5 in a PHP file without <?php). IV. A PPLICATIONS AND U SE C ASES

Use Case 1: Millisecond Corpus Curation via SQL. Selecting files across our ten-language release (52.1M rows) that exceed 4.4% 10 kB, include a docstring, and define ≥ 5 functions/methods 0% 5% 10% 15% 20% 25% 30% 35% 40% takes one SQL query over catalog.parquet. DuckDB Structurally repeated definitions (boilerplate share, %) executes it in 63 ms, returning 1.26M files (25.7 GB of source text; 38 ms for 117,203 Python files). Without SWHF OREST, Fig. 2. Percentage of non-trivial definitions (≥ 20 AST nodes) in each size-only filtering leaves 2.42M files (47.1 GB) to download language whose exact syntax-tree structure (shape) repeats more than 100 times across the corpus. In our 100k-file Python/JS sample, 73% of these and parse; SWHF OREST cuts download volume by 46% (72% repeated definitions rename identifiers or literals and are missed by exact-text on Python) without fetching or parsing rejected files. deduplication. Use Case 2: Boilerplate and Contamination Detection. Although The Stack v2 deduplicates files upstream, heavy repetition persists across definitions in different files. Using In structure-<k>.parquet, each building block is hashed shapes.parquet, finding definitions with identical AST to a 64-bit key and mapped to a sorted list of file IDs. A structure up to renaming is an instant lookup. Fig. 2 shows the query combines node types, field roles, literals, and nesting share of non-trivial definitions (≥ 20 AST nodes) whose shape constraints, translating directly to an ast-grep [13] relational repeats > 100 times: from 4.4% in C to 24.9% in PHP and rule. SWHF OREST extracts only the building blocks required 33.8% in Java, dominated by accessors and constructors. On a by every match and intersects their file lists (rarest first). 100k-file Python/JS sample, only 27% of definitions in these Because every match must contain all extracted blocks (and large clusters are byte-identical (Type-1); the remaining 73% hash collisions only add candidates), pruning never drops a differ in identifiers or literals, and 88% of these have token match (zero false negatives). When a query reduces to a single 5-gram Jaccard < 0.7 (median 0.32) to every cluster peer, block—one nesting pair (e.g., except enclosing pass at any escaping MinHash. In a manual audit of 60 non-identical pairs, depth) or one literal in a role (e.g., calling forName on any all are recurring boilerplate templates (constructors, accessors, receiver)—its posting list gives the exact set of matching files or wrapper calls) and 82% call the exact same API with (modulo 64-bit collisions, none observed) with zero blobs renamed variables (Type-2 clones). The same lookup also fetched; for multi-block queries, list intersection finds all files serves as a fast prefilter for benchmark contamination [24], where those blocks co-occur and ast-grep verifies that they [25]: matching reference solutions against our Python release form a match. We verified zero missed matches across 1.2M flags 15/164 HumanEval [26] and 227/496 MBPP [27] tasks. random pattern–file pairs in all ten grammars, 140k cached Manual inspection confirms that 10 of 15 HumanEval hits and files, and all 61 benchmark queries. 39 of 50 sampled MBPP hits implement the same task logic Public Release Construction and Validation. Although our (the rest are generic one-liners), and that most multi-statement pipeline runs on any local or remote corpus, we release a public matches again have token Jaccard < 0.7. catalog for Stack-Edu [3] (revision eeec5ca), an educational Use Case 3: Pattern and Security-Rule Search. Studies subset of The Stack v2 [2]. We cover its ten largest languages of API misuse, concurrency bugs, or training subsets need (Table I), excluding Markdown, SQL, Shell, and the two all files matching a pattern (e.g., defer in a Go loop, smallest (Ruby, Swift). Among deduplicated files of 200 B– transmute in Rust unsafe) or a static-analysis rule. SWH200 kB, we select all files whose Git SHA-1 (blob_id) starts F OREST compiles structural patterns into feature queries, below 66666666 (the lowest 40% of the hash space). Because intersects the posting lists locally, and optionally joins SHA-1 hashes are uniform, this yields a deterministic, seed- file IDs with catalog.parquet (license/size/doc filters) free 40% sample per language (39.95–40.03% of eligible files; or functions.parquet (to slice enclosing functions). We 52,145,222 total) verifiable from the ID and subsuming smaller translated 32 Semgrep [28] rules in Python/JS into this fragment prefix samples (e.g., 10% below 19999999). Streaming and (dropping negation, metavariable equality, and taint). Against parsing all 52.1M files from SWH with Tree-sitter 0.26.0 [14] the translated rules, the index misses zero matches, resolves on a 128-core node took 7.1 h (∼2,000 files/s; 84 unavailable 5 single-constraint rules directly without fetching blobs, and blobs). prunes multi-clause rules to a median of 0.3% of corpus bytes. Per-language JSON specifications map Tree-sitter node types V. S EARCH P ERFORMANCE AND P RUNING E VALUATION to functions, methods, classes, comments, and docstrings (counting lambdas/closures as functions in all languages except Local Verification Speedup and Baseline Comparison. We Python and C). On 50k Python files, our counts matched benchmark structural search across 61 queries spanning all CPython’s ast module on 99.98% of the 44,361 doubly- ten languages, using the Rust-based ast-grep engine [13] • Only 27% are byte-identical

[PHP] for { curl_exec() } (0.12%) [Py] def train { .fit() } (0.09%) [C++] for { goto } [exact] [JS] for { await } [exact] [TS] for { await } [exact] [C#] using { .ReadToEnd() } (0.66%) [C] gets() call [exact] [Rust] for { unsafe } [exact] [Rust] unsafe { ::transmute } (0.70%) [C++] for (:) { .emplace_back() } (0.74%) [Py] with { open() } body { json.load() } (1.1%) [Java] $X.forName() [exact] [Go] for { defer } [exact] [JS] addEventListener(() =>) (5.0%) 0.01 s

882× 499× 192× 175× 118× 110× 85× 81× 73× 60× 58× 37× 35× 14× 0.1 s

1s

10 s

Verification time per 10k files (s, log scale; speedup shown) SWHForest: verify index candidates only

Full scan: parse all files (ast-grep)

Fig. 3. Execution time for 14 representative structural queries out of 61 across all 10 languages (cached locally on one core; median of 3–5 runs), comparing a full scan with ast-grep against SWHF OREST index pruning followed by ast-grep verification on candidate files. Labels show the candidate percentage (or [exact] when answered directly by the index) and speedup; X { Y }: Y at any depth in X.

TABLE II E ND - TO - END SEARCH OVER THE FULL 10- LANGUAGE RELEASE (52.1M FILES ): WARM INDEX LOOKUP (104 S COLD LOAD ON P Y /JS), LIVE REMOTE FETCH (256 THREADS ), AND A S T - G R E P VERIFICATION ( E X A C T : EXACT MATCHING FILE SET FROM INDEX ). X { Y }: Y AT ANY DEPTH IN X. Lang. Query Pattern Py JS Java Rust PHP TS Py Java C# C Go C++

Cand. Matches Fetch

except { pass } 207,111 for (..) { await } 15,959 $X.forName(..) 149,572 unsafe { ::transmute } 3,321 for (..) { curl_exec() } 6,355 catch { console.error } 17,174 def train(): .. .fit() 9,109 synchronized { .wait() } 19,535 using (..) { .ReadToEnd() } 25,397 for (..) { realloc(..) } 25,157 defer { .Unlock() } 20,625 for (:) { .emplace_back() } 34,180

exact exact exact 2,959 523 14,400 7,568 18,476 22,881 5,014 17,594 13,441

Total

0 s <0.01 s 0 s <0.01 s 0 s 0.52 s 6 s 11.1 s 11 s 28.5 s 27 s 38.8 s 16 s 28.0 s 32 s 50.8 s 41 s 68.8 s 41 s 76.8 s 77 s 94.6 s 87 s 132.1 s

structural layer (metrics, byte spans, and shapes) keyed by blob_id. AST Indexing and Clone Detection. Boa [11], CodeQL [12], and Glean [34] query custom databases built from source, while Comby [35] and ast-grep [13] scan local disks. HyperAST [20], [36] hash-conses ASTs across versions of a single repository, where cross-commit sharing is high, unlike independent files. SWHF OREST combines AST-hashing clone detection [37], [38], [39] (exposing training-set duplication [24], [40]) with feature indexing [22], [23] for candidate pruning.

to verify candidate files after index pruning (Fig. 3). On cached local files (50k files each for Python/JS and 5k for the other eight languages), filtering with SWHF OREST accelerates ast-grep verification by a geometric mean of 52× (3.1×– 2,032× across all languages; 21× warm and 7× cold on Python/JS) with zero missed matches. Recording syntactic VII. L IMITATIONS , G OVERNANCE , AND AVAILABILITY roles and nesting yields 4.3× fewer false-positive candidates Limitations and Future Work. Our public release covers a than a typed-token index (leaf_type, text), and 8.8× fewer false positives on the 50 lexically anchored queries 40% uniform sample across ten of fifteen Stack-Edu languages, (207× on the 11 purely structural queries, e.g., loops with targets file- and function-level curation and indexing rather await or defer) than a substring/trigram prefilter (tighter on than cross-file analysis, and matches Type-1/2 structural clones 4 literal-heavy queries); pruning is least selective (3.1×) when rather than near-miss (Type-3) edits. As future work, we plan to: (i) port ingestion to Rust to index the remaining 60% common constructs co-occur (e.g., print in for). 8 Live End-to-End Search over 52.1M Remote Files. Across the of Stack-Edu (∼11 h) and scale to 10 + files across The full release, 16 of the 61 queries are answered directly by the Stack v2; (ii) deploy a server-side filtering API returning index with zero downloads (returning all matching blob_ids), view.json manifests without local index downloads; and while the rest only need to download and verify 0.53% of (iii) join shape clusters with the SWH provenance graph to the files (median). Table II shows end-to-end times for 12 study cross-repository reuse and AST deduplication for LLM queries across all ten languages live over the network: queries training. Governance and Availability. SWHF OREST distributes no answered directly by the index (exact) finish in <0.01–0.52 s with zero traffic, while the others download and verify the source text: only upstream metadata, syntactic counts, byte surviving files in 11 s to 2.2 min. Even compared to streaming spans, and 64-bit hashes (indexed literals are hashed, though a single language at 2,000 files/s (3.8 min on Rust to 2.4 h on common tokens are dictionary-guessable). We add no auJava, or 7.1 h for all ten), index filtering is 4× to 171× faster. thor/commit data; blobs were streamed with SWH’s agreement, users fetch under upstream terms, and future releases drop withdrawn IDs. Code (MIT): https://github.com/halykoss/swhforest; VI. R ELATED W ORK data (CC-BY-4.0): https://huggingface.co/datasets/halykoss/ Mining Infrastructures and Code Corpora. Software Her- swhforest (Zenodo DOI pending). itage [4], [5], the SWH Graph [9], [10], and World of ACKNOWLEDGMENTS Code [8] link content-addressed blobs, commits, and projects This work was made possible by Software Heritage, the at scale. Recent datasets attach licenses [29], copy-based universal source code archive: https://www.softwareheritage. reuse [30], and provenance [7] to blob IDs, while DéjàVu [31], org. Paolo Ferragina was supported by the Alfred P. Sloan CodeSearchNet [32], and The Vault [33] redistributed GitHub Foundation with the grant #G-2025-25193. functions or duplicates. SWHF OREST adds a source-free

R EFERENCES [1] D. Kocetkov, R. Li, L. Ben Allal, J. Li, C. Mou, Y. Jernite, M. Mitchell, C. Muñoz Ferrandis, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries, “The Stack: 3 TB of permissively licensed source code,” Transactions on Machine Learning Research, 2023. [2] A. Lozhkov, R. Li, L. Ben Allal, F. Cassano, J. Lamy-Poirier, N. Tazi et al., “StarCoder 2 and The Stack v2: The next generation,” arXiv preprint arXiv:2402.19173, 2024. [3] L. Ben Allal, A. Lozhkov, E. Bakouch, G. Martín Blázquez, G. Penedo, L. Tunstall et al., “SmolLM2: When smol goes big – data-centric training of a small language model,” arXiv preprint arXiv:2502.02737, 2025. [4] R. Di Cosmo and S. Zacchiroli, “Software Heritage: Why and how to preserve software source code,” in Proceedings of the 14th International Conference on Digital Preservation (iPRES 2017), Kyoto, Japan, Sep. 2017. [5] R. Di Cosmo, M. Gruenpeter, and S. Zacchiroli, “Referencing source code artifacts: A separate concern in software citation,” Computing in Science & Engineering, vol. 22, no. 2, pp. 33–43, 2020. [6] J. Katzy, R. M. Popescu, A. van Deursen, and M. Izadi, “An exploratory investigation into code license infringements in large language model training datasets,” in Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering (FORGE), 2024, pp. 74–85. [7] M. Jahanshahi and A. Mockus, “Cracks in The Stack: Hidden vulnerabilities and licensing risks in LLM pre-training datasets,” in 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), 2025, pp. 104–111. [8] Y. Ma, C. Bogart, S. Amreen, R. Zaretzki, and A. Mockus, “World of Code: An infrastructure for mining the universe of open source VCS data,” in Proceedings of the 16th International Conference on Mining Software Repositories (MSR). IEEE, 2019, pp. 143–154. [9] A. Pietri, D. Spinellis, and S. Zacchiroli, “The Software Heritage graph dataset: Public software development under one roof,” in Proceedings of the 16th International Conference on Mining Software Repositories (MSR). IEEE, 2019, pp. 138–142. [10] P. Boldi, A. Pietri, S. Vigna, and S. Zacchiroli, “Ultra-large-scale repository analysis via graph compression,” in Proceedings of the 27th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2020, pp. 184–194. [11] R. Dyer, H. A. Nguyen, H. Rajan, and T. N. Nguyen, “Boa: A language and infrastructure for analyzing ultra-large-scale software repositories,” in Proceedings of the 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 422–431. [12] P. Avgustinov, O. de Moor, M. Peyton Jones, and M. Schäfer, “QL: Object-oriented queries on relational data,” in 30th European Conference on Object-Oriented Programming (ECOOP 2016), ser. LIPIcs, vol. 56. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 2016, pp. 2:1–2:25. [13] ast-grep contributors, “ast-grep: A CLI tool for code structural search, lint, and rewriting,” https://github.com/ast-grep/ast-grep. [14] Tree-sitter contributors, “Tree-sitter: An incremental parsing system for programming tools,” https://github.com/tree-sitter/tree-sitter. [15] R. C. Merkle, “A digital signature based on a conventional encryption function,” in Advances in Cryptology – CRYPTO ’87, ser. Lecture Notes in Computer Science, vol. 293. Springer, 1988, pp. 369–378. [16] C. K. Roy, J. R. Cordy, and R. Koschke, “Comparison and evaluation of code clone detection techniques and tools: A qualitative approach,” Science of Computer Programming, vol. 74, no. 7, pp. 470–495, 2009. [17] R. Cox, “Regular expression matching with a trigram index, or how Google Code Search worked,” https://swtch.com/~rsc/regexp/regexp4. html, Jan. 2012. [18] T. Clem, “The technology behind GitHub’s new code search,” GitHub Blog, https://github.blog/engineering/architecture-optimization/ the-technology-behind-githubs-new-code-search/, Feb. 2023. [19] P. Buneman, M. Grohe, and C. Koch, “Path queries on compressed XML,” in Proceedings of the 29th International Conference on Very Large Data Bases (VLDB), 2003, pp. 141–152. [20] Q. Le Dilavrec, D. E. Khelladi, A. Blouin, and J.-M. Jézéquel, “HyperAST: Enabling efficient analysis of software histories at scale,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE). ACM, 2022, pp. 1–12. [21] R. Goldman and J. Widom, “DataGuides: Enabling query formulation and optimization in semistructured databases,” in Proceedings of the 23rd International Conference on Very Large Data Bases (VLDB), 1997, pp. 436–445. [22] X. Yan, P. S. Yu, and J. Han, “Graph indexing: A frequent structurebased approach,” in Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data. ACM, 2004, pp. 335–346.

[23] N. Zhang, M. T. Özsu, I. F. Ilyas, and A. Aboulnaga, “FIX: Feature-based indexing technique for XML documents,” in Proceedings of the 32nd International Conference on Very Large Data Bases (VLDB), 2006, pp. 259–270. [24] M. Allamanis, “The adverse effects of code duplication in machine learning models of code,” in Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!), 2019, pp. 143–153. [25] M. Riddell, A. Ni, and A. Cohan, “Quantifying contamination in evaluating code generation capabilities of language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 14 116–14 137. [26] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [27] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [28] Semgrep, Inc., “Semgrep: Lightweight static analysis for many languages,” https://semgrep.dev. [29] J. M. González-Barahona, S. Montes-León, G. Robles, and S. Zacchiroli, “The Software Heritage license dataset (2022 edition),” Empirical Software Engineering, vol. 28, no. 6, 2023, Art. no. 147. [30] M. Jahanshahi and A. Mockus, “Dataset: Copy-based reuse in open source software,” in Proceedings of the 21st International Conference on Mining Software Repositories (MSR). ACM, 2024, pp. 42–47. [31] C. V. Lopes, P. Maj, P. Martins, V. Saini, D. Yang, J. Zitny, H. Sajnani, and J. Vitek, “DéjàVu: A map of code duplicates on GitHub,” Proceedings of the ACM on Programming Languages, vol. 1, no. OOPSLA, pp. 84:1–84:28, 2017. [32] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “CodeSearchNet challenge: Evaluating the state of semantic code search,” arXiv:1909.09436, 2019. [33] D. Nguyen, L. Nam, A. Dau, A. Nguyen, K. Nghiem, J. Guo, and N. Bui, “The Vault: A comprehensive multilingual dataset for advancing code understanding and generation,” in Findings of the Association for Computational Linguistics: EMNLP 2023. ACL, 2023, pp. 4763–4788. [34] Meta Platforms, Inc., “Glean: System for collecting, deriving and querying facts about source code,” https://glean.software/. [35] R. van Tonder and C. Le Goues, “Lightweight multi-language syntax transformation with parser parser combinators,” in Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). ACM, 2019, pp. 363–378. [36] Q. Le Dilavrec and A. Zaidman, “HyperAST: Incrementally mining large source code repositories,” in Proceedings of the 22nd IEEE/ACM International Conference on Mining Software Repositories (MSR). IEEE, 2025, pp. 461–464. [37] I. D. Baxter, A. Yahin, L. Moura, M. Sant’Anna, and L. Bier, “Clone detection using abstract syntax trees,” in Proceedings of the International Conference on Software Maintenance (ICSM). IEEE, 1998, pp. 368–377. [38] L. Jiang, G. Misherghi, Z. Su, and S. Glondu, “DECKARD: Scalable and accurate tree-based detection of code clones,” in Proceedings of the 29th International Conference on Software Engineering (ICSE). IEEE, 2007, pp. 96–105. [39] H. Sajnani, V. Saini, J. Svajlenko, C. K. Roy, and C. V. Lopes, “SourcererCC: Scaling code clone detection to big-code,” in Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, 2016, pp. 1157–1168. [40] K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini, “Deduplicating training data makes language models better,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 8424– 8445.

Related documents

• CodeClusteringKrieg1870.R#1029608
• theblackcat102/evol-codealpaca-v1#786846
• openbmb/UltraData-SFT-Agent-2609#786967
• CIDR: A Large-Scale Industrial Source Code Dataset for Software Engineering Research#1387060
• The Long search#1302375