Mining Subscenario Refactoring Opportunities in Behaviour-Driven Software Test Suites: ML Classifiers and LLM-Judge Baselines
arXiv:2605.14568v1 [cs.SE] 14 May 2026
Ali Hassaan Mughal1*, Noor Fatima2 and Muhammad Bilal3 1*
Independent Researcher; Applied MBA (Data Analytics), Texas Wesleyan University, Fort Worth, TX, USA. 2 Independent Researcher; B.E. Computer Engineering, National University of Sciences and Technology (NUST), Pakistan. 3 Independent Researcher; M.Sc. Management, Technical University of Munich, Munich, Germany. *Corresponding author(s). E-mail(s): [email protected]; Contributing authors: [email protected]; [email protected]; Abstract Context. Behaviour-Driven Development (BDD) software test suites accumulate duplicated step subsequences. Three published refactoring patterns are available (within-file Background, within-repo reusable-scenario invocation, cross-organisational shared higher-level step), but no prior work automates which recurring subsequences are worth extracting or which mechanism applies. Objective. Rank recurring step subsequences (“slices”) by refactoring suitability (extraction-worthy), pre-map each to one of the three patterns, and quantify prevalence across the public BDD ecosystem. Method. Every contiguous L-step window (L ∈ [2, 18]) in a 339-repository / 276-upstream-owner Gherkin corpus is keyed by paraphrase-robust cluster identifiers and counted under three scopes. Sentence-BERT (SBERT) / Uniform Manifold Approximation and Projection (UMAP) / Hierarchical Density-Based Clustering (HDBSCAN) recovers paraphrase-equivalent slices. Three authors label a stratified 200-slice pool against a written rubric. An eXtreme Gradient Boosting (XGBoost) extraction-worthy classifier trained under 5-fold crossvalidation is compared with a tuned rule baseline and two open-weight Large Language Model (LLM) judges. Results. The miner produces 5,382,249 slices collapsing to 692,020 recurring patterns. Three-author Fleiss’ κ = 0.56 (extraction-worthy) and 0.79 (mechanism).
1
The classifier reaches out-of-fold F1 = 0.891 (95 % CI [0.852, 0.927]), outperforming both the rule baseline (F1 = 0.836, p = 0.017) and the better LLM judge (F1 = 0.728, p < 10−4 ). 75.0 %, 59.5 %, and 11.7 % of scenarios carry a within-file Background, within-repo reusable-scenario, or cross-organisational shared-step candidate. Conclusion. Paraphrase-robust subscenario discovery yields a corpus-wide census of BDD refactoring opportunities; pipeline, classifier predictions, labelled pool, and rubric are released under Apache-2.0. Keywords: behaviour-driven development, software test refactoring, sequence mining, software test-code duplication, machine learning for software testing, empirical software engineering
1 Introduction What this paper adds. We provide the first paraphrase-robust subsequence miner for BDD software-test specifications: a static pipeline that turns a 1.1M-step corpus of Gherkin tests into a ranked list of contiguous step-subsequences that are worth extracting, each pre-mapped to one of three concrete reuse mechanisms with a published Cucumber-Java implementation (Mughal 2024). Prior BDD work either operates at whole-scenario granularity (Binamungu et al. 2018b, 2020; Diniz et al. 2018) or catalogues smells without a corpus-scale miner (Irshad et al. 2022; Irshad and Petersen 2020; Irshad et al. 2021); sequence-mining classics (PrefixSpan, SPADE) work on exact symbol sequences and do not handle the surface-paraphrase variation BDD steps exhibit. Table 1 positions the contribution against the closest prior work along five axes. Three deliverables make the paper concrete and reproducible: (i) a slice inventory of 5,382,249 contiguous L-step windows (L ∈ [2, 18]) collapsing to 692,020 recurring cluster-id-sequence patterns across the 339-repo slice-bearing subset of the 347-repo cukereuse corpus, spanning 276 distinct upstream owners on GitHub (a mix of Organisation and User accounts; see Section 8), of which 30,955 recur across ≥ 2 distinct upstream owners; (ii) a stratified three-author labelling protocol (n = 200, 60-slice overlap, fourcategory Fleiss’ κ = 0.560) extending the pair-level cukereuse rubric to slice level; and (iii) an eXtreme Gradient Boosting (XGBoost) extraction-worthy classifier that reaches out-of-fold F1 = 0.891 (95 % bootstrap CI [0.852, 0.927]) and outperforms two open-weight Large Language Model (LLM) judges (F1 ≤ 0.728) on the same task. BDD software-test suites written in Gherkin accumulate the same maintenance hazards as unit-test corpora: copy-paste setup, parameter-varying near-duplicates, and silent drift. Our predecessor cukereuse documents the scale at the step level: across 347 public GitHub repos and 1,113,616 parsed Gherkin steps, 80.2 % of step occurrences are byte-identical copies seen elsewhere, materially higher than production-code clone rates (Kim et al. 2014). The unit of practical refactoring, however, is rarely an isolated step; it is a contiguous run of two or more steps (a slice). Mughal (2024) supplies a Cucumber-Java implementation of three reuse mechanisms
2
(Background, a reusable scenario invoked via a single-step call, and a custom higherlevel step from a two-stage code generator); Mughal et al. (2026) supplies the corpus and the per-step paraphrase-robust cluster identifier; the present paper supplies the discovery layer that automatically identifies which slices to extract and which of the three mechanisms applies. Research questions. We organise the analysis around three research questions, each tied to one Mughal2024 mechanism: RQ1 (within-file): how prevalent are step subsequences that recur across scenarios in the same .feature file (Background-block candidates)? RQ2 (within-repo cross-file): how prevalent are subsequences shared across .feature files of one repository (reusable-.feature candidates, invoked via I call feature file ⟨ X⟩)? RQ3 (cross-organisational): how prevalent are subsequences that paraphrase-cluster across repositories owned by different upstream owners (custom higher-level-step candidates)? Contributions. (1) a paraphrase-robust subsequence miner keyed by cukereuse hybrid cluster identifiers, with a Sentence-BERT (SBERT) / Uniform Manifold Approximation and Projection (UMAP) / Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) slice-embedding pass that recovers paraphrase-equivalent slices missed by exact cluster-id matching; (2) a three-scope ranking (within-file, within-repo, cross-org) that distinguishes the three Mughal-2024 mechanisms; (3) a stratified three-author labelling protocol on a 200-slice pool with a 60-slice overlap subset, extending the cukereuse pair-level methodology to slice level; (4) a two-stage XGBoost classifier (binary extraction-worthy + three-way mechanism), benchmarked head-to-head against a tuned rule baseline (McNemar χ2 = 5.69, p = 0.017) and against two open-weight LLM-judge baselines (openai/gpt-oss-120b, inclusionai/ling-2.6-1t; McNemar χ2 ≥ 14.4, p < 10−4 ) on the same humananchored pool, with all classifier predictions, per-judge raw outputs, labels, and rubric released under Apache-2.0. Table 1 Positioning against closest prior work. Columns: Granularity = duplication unit; Mode = static / dynamic / survey; Scale = largest corpus applied; Para.-robust = tolerates step-text paraphrase; Mech-mapped = candidate pre-mapped to a refactoring mechanism with published implementation. Work
Granularity
Mode
Binamungu et al. (Binamungu et al. 2018b, 2020) Diniz et al. (Diniz et al. 2018) Irshad et al. (Irshad et al. 2022; Irshad and Petersen 2020; Irshad et al. 2021) GivenWhenThen (Alcântara and Montandon 2026) PrefixSpan / SPADE (Pei et al. 2001; Zaki 2001) cukereuse (Mughal et al. 2026) This paper (cukereuse-subscenarios)
whole scenario whole scenario spec-level step + step-def symbol sequence single step contiguous slice
dynamic catalogue survey static static static static
3
Scale ∼1 repo manual 11 repos 1,720 repos — 347 repos 339 / 347 repos
Para.-robust
Mech-mapped
no no — no no yes yes
no partial no no — no yes (3)
2 Background and motivation We inherit two assets and target one gap. From cukereuse (Mughal et al. 2026): a 347-repo Gherkin corpus and a paraphrase-robust hybrid cluster_id per step (threeauthor Fleiss’ κ = 0.84 on a 1,020-pair benchmark). Re-keying the steps of a slice by their cluster_ids gives slice identity that is robust to surface paraphrasing across repositories and framework dialects. From Mughal (2024): three Cucumber-Java reuse mechanisms with a published implementation (Section 4.3): a Background: block (within-file), a reusable scenario invoked via I call feature file ⟨ X⟩ (withinrepo), and a shared higher-level step generated from a Java enum walk of features/ (cross-org). Section 6.4 of that paper notes that the discovery of which slices are worth extracting is left to manual review, and that the manual cost grows super-linearly with suite size, the gap this paper closes. cukereuse measures duplication at the single-step granularity (80.2 % of step occurrences are byte-identical copies); the practical refactoring unit, however, is the contiguous run of two or more consecutive steps, since each of the three mechanisms above operates on a multi-step block. The discovery question is therefore not which steps recur but which contiguous step subsequences recur in a way that maps onto an extraction mechanism.
3 Related work 3.1 BDD scenario quality, smells, and refactoring BDD was scoped as an engineering practice by North (2006) and characterised in agileacceptance-testing use by Solis and Wang (2011). The mapping study of Binamungu and Maro (2023) and the field study of Pereira et al. (2018) converge on the same pain points: duplicated setup, brittle assertions, opaque scenario boundaries; Scandaroli et al. (2019) report two industrial cases where this maintenance burden dominates steady-state cost. A parallel quality-rubric line (Oliveira and Marczak 2017; Oliveira et al. 2019; Wautelet et al. 2023; Sears et al. 2025) operationalises “a good BDD scenario” at the whole-scenario unit; the rubrics help an author judge a scenario but do not identify which sub-sequences recur and warrant extraction, the question we address. The closest empirical work is the Binamungu et al. trio (Binamungu et al. 2018a,b, 2020), which detects duplicate whole scenarios dynamically (by comparing executed step traces) on a small handful of repositories. We are static, operate at contiguous-subsequence granularity, and span ∼70× more repositories. Diniz et al. (2018) catalogue BDD “bad smells” on manual examples; Irshad et al. (2022); Irshad and Petersen (2020); Irshad et al. (2021) study refactoring at large-scale BDD adoption and document target identification as the limiting cost. We automate that identification step at corpus scale. A concurrent BDD dataset, GivenWhenThen (Alcântara and Montandon 2026), was released in the same cycle as cukereuse on a disjoint 1,720-repo sample with a different granularity (each scenario paired with its backing step-definition source); nothing in our pipeline depends on GivenWhenThen (GWT), but the slice-mining methodology generalises naturally to it.
4
3.2 Sequence mining Frequent-pattern mining over sequence databases has been studied since Agrawal and Srikant (1995); GSP (Srikant and Agrawal 1996), PrefixSpan (Pei et al. 2001), and SPADE (Zaki 2001) introduce the canonical pattern-growth and vertical id-list formulations, with the closely related discovery of partially ordered episodes covered by Mannila et al. (1997). The surveys of Fournier-Viger et al. (2017) and Walunj et al. (2023) catalogue the exact / approximate / gap-constrained / closed trade-offs; the closed-pattern formulation underlies our R6 closure filter (Section 8). We deliberately use exact n-gram counting on cluster-id sequences with L ∈ [2, 18] rather than PrefixSpan or SPADE in canonical form: the per-scenario search space is small, the cluster-id alphabet is finite, and exact counting is sufficient, parallelisable, and trivially correct. PrefixSpan with gap ≤ 1 is used (Phase 3) only as a robustness check for slices interrupted by one intervening step.
3.3 Code-clone detection The clone-detection literature supplies the algorithmic heritage. Abstract Syntax Tree (AST) and token-level clone detectors (Baxter et al. 1998; Kamiya et al. 2002; Li et al. 2004; Jiang et al. 2007) matured into corpus-scale tools (Sajnani et al. 2016; Saini et al. 2018), evaluated against the Bellon benchmark (Bellon et al. 2007) and more recently the Krinke and Ragkhitwetsagul (2024) re-labelling that exposes weak-Type3/Type-4 mis-labels in BigCloneBench. The Roy–Cordy taxonomy (Roy et al. 2009; Rattan et al. 2013) classifies clones from Type 1 (textual identity) through Type 4 (functional equivalence). In that taxonomy our exact cluster-id sequence match is Type 1 (cluster-ids are the token alphabet); the Phase 4 SBERT/UMAP/HDBSCAN clustering recovers Type 3/4 paraphrase equivalence by collapsing slices whose texts are semantically near-equivalent. We re-use the vocabulary; we do not re-implement clone detection.
3.4 Software-test-suite refactoring and minimisation Yoo and Harman (2012) survey regression-test minimisation, selection, and prioritisation; the subsumption / redundancy / coverage-equivalence vocabulary frames our recommendations as coverage-preserving refactoring. Software-test-smell catalogues (Garousi and Küçük 2018; Bavota et al. 2012; Spadini et al. 2022) identify duplicated setup as a maintainability hazard (the unit-test analogue of our RQ1 within-file recurrence), and Pontillo et al. (2024) extend the catalogue with an ML-based detector, a parallel to our extraction-worthy classifier. Recent software-test refactoring evidence (Spadini et al. 2024; Soares et al. 2024; Horikawa et al. 2025; Liu et al. 2024) agrees that an extraction gate plus a concrete mechanism mapping is the right intervention shape (the last finding: even strong LLM refactoring agents struggle without explicit refactoring-type guidance). The closest non-BDD analogue is the test-clone literature, which treats each test case as indivisible; we are not aware of a clone study at contiguous sub-sequences within test cases. Our slice formulation makes that granularity tractable for BDD
5
specifically by lifting identity from raw text to the cukereuse paraphrase-robust cluster id.
4 Approach 4.1 Slice as unit of analysis Let a scenario be a sequence of n Gherkin steps s1 , s2 , . . . , sn , each step parsed by the cukereuse pipeline into a record carrying (repo_slug, file_path, scenario, keyword, text, cluster_id, is_background, is_outline). A slice of length L ∈ [2, Lmax ] at position p is the sub-sequence ⟨ sp , sp+1 , . . . , sp+L−1 ⟩. Each slice carries a cluster-id sequence ⟨ cp , cp+1 , . . . , cp+L−1 ⟩, where ci is the cukereuse hybrid cluster_id of si . Two slices that share the same cluster-id sequence count as the same logical slice even when their underlying step text differs; this is our paraphrase-robust slice identity. We restrict the mining to scenarios with ≥ 2 steps remaining after dropping rows where is_background = True or where the step has no assigned cluster_id. The corpus contains 136,970 scenarios under the canonical key (repo_slug, file_path, scenario) after dropping is_background = True rows (the same key paper 1 uses implicitly via the cukereuse parser’s is_background field). After further filtering for empty scenario names (Karate-style *-only files) and length < 2 slices, the mining set is 134,635 scenarios. Pre-flight (Phase 0; Section 5) sets Lmax = 18, the 95th percentile of cleaned scenario lengths.
4.2 Three scopes A slice’s recurrence is interesting at three nested scopes (Figure 1), each mapping to a different Mughal-2024 mechanism: n_distinct_orgs ≥ 2
orgs A, B, C, . . . repo r
max_within_repo_files ≥ 2 max_within_file_recurrence ≥ 2
file f slice σ RQ1: within-file RQ2: within-repo cross-file RQ3: cross-organisational
Fig. 1 Three nested scopes for slice recurrence (RQ1 within-file ⊂ RQ2 within-repo ⊂ RQ3 crossorg); a single slice can qualify at multiple scopes.
• Within-file (RQ1). Recurrence across scenarios in the same .feature: candidate for a top-of-file Background block (Mughal 2024 Section 1). Metric: 6
max_within_file_recurrence, the maximum over (repo, file) pairs of distinct scenarios containing the slice. • Within-repo cross-file (RQ2). Recurrence across files in one repository: candidate for extraction to a reusable .feature invoked via I call feature file ⟨ ENUM⟩ (Mughal 2024 Section 4.1). Metric: max_within_repo_files, the maximum over repositories of distinct containing files. • Cross-organisational (RQ3). Recurrence across repos owned by different upstream owners: candidate for promotion to a custom higher-level step backed by an Algorithm 2 step-definition method from Mughal (2024). Metric: n_distinct_orgs, the count of distinct upstream owners (segment before the first underscore in repo_slug, equivalent to the top-level GitHub account-owner namespace; on GitHub this can be either an Organisation account or a User account, and the namespace boundary is what matters for cross-context recurrence; see Section 8). This is deliberately distinct from n_distinct_repos: a single owner publishing many repos (e.g., multi-language software-developmentkit (SDK) clients) inflates the cross-repo signal without genuine cross-owner reuse (magnitude in Section 7).
4.3 Mechanism mapping Each recurring slice is mapped to one of four conceptual targets; the Phase-6 binary classifier handles the no_op case and Phase 8 then assigns one of the three concrete mechanisms to surviving candidates. The four targets are: 1. background: prepend the slice to the file’s Background block. 2. reusable_scenario: emit a new .feature under features/reusable/⟨group⟩/ ⟨name⟩.feature, insert And I call feature file ⟨ENUM ⟩ at each call site, and regenerate the ENUM constants via Algorithm 1 of Mughal (2024). 3. shared_higher_level_step: promote the slice to a single named step backed by an Algorithm 2 step-definition method from Mughal (2024). 4. no_op: the slice is not a useful extraction target, despite recurring. A scope-driven rule-based predictor (RQ1 → background, RQ2 → reusable_scenario, RQ3 → shared_higher_level_step, otherwise no_op) provides a baseline; a learned classifier on the labelled pool refines it. Figure 2 shows the end-to-end mapping from scope signal to the concrete Mughal-2024 patch it implies.
7
scope signal
mechanism
Mughal-2024 patch
RQ1
background
prepend slice to Background: block
reusable_scenario
emit features/reusable/ ...feature; replace call sites with I call feature file
shared_higher_level_step
promote to single named step backed by Algorithm 2 step-def
no_op
do not extract; flag for human review
within-file
RQ2 within-repo
RQ3 cross-org
none no scope hit
Fig. 2 Mechanism mapping: scope signal → mechanism → concrete Mughal-2024 patch shape. Phase 6 gates whether the mapping fires; Phase 8 refines the mechanism choice.
5 Method The pipeline is organised into eleven phases, laid out in Figure 3. Phases 0–4 and 9a are fully automated; Phase 5 (three-author labelling against a written rubric) is the human critical-path bottleneck; Phases 6, 8, and 9b run after labels exist. A. Mining
B. Labelling (human)
C. Modelling
Phase 0 corpus audit 134,635 scenarios, Lmax = 18
Pilot 10-slice 4rater calibration informs Phase 2c
Phase 6 XGBoost EW classifier F1 = 0.891 [0.852, 0.927]
Phase 1 slice extraction 5,382,249 slices
Rubric written B-1..B-5 / N-1..N-5 / spec
Phase 2 + 2c n-gram + spec-suite detector 692,020 recurring patterns
Sampling stratified on L×scope×support n = 200, 60-overlap
Phase 7 LLMjudge baseline F1 ≤ 0.728 (gpt-oss-120b)
Phase 4 SBERT → UMAP → HDBSCAN 33,121 paraphrase clusters
Phase 5 3-author labels Fleiss κ = 0.560 (4-cat) Fleiss κ = 0.788 (mech)
Phase 9a preclassifier prevalence
Phase 8 mechanism predictor accuracy = 0.965 Phase 9b postclassifier headline 83.2 % / 43.7 % repos hit
Fig. 3 Pipeline overview: (A) mining → (B) three-author labelling (the human bottleneck) → (C) classifier and mechanism predictor. Solid arrows are within-column flow; dashed arrows are crosscolumn hand-offs.
5.1 Mining (Phases 0–2c) Phase 0: scenario identity. Scenarios are canonicalised by the (repo_slug, file_path, scenario) key with is_background = True rows excluded, reproducing the 136,970-scenario count of 8
Mughal et al. (2026) exactly. Length percentiles (median 6, p90 12, p95 18, p99 40) fix Lmax = 18. Phase 1: slice extraction. For each scenario of step length S, every contiguous step window (j, j + 1, . . . , j + L − 1) with L ∈ [2, min(S, 18)] and j ∈ [0, S−L] is emitted as a slice carrying (slice_id, repo_slug, file_path, scenario, ∑ position_start, 10 L, cluster_id_seq, text_seq); a 10-step scenario contributes L=2 (11−L) = 45 slices and the corpus contributes 5,382,249. Steps without an assigned cluster_id (7.4 % of the corpus) are dropped. Slices are strictly contiguous (g = 0); the gaptolerant family (Pei et al. 2001; Srikant and Agrawal 1996; Zaki 2001; Mannila et al. 1997) is left as future work because any recurrence found only under g ≥ 1 violates the rubric’s stable-context criterion and is necessarily less extraction-worthy, so the headline prevalences below are conservative lower bounds. Phase 2: exact n-gram counting. For each unique cluster-id sequence we accumulate support_total, n_distinct_files, and the per-RQ scope metrics defined in Section 4.2; patterns with support_total < 2 are dropped. The output is a 15.3 MB parquet of 692,020 distinct recurring patterns. Phase 2c: refinement. Two pilot-labelling corrections (Section 7) are applied: (i) adding n_distinct_orgs (split on the first underscore of repo_slug) as the primary RQ3 metric instead of n_distinct_repos, addressing same-owner multi-repo inflation; (ii) replacing the v1 spec-suite detector (per-file pattern density alone) with a v3 detector that requires both high density and a generator-template signature, namely > 50 distinct RQ1 patterns AND either top-pattern within-file recurrence > 100 or ≥ 30 % of distinct cluster canonical texts containing two adjacent quoted single-word placeholders. The v3 detector re-classifies legitimately heavily-duplicated software test code (e.g., Corvusoft/restq, git-town/git-town) as real signal.
5.2 Phase 4: SBERT slice-embedding clustering For each unique cluster-id sequence with a positive RQ scope signal, a slice embedding is mean-pooled from SBERT (all-MiniLM-L6-v2, 384-d) embeddings of the canonical texts of its constituent clusters (Reimers and Gurevych 2019), reduced to 50 dimensions with UMAP (McInnes et al. 2018) and clustered with HDBSCAN (Campello et al. 2013). On 619,827 input patterns this yields 33,121 paraphrase-equivalence clusters (25.3 % noise points; median size 10, p95 35). Hyperparameters are pinned in the released script.
5.3 Phase 5: rubric and three-author labelling A 200-slice pool is sampled stratified by L-bucket, scope (most-specific scope wins), and support bucket; 180 slices come from the real-signal stratum (outlier-fraction 9
≤ 0.5) and 20 from a spec-coverage stratum that probes the flagged-spec edge case. A 60-slice overlap subset is labelled by all three authors; the remaining 140 split 46/46/48. The rubric (released under Apache-2.01 ) defines (a) a four-category extraction-worthy label (yes / no / uncertain / flagged-spec) with five positive criteria B-1..B-5 and five negative criteria N-1..N-5; and (b) a four-category mechanism label (background / reusable_scenario / shared_higher_level_step / unsure) conditional on a yes verdict. A 10-slice pilot preceded the main pass and surfaced the three calibration findings that justified the Phase-2c refinement (Section 7).
5.4 Phases 6–9: classifiers and rollups Phase 6 trains an XGBoost binary extraction-worthy classifier on the labelled pool with bootstrap 95 % CIs over the out-of-fold predictions. Phase 7 re-labels the same pool with two open-weight LLM judges (Section 6.8). Phase 8 extends the classifier with a three-way mechanism head, against a rule-based scope-driven baseline. Phase 9 computes corpus-level prevalence: 9a is the raw upper bound, and 9b applies the Phase-6 gate to produce the practitioner-facing headline.
6 Results 6.1 Corpus characterisation (Phase 0) The cleaned mining set is 134,635 named, non-Background, length-≥ 2 scenarios across 339 repos and 21,946 .feature files. Scenarios with fewer than two clustered steps under the Mughal et al. (2026) filter cannot match anything and are dropped (8,014 scenarios; 5.9 %), leaving 126,621 scenarios as slice input. All 339 repos retain at least one slice. Scenario length: median 6, p90 12, p95 18, p99 40, max 1,373.
6.2 Slice inventory and recurring patterns (Phases 1, 2 + 2c) Slice generation yields 5,382,249 slices, monotonically decreasing in L (L = 2: 836,737 down through L = 18: 118,069). After deduplication on cluster-id sequence these collapse to 2,349,063 unique patterns, of which 692,020 have support_total ≥ 2 (the recurring pool). Figure 4 shows how this pool distributes across the 17 slice lengths and how the R1–R6 verification filter chain (Section 8) thins each length-bucket. Within the recurring pool, 403,745 patterns carry a within-file Background signal (max_within_file_recurrence ≥ 2), 246,007 carry a within-repo reusable-scenario signal (max_within_repo_files ≥ 2), and 30,955 carry a cross-organisational signal (n_distinct_orgs ≥ 2). The cross-owner count corrects the naive cross-repo metric (62,771): 31,816 patterns (51 %) of the apparent cross-repo signal are actually same-owner cross-repo (e.g., DataDog’s multi-language SDK clients). Table 2 shows representative top-ranked patterns per RQ scope under the real-signal restriction. RQ1 surfaces dense within-file repetition of background-style invariants; RQ2 surfaces shared assertion macros within a single repository; RQ3 surfaces generic HTTP request/response idioms shared across upstream owners. 1 Rubric and per-author labels at amughalbscs16/cukereuse_subscenarios_release; aggregated labels in methodology/labels.jsonl.
10
Pattern count by slice length L, pre- vs post-R1–R6 filter chain pre-R6 (Phase-6 EW = 464,073) post-R1...R6 closed (84,564)
patterns
104
103
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
slice length L
Fig. 4 Pattern count by slice length L (log-scaled), pre and post the R1–R6 filter chain. R6 cannot apply at Lmax (no L = 19 super-pattern exists), inflating the post-filter L = 18 bucket relative to neighbours.
6.3 Slice clustering (Phase 4) Of the 619,827 patterns with at least one positive scope signal, HDBSCAN on UMAPreduced SBERT slice embeddings produces 33,121 paraphrase-equivalence clusters. Cluster size is right-skewed: median 10 patterns, p95 35, max 607. Noise points account for 25.3 % (156,617 patterns), i.e., patterns whose slice embeddings do not cluster densely with any other.
6.4 Pre-classifier corpus-level prevalence (Phase 9a) Two views of scenario-level prevalence: full (all recurring patterns) and real-signal (patterns whose majority of occurrences fall on non-spec-suite files under the Phase-2c v3 filter). The real-signal column is the defensible headline. Recurring structure is pervasive within-file and within-repo (∼75 % and ∼70 % of scenarios respectively under the real-signal restriction); cross-organisational recurrence is rarer (∼17 %) but non-trivial. The full-vs-real-signal gap is largest at RQ1 (90.1 % → 75.1 %) and negligible at RQ3 (17.1 % in both), confirming that specsuite generation creates dense within-file recurrence but rarely escapes its originating upstream owner. The v3 spec-suite detector (Section 5.1) shrinks the outlier list from 2,459 (11.2 %) to 154 files (0.7 %), dominated by local-web-services/local-web-services (121 files) and the DataDog application-programming-interface (API) client suites; reclassifying the 2,305 dropped files raises the real-signal pattern count from 179,019 (v1) to 616,464 (v3) of the 692,020 recurring patterns.
6.5 Labelling results (Phase 5) Three authors labelled the 200-slice pool under the Section 5.3 rubric; aggregated distribution and inter-rater agreement are in Table 4.
11
Table 2 Representative top-ranked recurring patterns per RQ scope (real-signal only): RQ1 by max_within_file_recurrence×L; RQ2 by max_within_repo_files×L; RQ3 by n_distinct_orgs.
L
Canonical step text
RQ1 top: within-file Background candidate 2 sidekiq should have 0 "event-log" jobs sidekiq should have 1 "request-log" job 3
sidekiq should have 1 "webhook" job sidekiq should have 1 "event-log" job sidekiq should have 1 "request-log" job
RQ2 top: within-repo cross-file reusable candidate 4 the following style: the following input: I cite all items the result should be: 4
I generate a type for the schema I construct an instance of the schema type from the data I validate the instance the result will be <valid>
RQ3 top: cross-organisational shared candidate 2 method get status 200
Signal
Mechanism
file-rec=379, support=2,504, single repo
Background
file-rec=219, support=854, single repo
Background
repo-files=438, Reusable support=438, 1 scenario org repo-files=368, support=1,746, 1 org
Reusable scenario
11 orgs, 11 Shared repos, sup- higher-level port=4,897 step
2
method post status 200
11 orgs, 11 Shared repos, sup- higher-level port=3,438 step
3
the output should contain: the output should not contain: the output should not contain:
8 orgs, 8 repos, Shared support=65 higher-level step
Under the Landis–Koch interpretation (Landis and Koch 1977), the four-category extraction-worthy κ = 0.56 is moderate and the five-category mechanism κ = 0.79 is substantial. Most extraction-worthy disagreement concentrates on three failure modes: (a) the yes-vs-no boundary on L = 2 cross-organisational trivial content (e.g., method post/status 200), where the rubric’s worked Example 3 explicitly admits a borderline call; (b) the uncertain bucket, used 0–11 times by different authors; and (c) the flagged-spec vs. no boundary on placeholder-free heavily-duplicated fixtures. Three slices ended in three-way ties (released as tie, not tie-broken). The mechanism label follows almost mechanically from scope once extraction-worthiness is established, explaining its higher κ. The overlap majority is 41 yes / 9 no / 7 flagged-spec / 3 ties; excluding the spec-coverage stratum, the remainder is consistent with the population that survives
12
Table 3 Corpus-level prevalence by RQ scope across three pruning stages: full (all 692,020 recurring patterns), real-signal (non-spec-suite-majority under Phase-2c v3, the defensible pre-classifier headline), and post-EW (after the Phase-6 extraction-worthy gate). Scenario-level n is over 126,621 scenarios; repository-level n is over 339 repos. RQ3 uses n_distinct_orgs ≥ 2.
Full Scope
n
Real-signal %
n
Post-EW
%
n
%
Scenarios w/ ≥ 1 recurring slice 121,701 96.1 — — — — RQ1 (within-file Background) 114,131 90.1 95,126 75.1 95,007 75.0 RQ2 (within-repo reusable) 106,496 84.1 87,585 69.2 75,396 59.5 RQ3 (cross-org shared-step) 21,690 17.1 21,690 17.1 14,864 11.7 Repository-level (n=339 repos): Repos w/ ≥ 1 RQ2 candidate Repos w/ ≥ 1 RQ3 candidate
300 88.5 167 49.3
— —
— —
282 83.2 148 43.7
Table 4 Three-author labelling outcomes on the 200-slice stratified pool (60-slice overlap, 140 split 46/46/48). Fleiss’ κ (Fleiss 1971) is computed over the overlap under the four-category extraction-worthy and five-category mechanism labels (n/a included for non-yes verdicts).
Author A
Author B
Author C
Per-author extraction-worthy distribution: yes / no / uncertain / flagged-spec 77/12/4/13 total 106
72/11/11/12 106
72/23/0/13 108
0.850 0.933
0.560 / 0.788 0.750 0.800
Inter-rater agreement on the 60-slice overlap: Fleiss’ κ (extract 4-cat / mech 5-cat) Pairwise A–B / A–C / B–C (extract) Pairwise A–B / A–C / B–C (mech)
0.717 0.800
Majority verdict on the overlap (extraction-worthy): yes / no / flagged-spec / 3-way tie
41 / 9 / 7 / 3
the Phase-2c v3 filter: most surviving patterns are extraction-worthy, and the rest concentrate in the trivial-content tail at low L.
6.6 Extraction-worthy classifier (Phase 6) An XGBoost binary classifier (Chen and Guestrin 2016) (n_estimators=200, max_depth=4, learning_rate=0.1) is fit on the 197 non-tie labelled slices. Positive class = yes; negative = no ∪ uncertain ∪ flagged-spec. Features: L, support_total, the three count features (n_distinct_repos/orgs/files), max_within_file_recurrence, max_within_repo_files, outlier_fraction, has_template_structure, scope one-hots, and three derived ratios. Evaluation is
13
5-fold stratified cross-validation (CV) with 1,000-bootstrap 95 % percentile confidence intervals (CIs) (Efron and Tibshirani 1993). The trained classifier is applied to the scope-eligible pattern population: those satisfying at least one of {RQ1, RQ2, RQ3-cross-org} (n = 595,857), a strict subset of the 619,827 patterns clustered in Phase 4. The 23,970-pattern residual is same-owner cross-repo recurrence (RQ3 by the naive cross-repo metric but not by n_distinct_orgs ≥ 2), which maps to no Mughal-2024 mechanism and is excluded from classifier input. Table 5 Phase-6 extraction-worthy classifier (XGBoost, binary). Out-of-fold metrics from 5-fold stratified CV over 197 non-tie labels; 95 % CIs are 1,000-bootstrap percentile intervals.
Metric Precision Recall F1 ROC-AUC
Median
95 % CI
0.868 0.916 0.891 0.881
[0.811, 0.920] [0.869, 0.959] [0.852, 0.927] [0.818, 0.931]
The classifier reaches an out-of-fold F1 = 0.891 (95 % CI [0.852, 0.927]) and bootstrap-median area under the receiver-operating-characteristic curve (ROC-AUC) = 0.881. Figure 5 plots per-fold scores against the bootstrap median and 95 % CI; the across-fold variance is modest, with fold-1 the lower outlier on F1 and ROC-AUC. Non-ML rule baselines. To check that XGBoost earns its complexity, we compare against two baselines on the same 197-item out-of-fold setting. The trivial all-yes predictor reaches F1 = 0.841 (driven by the 72.6 % base rate of yes on the labelled pool); a singlefeature rule outlier_fraction < 0.3 reaches F1 = 0.836 (P = 0.801, R = 0.874). XGBoost’s lift over the rule is concentrated in precision (P = 0.868 vs 0.801), with recall roughly matched. On the 45 discordant items (31 XGB-right, 14 rule-right) McNemar’s test gives χ2 = 5.69, p = 0.017: the lift is statistically significant though modest. The top five features by mean fold-level XGBoost importance (Figure 6) are outlier_fraction (0.33), support_total (0.12), L (0.11), ratio_within_repo (0.10), and max_within_file_recurrence (0.09); the dominance of outlier_fraction reflects the Phase-2c spec-suite signal, with density features as a secondary cluster. After per-fold evaluation, we re-fit on all 197 non-tie labels and apply the model to the 595,857 scope-eligible patterns, of which 464,073 (77.9 %) are predicted extraction-worthy.
6.7 Mechanism predictor (Phase 8) Conditional on a slice being labelled extraction-worthy, the mechanism label predicts which of the three Mughal-2024 targets (background, reusable_scenario, 14
Phase-6 classifier: per-fold scores and bootstrap 95\,\% CIs 1.00 0.95
score
0.90 0.85 0.80 0.75
fold 1
fold 3
fold 5
fold 2
fold 4
bootstrap median + 95\,\% CI
0.70 Precision
F1
Recall
ROC-AUC
Fig. 5 Phase-6 classifier: per-fold precision, recall, F1 , and ROC-AUC across 5-fold stratified CV, overlaid with the bootstrap median and 95 % CI from Table 5. Phase-6 classifier feature importance (extraction-worthy gate) 0.326
outlier\_fraction 0.124
support\_total
0.114
L
0.097
ratio\_within\_repo
0.088
max\_within\_file\_recurrence 0.058
n\_distinct\_repos
0.055
ratio\_within\_file
0.048
max\_within\_repo\_files
0.042
n\_distinct\_orgs n\_distinct\_files scope\_RQ2 0.00
0.035 0.013
0.05
0.10
0.15
0.20
0.25
0.30
0.35
XGBoost mean fold-level feature importance
Fig. 6 Phase-6 classifier feature importance, mean across the five XGBoost folds. Three feature families dominate: spec-suite signal (outlier_fraction), recurrence density (support_total, L, max_within_file_recurrence), and structural ratios. Scope one-hots are near-zero because L and the density features already encode scope.
shared_higher_level_step) is appropriate. Among the 143 yes-labelled slices, the mechanism distribution is 24 background, 70 reusable_scenario, and 49 shared_higher_level_step; four unsure verdicts on yes-labelled slices were resolved by majority vote during aggregation. Rule-based scope-driven baseline. The natural rule-based predictor maps RQ1 → background, RQ2 → reusable_scenario, and RQ3 → shared_higher_level_step. On the 143 labels this baseline achieves accuracy 0.972 and macro-F1 = 0.965, with the four mismatches all in the RQ1 bucket where the labellers preferred reusable_scenario
15
over background (a slice that recurred in one file plus one other file qualifies for both mechanisms; the labellers preferred the cross-file mechanism for those four cases). Learned multi-class XGBoost predictor. A multi-class XGBoost classifier (same features as Phase 6, objective=multi:softprob) reaches out-of-fold accuracy 0.965 and macroF1 = 0.955 in 5-fold stratified cross-validation, statistically indistinguishable from the rule-based baseline at this sample size. Per-class precision / recall on out-of-fold predictions are 0.85/0.96 for background, 0.99/0.94 for reusable_scenario, and 1.00/1.00 for shared_higher_level_step. The shared_higher_level_step class is perfectly separable in the feature space because cross-organisational recurrence (n_distinct_orgs ≥ 2) is a hard precondition for the class.
Phase-6 EW patterns assigned
Application to the 464,073 predicted-extraction-worthy patterns. Re-fitting on all 143 labels and applying to the predicted-EW population yields 232,129 background (50.0 %), 201,251 reusable_scenario (43.4 %), and 30,693 shared_higher_level_step (6.6 %). XGBoost and the rulebased baseline agree on 98.6 % of patterns; differences concentrate at the background ↔ shared_higher_level_step boundary, where the learned predictor exploits a secondary cross-org signal. Both predictors are released so a downstream command-line tool (CLI) can pick its preferred one. Figure 7 shows the full mechanism distribution for both predictors side by side. Phase-8 mechanism distribution over 464,073 predicted-EW patterns XGBoost vs rule baseline; agreement = 98.6\,\% 250000
232,129
238,026 201,251
200000
201,356
XGBoost (Phase 8) rule-based scope mapping
150000 100000 50000
30,693
24,691
0 background (RQ1)
reusable\_scenario (RQ2)
shared\_higher\_level\_step (RQ3)
Fig. 7 Phase-8 mechanism distribution over the 464,073 predicted-EW patterns: XGBoost vs the rule-based scope-mapping baseline. Predictors agree on 98.6 %; differences concentrate at the background ↔ shared_higher_level_step boundary, where the learned predictor exploits a secondary cross-org signal.
6.8 LLM-judge baseline (Phase 7) To pre-empt the did-you-try-an-LLM-baseline? reviewer concern, the 200-slice pool is re-labelled with two open-weight LLMs via OpenRouter (Zheng et al.
16
2023; Gu et al. 2024) at temperature = 0: openai/gpt-oss-120b (120 B) and inclusionai/ling-2.6-1t (1 T-parameter MoE). Each query supplies the condensed rubric (B-1..B-5, N-1..N-5, spec-suite handling, calibration notes), the slice’s L-step canonical text, the per-scope recurrence signals, and outlier_fraction from Phase 2c. Both models return a JSON verdict with extraction_worthy and mechanism fields; the parser tolerates markdown fences and prose around the JSON. Agreement results. gpt-oss-120b reaches κ = 0.348 (fair) with F1 (yes) = 0.728; ling-2.6-1t reaches κ = 0.243 with F1 (yes) = 0.587 (Table 6). Both models are highly precise on the yes class (P = 0.91 / 0.94) but conservative in recall (R = 0.61 / 0.43), under-calling extraction-worthiness and over-calling no or flagged-spec. Inter-LLM Fleiss’ κ on the full 200 patterns is 0.393 (4-cat) / 0.304 (binary), materially below the human triad’s 0.560 (4-cat) and pairwise raw agreement 0.717–0.850 (Table 4). On the conditional mechanism-given-yes task both models reach ≥ 0.95 accuracy: once the extractionworthy gate is passed, the three-way mechanism call is the easier sub-task. Table 6 LLM-judge baseline (Phase 7) on the two full-coverage open-weight models. nv counts parseable verdicts. Binary metrics (accb , κb , F1 (yes)) are versus the human aggregated label on the n=197 non-tie subset; mech. is mechanism accuracy conditional on both rater and model saying yes.
Rater
nv
accb
κb
F1 (yes)
mech.
gpt-oss-120b ling-2.6-1t
200 200
0.67 0.56
0.348 0.243
0.728 0.587
0.977 0.951
Implication for deployment. LLM-as-judge with off-the-shelf open-weight models does not match the Phase-6 classifier on this task: F1 = 0.891 (95 % CI [0.852, 0.927]) against F1 = 0.728 / κb = 0.348 for the better LLM. McNemar’s test on the discordant out-of-fold predictions confirms the gap is statistically significant: against gpt-oss-120b, XGBoost is right-only on 52 items and the LLM is right-only on 19 (χ2 = 14.4, p = 1×10−4 ); against ling-2.6-1t, 71 vs 17 (χ2 = 31.9, p < 10−4 ). The classifier costs one 200-slice three-author label pool plus a CPU-minute to fit; scoring the 595,857 scope-eligible patterns is a single batch predict_proba call. The Phase-6 classifier therefore remains the primary gate; the LLM-judge numbers stand as a methodological reference (Figure 8).
6.9 Post-classifier corpus headline (Phase 9b) The post-EW columns of Table 3 report the same prevalences after the Phase-6 gate. The classifier prunes 0.1 pp from RQ1 (75.1 % → 75.0 %), 9.6 pp from RQ2
17
F1(yes) on the binary EW task
Phase-6 classifier vs Phase-7 LLM judges on the 200-slice pool (humans aggregated, 197 non-tie items) 1.0
0.891 [0.852, 0.927] 0.728 κb = 0.348
0.8
0.587 κb = 0.243
0.6 0.4 0.2 0.0 Phase-6 XGBoost
gpt-oss-120b
ling-2.6-1t
Fig. 8 Yes-class F1 on the binary extraction-worthy task vs the human aggregated label (197 nontie items). Phase-6 classifier reaches F1 = 0.891; the better LLM judge reaches F1 = 0.728. Error bar is the classifier bootstrap 95 % CI; LLM annotations show Cohen’s κb .
(69.2 % → 59.5 %), and 5.4 pp from RQ3 (17.1 % → 11.7 %); pruning concentrates at RQ2 and RQ3 because most filtered candidates are L=2 trivial-content crossorg HTTP pairs or within-repo low-content macros that the rubric flags under N-4 (slice-too-short-for-value). At repository scale, 83.2 % of repos still host an EW reusable-scenario candidate and 43.7 % host an EW cross-org shared-step candidate.
7 Discussion 7.1 Pilot labelling and rubric calibration A 10-slice pilot surfaced three calibration findings. Finding 1 (owner vs repo). Two RQ3 entries with n_distinct_repos = 5 were one upstream owner’s multilanguage SDK clients (e.g., DataDog’s Go/Java/Python/Ruby/TS): cross-repo but not cross-owner. n_distinct_orgs was therefore adopted as the primary RQ3 metric; the same-owner-multi-repo cohort is 31,816 patterns (51 % of the naive cross-repo signal). Finding 2 (long-L sub-extraction). A length-18 pilot slice (Kolibri coach lesson-report workflow) contained four repetitions of an inner 4-step pattern; the right target is the inner pattern, not the enclosing block. The current rubric is binary on the slice as given; sub-slice preference is future work. Finding 3 (spec-suite v1 over-broad). A heavily-duplicated but template-free pilot slice (Corvusoft/restq, support 786) was wrongly flagged by the v1 detector. The v3 detector (Section 5.1) requires both density and a template-structure signature, shrinking the outlier list from 2,459 (11.2 %) to 154 files (0.7 %). All three findings translate directly into Phase6 classifier features: a template-structure flag (Finding 3) and n_distinct_orgs distinct from n_distinct_repos (Finding 1) are implemented; sub-slice preference (Finding 2) remains future work (Phase 2.5).
18
7.2 Cross-organisational signal magnitude The 30,955 RQ3 candidates are 4.5 % of the recurring pool but carry disproportionate practical interest: they are the only candidates for which the shared-higher-level-step mechanism is the appropriate target. The leaderboard (Figure 9) is dominated by what Binamungu et al. (2018b) call infrastructural duplication (HTTP request-response idioms such as method get / status 200 across 11 distinct upstream owners, and CLI output assertions) rather than domain duplication. The returns drop sharply past L = 3, so the shared-higher-level-step mechanism is most useful for short, frequentlyrepeated infrastructural macros rather than long business-logic sequences. Top-20 cross-organisational (RQ3) patterns by quality score L=3, νorg=3 | 23,34,7 ... L=3, νorg=3 | 34,7,11 ... L=4, νorg=3 | 23,34,7,11 ... L=4, νorg=3 | 34,7,11,13 ... L=5, νorg=3 | 23,34,7,11,13 ... L=2, νorg=7 | 222,284 ... L=3, νorg=2 | 41,35,89 ... L=4, νorg=2 | 0,41,35,89 ... L=2, νorg=6 | 240,328 ... L=2, νorg=5 | 271,153 ... L=3, νorg=3 | 7,11,190 ... L=3, νorg=3 | 23,7,11 ... L=4, νorg=3 | 23,7,11,13 ... L=4, νorg=3 | 34,7,11,190 ... L=5, νorg=3 | 23,34,7,11,190 ... L=2, νorg=5 | 0,12 ... L=2, νorg=3 | 7,388 ... L=3, νorg=3 | 7,388,190 ... L=2, νorg=5 | 102,3 ... L=3, νorg=2 | 12,0,41 ...
0
50000
100000
150000
200000
250000
quality\_score = nscen ⋅ √ support ⋅ norgs
Fig. 9 Top 20 cross-organisational (RQ3) candidates by quality score after R6 closure. Fingerprints are abbreviated to the first six cluster ids; the highest-ranked candidates are predominantly L ∈ {2, 3} infrastructural macros, not long business-logic sequences.
7.3 Industrial relevance Irshad et al. (2022) report that BDD software-test specification refactoring is a recurring but under-tooled industrial task in which the manual cost of identifying targets is the limiting factor; Kim et al. (2014) find the same cost asymmetry at Microsoft on production code, where discovery and prioritisation dominate. The post-EW columns of Table 3 are the practitioner-facing prior: 83.2 % of repos still host an EW reusablescenario candidate, 43.7 % host an EW cross-org shared-step candidate, and the median repo contains 115 recurring patterns (p25 = 23, p75 = 555) before the gate. A 19
practitioner running cukereuse-subscenarios on their own repository can therefore expect a non-empty report of EW-classified candidates with high probability. Inspection-burden quantification. At 30 s per-candidate triage, the 464,073 predicted-EW patterns imply ∼3,867 reviewer-hours; the R1–R6 closure chain (Section 8) reduces the surviving set to 84,564 patterns (∼705 hours); ranking-by-quality-score then lets a reviewer focus on the top-K, and at K=200 inspection drops to ∼1.7 hours, a ∼2,300× reduction from the unfiltered set (∼420× from the R1–R6-filtered set). The per-candidate ranking is released so any K can be chosen to match the reviewer budget.
8 Threats to validity 8.1 Internal validity Cluster-id sequence collisions. Two semantically distinct slices may share a cluster-id sequence if the cukereuse hybrid clusterer over-merges their steps. Mughal et al. (2026) report Fleiss’ κ = 0.84 on a 1,020-pair benchmark, which bounds the within-cluster confusion rate but not the slice-level collision rate (a slice is correct only if every constituent step is correctly clustered). The mitigation is manual inspection of the top-100 ranked slices per RQ scope; the audit log is released alongside the ranking parquet. Phase 4 catches the inverse direction (semantically equivalent slices with divergent cluster-id sequences) by collapsing them into a paraphrase-equivalence cluster. Subjectivity of extraction-worthy. The Phase 6 classifier learns from human judgements that may not generalise. Fleiss’ κ (Fleiss 1971) on the 60-slice overlap subset is the inter-rater agreement floor, interpreted under the same Landis–Koch bands (Landis and Koch 1977) as cukereuse; the pair-level rubric achieved κ = 0.84, the calibration target for the slice-level extension. Detector-threshold sensitivity. The Phase-2c v3 spec-suite detector requires file-level density > 50 AND either toppattern within-file recurrence > 100 OR template-structure fraction ≥ 0.30. These thresholds were calibrated on three pilot entries (Section 7) and on visual inspection of the file-level density histogram. Table 3 reports the headline both with and without the v3 filter so a reader who disagrees with the threshold choice can recover a defensible interval; a ±50 % sensitivity sweep is in the supplementary material. Post-classifier verification filter chain (R1–R6). Manual review of the 464,073 Phase-6 EW candidates surfaced six classes of degenerate or redundant pattern that the classifier does not reject. Each class has a transparent rule, with per-pattern flag columns in the released CSV so a reader who disagrees with any filter can recover the unfiltered set. Figure 10 plots the per-rule attrition. 20
Phase-6 EW: 464,073 (100.0 %) After R1 (templated-outline): 427,603 (92.1 %) −36,470 After R2 (single-cluster rep.): 425,585 (91.7 %) −2,018 After R3 (single-scenario): 425,585 (91.7 %) −0 After R4 (overlap-dom., < 0.20): 424,662 (91.5 %) −923 After R5 (SHL needs ≥ 2 orgs): 418,717 (90.2 %) −5,945 After R6 (closed only): 84,564 (18.2 %) −334,153
Fig. 10 Six-rule verification filter funnel on the 464,073 Phase-6 EW candidates (bar width ∝ surviving count). The closure rule R6 dominates (72 % of R1–R5 survivors) by collapsing nested representations of the same reuse opportunity. Per-rule flag columns are exposed in the released CSV.
R1 drops angle-bracket templated outlines the v3 detector misses; R2 drops singlecluster repetition runs; R3 drops single-scenario patterns; R4 drops overlap-dominated patterns whose ratio of n_distinct_scenarios to support_total is below 0.20; R5 drops shared_higher_level_step candidates with fewer than two distinct upstream owners. R6 keeps only closed sequential patterns (Pei et al. 2001; Zaki 2001), dropping pattern P if a length-(L+1) super-pattern Q exists at the same support; R6 alone removes 72 % of R1–R5 survivors without losing any underlying reuse opportunity. Closure is applied after Phases 6 and 8 because the classifiers depend on L and support_total. Nested-mirror inflation of n_distinct_orgs. A handful of repositories contain bundled copies of other projects under sub-paths (e.g., 4shen_webshell/dataset/benign/Sylius/…), which inflates n_distinct_orgs for any pattern that recurs in the embedded copy. Corpus-level prevalence is unaffected (the patterns are present at the original repo either way); what is affected is the per-pattern cross-org reach in the verification report. A followup corpus pass that detects nested feature directories against a known-repos manifest is recommended. Single corpus, single labelling team. The rubric, pilot calibration, and three-author labels share the authorship that produced the corpus. The mitigation is releasing the rubric, pool, and per-author labels under Apache-2.0 so an external party can re-label either the overlap subset or the full pool.
8.2 External validity Corpus-bounded prevalence. The 339-repository / 276-upstream-owner corpus is a sample of public GitHub repositories with permissively-licensed Gherkin .feature files at corpus-construction time (Mughal et al. 2026). The RQ3 cross-organisational prevalence is therefore a function of the corpus, not of the global population of BDD-using software projects. 21
Kalliamvakou et al. (2016) catalogue the well-known biases of GitHub-mined corpora; the standard mitigation of pinned commit SHAs is applied so re-mining the same corpus produces byte-identical inputs. “Organisation” is a GitHub-namespace boundary. We use organisation as shorthand for the segment before the first underscore in repo_slug; equivalently, the top-level GitHub account-owner namespace. On GitHub that namespace may be an Organisation account or a User account, and the 276 distinct owners in our corpus are a mix of both. The RQ3 test is therefore properly cross-account-owner: whether a slice recurs across distinct top-level namespaces, regardless of whether each is a team or a single maintainer. The 276 count is an upper bound on distinct human teams (two User accounts may share a person; we do not deduplicate), so the reported RQ3 prevalence is a conservative estimate of cross-team reuse. Cucumber dialect heterogeneity. The Mughal-2024 mechanisms are Cucumber-Java-specific; portability to Behave, Godog, SpecFlow, Karate, Cucumber-Ruby, and the long-tail dialects is unverified. The strong mechanism-applicability claim is restricted to repositories with pom.xml; non-Java dialects fall back to structurally equivalent mechanisms (Behave’s environment.py before_scenario, SpecFlow’s [Scope] attributes, Karate’s Background:). Mining is dialect-agnostic; only patch generation is dialectspecific. Recurrence is necessary, not sufficient, for reuse. A pattern that recurs n times is a candidate for extraction; whether extraction should happen depends on stability, coupling, and team conventions that no static miner can assess. The mitigation is the three-author labelling gate, rather than treating recurrence prevalence as the extraction headline.
8.3 Construct validity Slice boundaries are coarse-grained. A slice is a contiguous L-step window: two slices that share an inner sub-pattern but differ in their first or last step do not share a cluster-id sequence and do not aggregate. Pilot Finding 2 shows this matters at long L. The rubric admits a labeller-notes field for sub-slice preferences; a formal Phase 2.5 sub-slice detector is future work. Behavioural equivalence is asserted, not verified. A companion CLI (cukereuse-extract) emits patches that are syntactically valid Gherkin and Cucumber-Java but cannot be verified behaviourally without compiling and running each repository’s suite under its framework runtime, which the corpus does not pin. Equivalence checks are restricted to a hand-validated subset; the rest is staff-reviewable.
22
Real-world acceptance. A slice flagged extraction-worthy by the pipeline and accepted by the authors is still a synthetic claim. Acceptance by an upstream maintainer via a real pull request is a stronger but slower-to-collect signal; we plan a follow-on study (Section 9, future work ii) that files extraction PRs against five to ten repositories and reports maintainer responses without gating the present paper on acceptance.
9 Conclusion cukereuse-subscenarios is a static, paraphrase-robust subsequence miner for BDD suites: it ranks candidates against three nested scopes (within-file, within-repo cross-file, and cross-organisational), each mapped to a concrete Mughal-2024 reuse mechanism. On the 1.1M-step cukereuse corpus the miner produces 5.4M slices collapsing to 692,020 distinct recurring patterns, of which 30,955 recur across ≥ 2 distinct upstream owners (Section 8). Recurring structure is pervasive (75.1 % within-file, 69.2 % within-repo cross-file). The cross-organisational signal is rarer (17.1 % of scenarios, 49.3 % of repositories) and dominated by the HTTP-request-response and CLI-output assertion macros that Binamungu et al. (2018b) calls infrastructural duplication. The cross-owner / crossrepo distinction matters: 51 % of the naive cross-repo signal is one upstream owner’s multi-language SDK clones, not extraction-worthy in the shared-higher-level-step sense. Within the three-paper arc, Mughal (2024) supplied the how (three reuse mechanisms implemented in Cucumber-Java), Mughal et al. (2026) the how much (step-level duplication at corpus scale), and this paper the which (per-slice extraction decision and mechanism mapping). What remains is real-world acceptance evidence: the field-study lever in future work (ii). Future work: (i) the cukereuse-extract CLI that emits per-repo diffs from the mechanism predictions; (ii) a small-n field study filing extraction PRs against five to ten upstream repos to capture the maintainer-acceptance signal Liu et al. (2024) report missing for LLM-driven refactoring; (iii) a Phase 2.5 sub-slice mining pass that prefers internally repeated short patterns to long enclosing slices; and (iv) patchgeneration for non-Cucumber-Java dialects (Behave, SpecFlow, Karate), covering the non-Java corpus tail (Farooq et al. 2023; Arredondo-Reyes et al. 2024). Statements and Declarations. Funding This research received no external funding from any agency in the public, commercial, or not-for-profit sectors. The authors are independent researchers; all compute, storage, and OpenRouter application-programming-interface (API) costs incurred during preparation of the artefacts were borne personally by the first author from personal funds. All artefacts are released under the Apache-2.0 licence for the benefit of the broader research community.
23
Competing interests The authors declare no known competing financial interests or personal relationships that could have influenced the work reported here. Ethics approval and consent to participate This study analyses publicly available source code retrieved from GitHub via its public REST API and does not involve human participants, animal subjects, or any personally identifying information. No institutional review board (IRB) approval was required. The 200-slice labelling pool was annotated by the three named authors themselves against a written rubric, with no external participants and no personal data collected; consequently, no informed-consent procedure or General Data Protection Regulation (GDPR) style data-subject documentation was required. Consent for publication All three named authors have read the final manuscript and consent to its publication. Data and code availability All artefacts needed to reproduce this paper end-to-end are released under the Apache2.0 licence at https://github.com/amughalbscs16/cukereuse_subscenarios_release: mining scripts, the 5,382,249-row slice inventory, the 692,020-row exact-subsequence ranking, slice cluster assignments, the 200-slice three-author labelled pool with the written rubric and inter-rater summaries, the XGBoost extraction-worthy and mechanism classifiers with their out-of-fold predictions, and the per-judge raw outputs of the two open-weight LLMs evaluated as judges. The upstream 1.1M-step Gherkin corpus and the cukereuse hybrid clusterer that produces the cluster identifiers underlying every slice in this work are released at https://github.com/amughalbscs16/ cukereuse-release with a versioned Zenodo archive at https://doi.org/10.5281/ zenodo.19754359. The two LLM-judge models evaluated (openai/gpt-oss-120b, inclusionai/ling-2.6-1t) are open-weight models accessed via OpenRouter; the full per-slice prompt-and-response logs are released alongside the human labels so that reviewers can audit the LLM outputs end-to-end. Author contributions A.H.M. conceived the study, designed the methodology, and built the mining and classifier pipeline (slice inventory, exact-subsequence ranking, paraphrase-robust slice clusters, XGBoost extraction-worthy and three-way mechanism classifiers, LLM-judge harness). A.H.M., N.F., and M.B. jointly drafted and applied the written rubric, independently labelled the stratified 200-slice pool (with a 60-slice three-way overlap subset for inter-annotator agreement), and adjudicated borderline cases. A.H.M. performed the statistical analyses (Fleiss’ kappa, McNemar tests, bootstrap confidence intervals, scope rollups) and prepared all figures and tables. A.H.M. wrote the original draft. N.F. and M.B. contributed to methodology refinement, validated the rubric application and labelling decisions, and reviewed and edited the manuscript. All authors reviewed and approved the final manuscript.
24
Use of generative AI in the writing process During preparation of this work, the authors used large language models (LLMs) to assist with proofreading, copy-editing, and the local phrasing of selected paragraphs only: the category of use that Springer Nature’s editorial policy classifies as AI-assisted copy editing. After using these tools the authors reviewed and edited all output and take full responsibility for the content of the publication. No AI-generated content was used to produce labels, classifier predictions, statistical analyses, or any of the empirical numbers in this paper. The LLM-judge experiments described in Section 6.8 are a separately declared methodological baseline against which the authors’ rubricbased labels and the XGBoost classifier are evaluated; they are not a source of any labels or conclusions in the paper, and the raw per-slice prompts and responses are released for reviewer audit.
References Agrawal R, Srikant R (1995) Mining sequential patterns. In: Proceedings of the 11th International Conference on Data Engineering (ICDE). IEEE, pp 3–14, https://doi. org/10.1109/ICDE.1995.380415 Alcântara F, Montandon JE (2026) GivenWhenThen: A dataset of BDD test scenarios mined from open source projects. In: Proceedings of the 23rd International Conference on Mining Software Repositories (MSR), Data and Tool Showcase Track. ACM, https://doi.org/10.1145/3793302.3793308 Arredondo-Reyes VM, Domínguez-Isidro S, Sánchez-García AJ (2024) Analysis of behavior-driven development: A thematic synthesis. Programming and Computer Software 50:701–713. https://doi.org/10.1134/S0361768824700713 Bavota G, Qusef A, Oliveto R, et al (2012) An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In: 28th IEEE International Conference on Software Maintenance (ICSM). IEEE, pp 56–65, https://doi.org/10.1109/ICSM.2012.6405253 Baxter ID, Yahin A, Moura L, et al (1998) Clone detection using abstract syntax trees. In: Proceedings of the International Conference on Software Maintenance (ICSM). IEEE, pp 368–377, https://doi.org/10.1109/ICSM.1998.738528 Bellon S, Koschke R, Antoniol G, et al (2007) Comparison and evaluation of clone detection tools. IEEE Transactions on Software Engineering 33(9):577–591. https: //doi.org/10.1109/TSE.2007.70725 Binamungu LP, Maro S (2023) Behaviour driven development: A systematic mapping study. Journal of Systems and Software 203:111749. https://doi.org/10.1016/j.jss. 2023.111749
25
Binamungu LP, Embury SM, Konstantinou N (2018a) Detecting duplicate examples in behaviour driven development specifications. In: IEEE Workshop on Validation, Analysis and Evolution of Software Tests (VST). IEEE, pp 6–10, https://doi.org/ 10.1109/VST.2018.8327150 Binamungu LP, Embury SM, Konstantinou N (2018b) Maintaining behaviour driven development specifications: Challenges and opportunities. In: IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, pp 175–184, https://doi.org/10.1109/SANER.2018.8330208 Binamungu LP, Embury SM, Konstantinou N (2020) Characterising the quality of behaviour driven development specifications. In: 21st International Conference on Agile Software Development (XP), Lecture Notes in Business Information Processing, vol 383. Springer, pp 87–102, https://doi.org/10.1007/978-3-030-49392-9_ 6 Campello RJGB, Moulavi D, Sander J (2013) Density-based clustering based on hierarchical density estimates. In: Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD). Springer, pp 160–172, https://doi.org/10.1007/ 978-3-642-37456-2_14 Chen T, Guestrin C (2016) XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, pp 785–794, https://doi.org/10.1145/2939672.2939785 Diniz JP, Figueiredo E, Maia MdA (2018) Bad smells in behavior-driven development scenarios. In: Proceedings of the Internetware Conference. ACM, https://doi.org/ 10.1145/3275219.3275227 Efron B, Tibshirani RJ (1993) An Introduction to the Bootstrap. Chapman & Hall/CRC Farooq MS, Omer U, Ramzan A, et al (2023) Behavior driven development: A systematic literature review. IEEE Access 11:88008–88024. https://doi.org/10.1109/ ACCESS.2023.3303194 Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5):378–382. https://doi.org/10.1037/h0031619 Fournier-Viger P, Lin JCW, Kiran RU, et al (2017) A survey of sequential pattern mining. Data Science and Pattern Recognition 1(1):54–77 Garousi V, Küçük B (2018) Smells in software test code: A survey of knowledge in industry and academia. Journal of Systems and Software 138:52–81. https://doi. org/10.1016/j.jss.2017.12.013
26
Gu J, Jiang X, Shi Z, et al (2024) A survey on LLM-as-a-judge. arXiv preprint, https://doi.org/10.48550/arXiv.2411.15594, URL https://arxiv.org/abs/ 2411.15594, arXiv:2411.15594 Horikawa K, Kashiwa Y, Lin B, et al (2025) How does test code differ from production code in terms of refactoring? an empirical study. In: 41st IEEE International Conference on Software Maintenance and Evolution (ICSME), New Ideas and Emerging Results Track. IEEE Irshad M, Petersen K (2020) An investigation of effort distribution among development phases in BDD. In: Proceedings of the Evaluation and Assessment in Software Engineering (EASE). ACM, pp 220–229, https://doi.org/10.1145/3383219.3383242 Irshad M, Britto R, Petersen K (2021) Adapting behavior driven development (BDD) for large-scale software systems. Journal of Systems and Software 177:110944. https: //doi.org/10.1016/j.jss.2021.110944 Irshad M, Börstler J, Petersen K (2022) Supporting refactoring of BDD specifications: An empirical study. Information and Software Technology 141:106717. https://doi. org/10.1016/j.infsof.2021.106717 Jiang L, Misherghi G, Su Z, et al (2007) DECKARD: Scalable and accurate treebased detection of code clones. In: Proceedings of the 29th International Conference on Software Engineering (ICSE). IEEE, pp 96–105, https://doi.org/10.1109/ICSE. 2007.30 Kalliamvakou E, Gousios G, Blincoe K, et al (2016) The promises and perils of mining GitHub. Empirical Software Engineering 21(5):2035–2071. https://doi.org/10.1007/ s10664-015-9393-5 Kamiya T, Kusumoto S, Inoue K (2002) CCFinder: A multilinguistic token-based code clone detection system for large scale source code. IEEE Transactions on Software Engineering 28(7):654–670. https://doi.org/10.1109/TSE.2002.1019480 Kim M, Zimmermann T, Nagappan N (2014) An empirical study of refactoring challenges and benefits at Microsoft. IEEE Transactions on Software Engineering 40(7):633–649. https://doi.org/10.1109/TSE.2014.2318734 Krinke J, Ragkhitwetsagul C (2024) How the misuse of a dataset harmed semantic clone detection. arXiv preprint, https://doi.org/10.48550/arXiv.2505.04311, URL https://arxiv.org/abs/2505.04311, arXiv:2505.04311 Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. https://doi.org/10.2307/2529310 Li Z, Lu S, Myagmar S, et al (2004) CP-Miner: A tool for finding copy-paste and related bugs in operating system code. In: Proceedings of the 6th Symposium on
27
Operating Systems Design and Implementation (OSDI). USENIX, pp 289–302 Liu B, Jiang Y, Zhang Y, et al (2024) An empirical study on the potential of LLMs in automated software refactoring. arXiv preprint, https://doi.org/10.48550/arXiv. 2411.04444, URL https://arxiv.org/abs/2411.04444, arXiv:2411.04444 Mannila H, Toivonen H, Verkamo AI (1997) Discovery of frequent episodes in event sequences. Data Mining and Knowledge Discovery 1(3):259–289. https://doi.org/ 10.1023/A:1009748302351 McInnes L, Healy J, Melville J (2018) UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint https://doi.org/10.48550/arXiv. 1802.03426, arXiv:1802.03426 Mughal AH (2024) Advancing BDD software testing: Dynamic scenario reusability and step auto-complete for Cucumber framework. arXiv preprint, https: //doi.org/10.48550/arXiv.2402.15928, URL https://arxiv.org/abs/2402.15928, arXiv:2402.15928 Mughal AH, Fatima N, Bilal M (2026) Reducing maintenance burden in behaviourdriven development: A paraphrase-robust duplicate-step detector with a 1.1M-step open benchmark. Information and Software Technology Under review; preprint and tool released at https://github.com/amughalbscs16/cukereuse-release North D (2006) Introducing BDD. Better Software Magazine URL https://dannorth. net/introducing-bdd/ Oliveira G, Marczak S (2017) On the understanding of BDD scenarios’ quality. In: IEEE 25th International Requirements Engineering Conference Workshops (REW). IEEE, pp 318–321, https://doi.org/10.1109/REW.2017.77 Oliveira G, Marczak S, Moralles C (2019) How to evaluate BDD scenarios’ quality? In: Proceedings of the 33rd Brazilian Symposium on Software Engineering (SBES). ACM, pp 481–490, https://doi.org/10.1145/3350768.3353816 Pei J, Han J, Mortazavi-Asl B, et al (2001) PrefixSpan: Mining sequential patterns efficiently by prefix-projected pattern growth. In: Proceedings of the 17th International Conference on Data Engineering (ICDE). IEEE, pp 215–224, https: //doi.org/10.1109/ICDE.2001.914830 Pereira L, Sharp H, de Souza C, et al (2018) Behavior-driven development benefits and challenges: reports from an industrial study. In: Proceedings of the 19th International Conference on Agile Software Development (XP). ACM, https: //doi.org/10.1145/3234152.3234167 Pontillo V, Palomba F, Ferrucci F (2024) Machine learning-based test smell detection. Empirical Software Engineering 29. https://doi.org/10.1007/s10664-023-10436-2
28
Rattan D, Bhatia R, Singh M (2013) Software clone detection: A systematic review. Information and Software Technology 55(7):1165–1199. https://doi.org/10.1016/j. infsof.2013.01.008 Reimers N, Gurevych I (2019) Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp 3982–3992, https://doi.org/10.18653/v1/D19-1410 Roy CK, Cordy JR, Koschke R (2009) Comparison and evaluation of code clone detection techniques and tools: A qualitative approach. Science of Computer Programming 74(7):470–495. https://doi.org/10.1016/j.scico.2009.02.007 Saini V, Farmahinifarahani F, Lu Y, et al (2018) Oreo: Detection of clones in the twilight zone. In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, pp 354–365, https://doi.org/10.1145/3236024. 3236026 Sajnani H, Saini V, Svajlenko J, et al (2016) SourcererCC: Scaling code clone detection to big code. In: Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, pp 1157–1168, https://doi.org/10.1145/2884781. 2884877 Scandaroli A, Leite R, Kiosia ASG, et al (2019) Behavior-driven development as an approach to improve software quality and communication across remote business stakeholders, developers and QA: Two case studies. In: Proceedings of the 14th IEEE International Conference on Global Software Engineering (ICGSE). IEEE, pp 95–102, https://doi.org/10.1109/ICGSE.2019.00027 Sears C, Tsilionis K, Wautelet Y (2025) Improving behavior-driven development scenarios: Empirical evaluation of a quality assessment framework. In: International Conference on Product-Focused Software Process Improvement (PROFES). Springer Soares E, Ribeiro M, Ferreira F, et al (2024) A comprehensive catalog of refactoring strategies to handle test smells in Java-based systems. Software Quality Journal 32(3). https://doi.org/10.1007/s11219-024-09663-7 Solis C, Wang X (2011) A study of the characteristics of behaviour driven development. In: 37th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA). IEEE, pp 383–387, https://doi.org/10.1109/SEAA.2011.76 Spadini D, Schvarcbacher M, Oprescu AM, et al (2022) Test smells 20 years later: Detectability, validity, and reliability. Empirical Software Engineering 27. https: //doi.org/10.1007/s10664-022-10207-5
29
Spadini D, Palomba F, Bacchelli A, et al (2024) Test code refactoring unveiled: Where and how does it affect test code quality and effectiveness? Empirical Software Engineering 29(6):149. https://doi.org/10.1007/s10664-024-10577-y Srikant R, Agrawal R (1996) Mining sequential patterns: Generalizations and performance improvements. In: Advances in Database Technology — EDBT’96, Lecture Notes in Computer Science, vol 1057. Springer, pp 1–17, https://doi.org/10.1007/ BFb0014140 Walunj V, Trabelsi A, Sallaberry C (2023) From basic approaches to novel challenges and applications in sequential pattern mining. Applied Computing and Intelligence 3(1):44–90. https://doi.org/10.3934/aci.2023004 Wautelet Y, Nassiri S, Tsilionis K (2023) A CASE tool for evaluating BDD scenarios. In: The Practice of Enterprise Modeling (PoEM). Springer, Lecture Notes in Business Information Processing, https://doi.org/10.1007/978-3-031-48583-1_11 Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: A survey. Software Testing, Verification and Reliability 22(2):67–120. https://doi. org/10.1002/stvr.430 Zaki MJ (2001) SPADE: An efficient algorithm for mining frequent sequences. Machine Learning 42(1–2):31–60. https://doi.org/10.1023/A:1007652502315 Zheng L, Chiang WL, Sheng Y, et al (2023) Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In: Advances in Neural Information Processing Systems 36 (NeurIPS) Datasets and Benchmarks Track
30