OpenRE-Bench: A Benchmark Harness for Multi-Agent Requirements Engineering with Mined GitHub Issues

OpenRE-Bench: A Benchmark Harness for Multi-Agent Requirements Engineering with Mined GitHub Issues Haowei Cheng∗ , Milhan Kim∗ , Bowen Jiang† , Weixing Zhang† , Yuhong Fu‡ , Zhixu Liu∗ , Naoyasu Ubayashi∗ , Hironori Washizaki∗ ∗ Waseda University, Tokyo, Japan † Karlsruhe Institute of Technology (KIT), Germany

arXiv:2611.13925v1 [cs.SE] 10 Nov 2026

‡ Adelaide University, Adelaide, Australia

Abstract—OpenRE-Bench is an open-source benchmarking infrastructure for large language model (LLM) based multi-agent requirements-engineering (RE) frameworks. Such frameworks are increasingly common, but they use different agent roles, negotiation policies, and output shapes, and are typically evaluated on their own runtimes, metrics, and small sets of curated system descriptions, which makes fair, reproducible comparison difficult. OpenRE-Bench addresses this with four reusable pieces: a shared run harness with one input, phase-artifact, and metric contract; an adapter mechanism through which a framework joins by implementing a single adapter, leaving cases, metrics, and other adapters untouched; a reproducible GitHub issuemining pipeline that grows the corpus from real feature requests; and a versioned case suite. As a demonstration, we mine 3,000 fetched issues across five projects (three languages, five domains) into five cases, package them with five hand-built cases as the compact v1.0 suite, and run three heterogeneous frameworks (MARE and iReDev as paper-faithful reimplementations, plus QUARE) under one protocol. OpenRE-Bench originated as the evaluation back-end of our QUARE study and has since been reused by two further studies of our group. It is released with a pinned environment, a Docker image, frozen API snapshots, a 651-test suite, and the full pool of 1,396 filtered issues behind the mined cases. Index Terms—requirements engineering, multi-agent systems, LLM, benchmark, mining software repositories, reproducibility

I. I NTRODUCTION LLM-based multi-agent frameworks for requirements engineering (RE) are becoming common. In these systems, rolespecialized agents elicit, model, negotiate, and verify requirements. Recent examples such as MARE [2], iReDev [3], and QUARE [1] differ in agent roles, negotiation protocols, and produced artifacts, which makes a basic question hard to answer: under identical conditions, how do these frameworks actually compare? Answering it is hard for two reasons. First, each framework reports results from its own pipeline, cases, and metrics. MARE scores generated requirements models with precision/recall against modeling baselines and rates its specifications by human judgment [2]; iReDev evaluates its artifacts on ten systems from a public requirements-development dataset with its own metrics [3]. In our QUARE study [1] we reimplemented both from their papers to run them side by

side. Second, the evaluation inputs are small, curated sets of system descriptions, such as the ATM and library systems drawn from prior literature [2]; they are not mined from software repositories with recorded provenance, and they do not grow as new projects appear. Results on such inputs may not reflect how frameworks handle requirements as they arise in active projects, which limits external validity; mining real issue trackers offers a natural way to close this gap. We address both obstacles with OpenRE-Bench, a reusable benchmarking infrastructure whose contributions are: 1) A shared run harness and contract that executes any conforming multi-agent RE framework behind one input / phase-artifact / run-record boundary and scores it with a common metric suite, so heterogeneous frameworks are run and compared under identical conditions (§II). 2) An adapter extension mechanism: a new framework joins the comparison by implementing a single adapter against the contract, with no change to the cases, the metrics, or the other adapters (§II). 3) A reproducible GitHub issue-mining pipeline that constructs benchmark cases from real feature-request issues with a fixed seed, frozen API snapshots, and perstage funnel logging, so the corpus grows without handcrafting (§III). 4) A versioned, compact case suite (v1.0: five mined cases spanning three languages and five domains, plus five hand-built cases); the contribution is the mining-andadapter extension mechanism, not raw corpus size (§IV). 5) Reference runs and reuse evidence: measured runs of the three bundled frameworks demonstrating the harness end to end (§V), and reuse across our group’s studies, including an adapter that reuses the originating pipeline while replacing its negotiation stage (see “Relation to our prior work” below). OpenRE-Bench is available for review at https://github. com/haowei614/OpenRE-Bench-Artifact/tree/msr2027-v1.0, and this exact release is permanently archived on Zenodo under DOI 10.5281/zenodo.23875694.

(a) Mining ingestion pipeline (§III) 1 Verify projects 5 Convert to CaseInput

2 Fetch issues (frozen snapshots) 6 Validate (runtime schema)

3 Filter (staged funnel) 7 Statistics (per-stage counts)

4 Aggregate (densest quarter) Mined cases + issue pool

fixed seed 42 · offline replay (--skip-fetch) · funnel logged per stage

(b) Run harness and shared contract (§II) Adapters 5 hand-built cases MARE CaseInput case_name case_description requirement

Shared Phase 1–4 artifacts P1 initial models

iReDev

P2 negotiation trace

QUARE

P3 integrated model

new adapter

P4 verification

Metrics elements, MDC, Sem. F1, ISO ver., time Run record provenance, setting, model, seed, fallback

Fig. 1. OpenRE-Bench architecture. (a) Seven-step pipeline turning GitHub issues into validated CaseInput cases. (b) Any registered adapter consumes mined or hand-built cases and emits shared Phase 1–4 artifacts, which the harness scores and records.

a) Relation to our prior work: The run harness and metric suite began as the evaluation back-end of QUARE [1], our multi-agent RE framework. This paper is the first to describe OpenRE-Bench as a standalone, reusable tool, and it contributes components that were not part of QUARE and are previously unpublished: the GitHub issue-mining pipeline, the five mined cases and the released pool of 1,396 filtered issues, the filter audit, and the reference runs on the mined cases. The infrastructure has since been reused across our group: ConfigRE [4] reused the harness for process-level agentconfiguration experiments, and ArgRE [5] joined through the adapter. ArgRE adapts QUARE’s roles and Phases 1, 3, 4, and 5 and replaces Phase 2 (QUARE’s Phase 5 only generates final documentation, emitted as optional artifacts outside the scored Phase 1–4 contract); its evaluation compares it with MARE, iReDev, and an ablation of ArgRE itself. In both cases the harness and the other adapters were used unchanged. b) Related work: Three lines of work are related. (i) LLM/agent benchmarks for software engineering such as SWE-bench [8] mine real GitHub issues but target code patches, not requirements artifacts. (ii) RE datasets such as PROMISE NFR [9] and PURE [10] provide requirements text for classification or extraction, but are static corpora, not an executable harness. (iii) Issue-mining infrastructure and tooling in MSR, such as GHTorrent [11] (a continuous GitHub mirror and tool suite) and Ticket Tagger [12] (a GitHub app for issue-type labeling), support repository mining or issue classification but are not executable framework comparisons. Prior multi-agent RE frameworks are each evaluated on their own runtime and mostly curated specifications (MARE [2], iReDev [3]; MARE also uses a PURE-derived public dataset), so their numbers are not comparable. OpenRE-Bench differs by combining an executable, framework-agnostic harness with a repository-mining pipeline that builds the cases it runs. II. O PEN RE-B ENCH OVERVIEW OpenRE-Bench treats each RE framework as an executable adapter behind a shared runtime boundary (Fig. 1b): it standardizes the execution and evaluation interface (input, Phase 1–4 artifacts, metrics, provenance), not the frameworks’ internal architectures or native representations. a) Shared input contract: Every adapter consumes the same CaseInput (case_name, case_description, requirement). Cases are discovered by convention

(*_input.json), so adding a hand-built or mined case requires no code change. b) Adapters: The three bundled adapters use different role sets and negotiation policies: MARE has five taskspecialized agents (Stakeholders, Collector, Modeler, Checker, Documenter), iReDev six knowledge-driven agents (Interviewer, EndUser, Deployer, Analyst, Archivist, Reviewer), and QUARE five quality-axis agents (Safety, Efficiency, Green, Trustworthiness, Responsibility) plus an Orchestrator for dialectical negotiation. All emit the same shared Phase 1–4 artifacts and a common run-record capturing provenance and comparability metadata. c) Shared representation and phases: Outputs use a KAOS-inspired [6] shared encoding (Goal, Softgoal, Obstacle, Agent, plus Task and Resource from i∗ -style modeling), and all adapters emit Phase 1 (initial models), Phase 2 (negotiation/debate trace), Phase 3 (integrated model), and Phase 4 (verification), so heterogeneous outputs become comparable. d) Settings ladder: A configuration ladder runs from a deterministic, key-free single_agent baseline up to negotiation_integration_verification, so reviewers run a cheap offline smoke check or a full LLM experiment from the same entry point. e) Metrics: Computed identically for every adapter: requirement and integrated-model element counts (a descriptive size of the shared KAOS encoding); MDC, the mean distance of Phase 3 elements to their centroid in a five-dimensional quality space (a dispersion measure, where higher means more spread-out elaboration); BERTScore [7] F1 of Phase 3 versus that run’s own Phase 1 as within-run semantic preservation; an ISO/IEC/IEEE 29148 verifiability score on a 1–5 scale, computed deterministically from structural-compliance signals (phase-level deterministic validity and compliance coverage) rather than by an LLM judge, so it is reproducible from the artifacts; and wall-clock time. A two-annotator study of verifiability ships in the artifact (quadratic-weighted κ = 0.76). f) Evaluation protocol: Given a CaseInput, a framework emits Phase 1–4 artifacts that the harness scores under a fixed setting, model, temperature, and seed. Table II (§V) reports reference runs under this protocol; they demonstrate the tool and do not rank the frameworks (§VI). g) Versioning and reproducibility: Each case suite is a frozen, citable, DOI-versioned snapshot (v1.0: five mined plus five hand-built cases), so cross-paper results refer to the same version while the pipeline extends future ones. The

TABLE I M INING FUNNEL ( ISSUES SURVIVING EACH STAGE ; R AW: 600 MOST- RECENT LABEL - MATCHING ISSUES PER PROJECT ) AND RESULTING CASES ( WINDOW, # ISSUES , requirement WORDS ; CAP 120/ ISSUE ). L ABEL AND SIGNAL STAGES ARE NO - OPS HERE AND OMITTED : THE FETCH QUERIES BY LABEL , AND THE SIGNAL GATE IS OFF IN V 1.0. Project

Lang.

Domain

Raw

State

Length

Bug

Thread cap

Dedup

Window

#Iss.

Words

scikit-learn/scikit-learn huggingface/transformers microsoft/vscode elastic/elasticsearch pandas-dev/pandas

Python Python TS Java Python

ML Library ML / NLP Editor / DevTool Database / Search Data Processing

600 600 600 600 600

425 423 399 203 446

344 380 304 164 442

267 367 258 150 370

261 363 258 150 370

261 358 258 150 369

2021Q4 2024Q3 2026Q3 2026Q1 2023Q3

4 4 3 4 3

438 420 372 429 368

3,000

1,896

1,634

1,412

1,402

1,396

5 cases

18

Total

environment pins Python 3.13 with a uv lockfile and a Docker image, stores per-run provenance, and is guarded by a 651test suite. Adding a framework (§V) leaves cases, metrics, and other adapters untouched. III. M INING I NGESTION P IPELINE The ingestion module converts real GitHub issues into harness cases. It is organized as seven independently runnable, configurable steps (Fig. 1a) (project verification, issue fetch, filtering, aggregation, schema conversion, case validation, and statistics) so that each step’s effect is auditable and reproducible. The filtering step itself applies a staged funnel, whose per-stage survivor counts are reported in Table I (§IV). a) Project selection: Candidate projects must pass explicit thresholds on stars, recent activity, and open/closed issue volume, and are chosen for language/domain diversity. A verification step queries each project’s issue endpoint and rejects unminable projects (e.g. one routing issues to an external tracker, with GitHub issues disabled) rather than silently yielding an empty corpus. The final five projects (scikit-learn, transformers, vscode, elasticsearch, pandas) span three languages (Python, TypeScript, Java) and five domains. b) Issue filtering: Each project’s issues pass through a staged funnel: (1) label allow-list, (2) state/reactions gate, (3) minimum body length, (4) bug exclusion, (5) optional requirement-signal keyword gate (default off in v1.0; see audit below), (6) a mega-thread comment cap, and (7) TF-IDF cosine deduplication. A reproducibility subtlety we document is that the GitHub labels query parameter uses conjunctive semantics; to obtain the intended union we query each label separately and merge by issue number. c) Aggregation: Surviving issues are clustered by calendar-quarter time window; the pipeline selects the densest window and the top issues by community reactions, applying a per-issue word cap so that a mined case matches the textual scale of the hand-built cases. d) Conversion and validation: Each aggregated case is emitted as a CaseInput JSON and validated with the same schema model the harness loads at runtime. Any case that validates is therefore runnable; source issue URLs are retained in a metadata sidecar for traceability.

e) Reproducibility guarantees: All sampling and clustering use a fixed seed (42); raw API responses are committed as timestamped snapshots, so the pipeline re-derives the same cases offline (run_all.py --skip-fetch); the API token is read from the environment or a git-ignored file, never committed; every stage logs input/output counts. IV. M INED DATASET AND S TATISTICS Table I reports the full mining funnel: from 3,000 fetched issues the pipeline retains 18 across five cases, with the state/reactions gate removing the most. The last three columns of Table I describe the resulting cases. Mined cases (368–438 words) are modestly larger than the five hand-built cases (58–337 words) but of the same order, and load under the identical CaseInput schema. The reduction from 1,396 survivors to 18 issues is aggregation, not further filtering (the densest-window selection above). We release the full 1,396-issue pool as an issue-level dataset (repository, number, URL, title, labels, timestamps, reactions, per-stage markers, and a flag for the 18 aggregated issues). Because the 18 aggregated issues were selected by community reactions, they cannot by themselves validate the filter. We therefore audited an earlier configuration that enabled the requirement-signal keyword gate: a seeded sampler (seed 42) drew 50 issues from the 958 survivors of that gate and 30 from issues the gate dropped, with a one-page annotation guideline and two independent raters (κ = 0.65, 8 disagreements, later majority-adjudicated). On the adjudicated retained sample, precision is 84.0% (Wilson 95% CI [71.5, 91.7]%; n = 50). On the adjudicated dropped sample we report the false-omission rate: the share of dropped issues that are still genuine requirements (i.e., FN/(FN + TN); not the miss rate FN/(FN + TP)), which is 90.0% ([74.4, 96.5]%; n = 30). Consensus-only rates (disagreements excluded) are 82.6%/92.3%. The gate therefore kept a moderately pure set but discarded mostly real requirements, so it lacked discriminative power. v1.0 consequently disables the gate by default while retaining it as a config flag for ablation; the released pool has size 1,396 = 958+438. Reusing the same labels as a twostratum estimate of pool precision (N1 = 958 signal-matching, N2 = 438 signal-absent; P = (N1 /N )p1 + (N2 /N )p2 ) yields 85.9% (normal-approx 95% CI [78.1, 93.6]%). All audit statistics are recomputable via scripts/audit_stats.py.

TABLE II R EFERENCE RUNS ON THE MINED CASES (claude-haiku-4-5, negotiation_integration_verification): MEAN ± SD, 5 CASES × 3 SEEDS (n=15). M ETRICS AS IN § II- E . Adapter

P3 elems

MDC

Sem. F1

ISO ver.

Time (s)

MARE iReDev QUARE

24.9 ± 0.3 30.0 ± 0.0 35.0 ± 0.0

0.793 ± 0.052 0.743 ± 0.036 0.675 ± 0.002

0.924 ± 0.011 0.943 ± 0.004 0.951 ± 0.003

3.02 ± 0.04 3.03 ± 0.07 4.82 ± 0.17

37.2 ± 2.8 248.5 ± 14.4 109.7 ± 6.0

V. D EMONSTRATION AND R EFERENCE RUNS a) Using and extending OpenRE-Bench: The core workflows are one command each and run in seconds with no API key or cost. uv sync --all-groups # pinned Python 3.13 env # (a) Run a case offline, no key (smoke test): uv run openre_bench --run-case --system quare \ --case-input data/case_studies/vscode_2026q3_input .json \ --artifacts-dir out --setting single_agent # (b) Replay mining offline (seed 42): uv run python ingestion/run_all.py --skip-fetch # (c) Recompute the reference table from runs: uv run python scripts/recompute_reference_table.py

A new framework joins by implementing one adapter module that exposes three callables against the shared contract: build_phase1 (Phase 1 elements, runtime semantics, and run metadata), build_negotiation_history (the Phase 2 trace), and build_optional_artifacts, plus one registry line. No case, metric, or other adapter changes are required. b) Reference runs: To demonstrate the harness end to end, we ran every mined case through all three adapters with a real LLM (claude-haiku-4-5, temperature 0.7, round cap 3, seeds 101/202/303, four settings). The harness flags any run that falls back to the deterministic scaffold as fallbacktainted; all reported runs are LLM-backed and pass the comparison validator. Table II reports the negotiation-integrationverification setting as mean ± SD over five cases and three seeds. For sklearn_2021q4 and pandas_2023q3 the runs use the v1.0 (signal-off) case texts; for the other three cases the v1.0 texts are byte-identical to those of the earlier runs, which are reused unchanged; all values are recomputable (step (c)). The runs show that heterogeneous adapters execute under one protocol, emit contract-valid artifacts, and yield seed-stable, recomputable metric profiles. Element counts are largely fixed by each adapter’s deterministic Phase 1 KAOS scaffold (MARE 25, iReDev 30, QUARE 35 slots; observed P3 means 24.9 ± 0.3 / 30.0 ± 0.0 / 35.0 ± 0.0), so the LLM mainly fills slot content (semantic preservation and structural verifiability) rather than slot count. On the five hand-built cases the same protocol reproduces the same metric profile (experiment_outputs/handbuilt-claude/), which suggests the pattern is not tied only to the mined corpus. c) Cost: Reproduction is inexpensive: the singleseed matrix (60 runs, 752 LLM calls) costs $3.59 on claude-haiku-4-5 and the full three-seed matrix (180 runs, 2,266 calls) $10.96 (from the packaged

usage_report.json); the key-free setting reproduces the pipeline offline at zero cost. VI. L IMITATIONS AND T HREATS TO VALIDITY Adapter-induced and representation-induced bias. OpenREBench keeps the evaluation interface consistent, but does not eliminate all representation-induced biases. A framework is comparable only if its output maps to the shared Phase 1– 4 / KAOS contract; that mapping can mask native behavior, and the bundled adapters’ deterministic Phase 1 scaffolds fix element counts before the LLM runs. Content-facing scores (semantic preservation vs. a run’s own Phase 1; structural ISO verifiability) remain useful as protocol diagnostics, yet must not be read as scaffold-independent rankings. MARE and iReDev are reimplemented from their papers, so absolute fidelity to the originals is a remaining threat. Training-data contamination. For claude-haiku-4-5 (training cutoff July 2025), three mined windows predate the cutoff and two postdate it; re-mining a post-cutoff window yields a refreshable, contamination-free suite on demand. Aggregation and sampling. Calendar-quarter grouping can mix unrelated topics; the fetch is capped at 600 recent label-matching issues per project; the signal gate is off by default after the §IV audit; fixed seeds, model identifiers, and temperatures aid comparability, but exact LLM outputs are not guaranteed to reproduce. Suite size. v1.0 is ten cases; the frozen release is kept separate from the extensible pipeline. a) Future extensions: Planned extensions include other issue trackers (Jira, GitLab) behind the same fetch/filter interface; topic-cluster aggregation as an alternative to time windows; more framework adapters via the one-module contract; and periodic post-cutoff releases to keep a contamination-free suite current. VII. AVAILABILITY AND C ONCLUSION OpenRE-Bench releases its code under AGPL-3.0 and its curated data under CC BY 4.0, with a pinned environment, Docker image, frozen snapshots, and citation metadata. Embedded GitHub issue text stays its authors’ copyright, redistributed for research use with source URLs retained and @-mentions de-identified. New frameworks can be added as adapters and new cases mined from further projects, so the suite can grow with community contributions.

R EFERENCES [1] H. Cheng, M. Kim, F. Khomh, T. Racharak, N. Yoshioka, N. Ubayashi, and H. Washizaki, “QUARE: Quality-aware requirements analysis through multi-agent dialectical negotiation,” in Proc. IEEE 34th Int. Requirements Engineering Conf. (RE), 2026, pp. 1–12, doi: 10.1109/RE68928.2026.00025. [2] D. Jin, Z. Jin, X. Chen, and C. Wang, “MARE: Multi-agents collaboration framework for requirements engineering,” arXiv:2405.03256, 2024. [3] D. Jin, W. Sun, J. Huang, P. Liang, J. Xuan, Y. Liu, and Z. Jin, “iReDev: A knowledge-driven multi-agent framework for intelligent requirements development,” arXiv:2507.13081, 2025. [4] H. Cheng, M. Kim, B. Jiang, W. Zhang, Y. Fu, D. T. V. Truong, T. P. T. Huyen, F. Khomh, N. Yoshioka, N. Ubayashi, and H. Washizaki, “Does agent configuration matter? An empirical study of process-level quality adaptation in multi-agent requirements engineering,” in Proc. Int. Conf. Product-Focused Software Process Improvement (PROFES), 2026, to appear. [5] H. Cheng, M. Kim, C. Liu, T. Racharak, T. V. T. Duy, P. T. H. Thanh, J. Li, N. Ubayashi, and H. Washizaki, “ArgRE: Formal argumentation for conflict resolution in multi-agent requirements negotiation,” IEEE Access, vol. 14, 2026, doi: 10.1109/ACCESS.2026.3713484. [6] A. van Lamsweerde, “Goal-oriented requirements engineering: A guided tour,” in Proc. 5th IEEE Int. Symp. Requirements Engineering (RE’01), Toronto, Canada, 2001, pp. 249–262. [7] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating text generation with BERT,” in Proc. Int. Conf. Learning Representations (ICLR), 2020. [8] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” in Proc. Int. Conf. Learning Representations (ICLR), 2024. [9] J. Cleland-Huang, R. Settimi, X. Zou, and P. Solc, “The detection and classification of non-functional requirements with application to early aspects,” in Proc. 14th IEEE Int. Requirements Engineering Conf. (RE), 2006, pp. 39–48, doi: 10.1109/RE.2006.65. [10] A. Ferrari, G. O. Spagnolo, and S. Gnesi, “PURE: A dataset of public requirements documents,” in Proc. 25th IEEE Int. Requirements Engineering Conf. (RE), 2017, pp. 502–505, doi: 10.1109/RE.2017.29. [11] G. Gousios, “The GHTorrent dataset and tool suite,” in Proc. 10th Working Conf. Mining Software Repositories (MSR), 2013, pp. 233–236, doi: 10.1109/MSR.2013.6624034. [12] R. Kallis, A. Di Sorbo, G. Canfora, and S. Panichella, “Predicting issue types on GitHub,” Science of Computer Programming, vol. 205, art. 102598, 2021, doi: 10.1016/j.scico.2020.102598.

Related documents

• Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives#822166
• Agent4Spec: An LLM-based Agentic System to Optimize API Specifications through Automated Validation and Market Analysis#1297625
• Agent4Spec: An LLM-based Agentic System to Optimize API Specifications through Automated Validation and Market Analysis#1291872
• OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration#1345191
• OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration#1336059