Robo-SWE: An Executable Issue-Resolution Dataset Mined from Robot Software Repositories

ROBO - SWE: An Executable Issue-Resolution

Dataset Mined from Robot Software Repositories Vahid Haratian∗† , Vahid Ansari† , Mohammadhossein Malmir‡ , Alois Knoll‡ ∗ Bilkent University, Ankara, Turkey † Evalium, Yerevan, Armenia ‡ Technical University of Munich, Munich, Germany

arXiv:2611.16116v1 [cs.SE] 11 Nov 2026

{vahid.haratian,vahid.ansari}@evalium.ai, {hossein.malmir,k}@tum.de

Abstract—Coding agents are evaluated mostly on generalpurpose software, but domain-specific systems require domainaware evaluation. Robot software is a particularly challenging case: Robot Operating System 2 (ROS 2) packages have their own build tools, simulators and learning frameworks are built on physics engines, and behavior that depends on timing and hardware is hard to test. We present ROBO - SWE, a dataset of 236 executable issue-resolution tasks mined from merged pull requests in 28 public robotics repositories in four domains: ROS 2 systems, simulation and physics engines, robotics algorithm libraries, and robot-learning frameworks. Each task contains the original issue text, a container recipe for the repository at the commit before the fix, the developers’ fix, and the developers’ tests, which fail before the fix and pass after it in repeated offline runs. The dataset also keeps a provenance ledger of all 1,567 candidate pull requests, with the stage each reached and the reason for every rejection, and, as a first use, 3,540 graded attempts by five coding-agent setups with recorded patches and 3,529 available trajectories. Across three runs per primary setup, resolve rates range from 45.8% for Claude Sonnet 5.5 to 56.5% for Claude Opus 5.5; GPT-6.1 Sol reaches 54.1% at maximum effort and 52.5% at medium effort, and GPT-6 Astra 51.4%. GLM-5.3 resolves 46.2%, and Kimi K3.1 Pro 43.9%. The tasks are validated by execution; verification of their fairness by human reviewers is future work. The dataset and public leaderboard are available online. Index Terms—dataset, mining software repositories, issue resolution, coding agents, robot software, ROS 2, simulation

I. I NTRODUCTION Robot software consists of large, interdependent stacks of reusable components [1]. A mobile manipulator built on the Robot Operating System (ROS), in its current generation ROS 2 [2], combines navigation, control, perception, hardware drivers, and middleware across C++ and Python packages built with colcon and CMake. Development depends on simulators and physics engines, while robot-learning frameworks add substantial codebases for policies trained in simulation before deployment to hardware [3], [4]. Maintaining these stacks is central to robotics engineering, yet testing them is difficult: behavior depends on timing, concurrency, and hardware, and practitioners still rely heavily on field testing [5]. These characteristics distinguish robot-software repair from repair in self-contained applications. Correctness spans packages and processes: a patch may compile yet violate a ROS message or lifecycle contract, change launch-time behavior, Vahid Haratian and Vahid Ansari contributed equally to this work.

or fail only with a particular simulator, middleware, timing, or hardware configuration. Reproducing a task therefore requires more than checking out source and running unit tests. Evaluation must preserve the domain toolchain and execution environment, account for concurrency and nondeterminism, and separate infrastructure failures from incorrect repairs; otherwise, scores may measure environment mismatch rather than an agent’s repair ability. Despite these domain constraints, coding agents are typically evaluated by whether they resolve GitHub issues in general-purpose repositories. SWE-bench [6] and its humanchecked subset [7] defined the task; later datasets extended it to more languages and automated its collection [8]–[10]. None of them targets robot software or its build systems. Roboticists, meanwhile, ask whether such agents can supply the engineered, testable modules that robot learning does not obtain from data [11], [12]. Current robotics benchmarks do not answer this for existing codebases: agents write new programs against a robot application programming interface (API) or a simulator [13]–[16], complete single functions [17], or repair a small curated defect set in one stack [18]. We present ROBO - SWE, a dataset mined from the history of robotics projects to close this gap. It is meant for software engineering researchers who study or train coding agents, and for robotics teams who want to know which parts of their stack an agent can maintain. It provides: 236 executable issue-resolution tasks from 28 repositories in four domains (Table I), among them Navigation2, ros2_control, rosbag2, MoveIt 2, Newton, MuJoCo Warp, python-control, and LeRobot. Each pairs the original issue with a pre-fix workspace, the developers’ fix, and developer tests that fail before the fix and pass after it. • The provenance of the tasks: a 2026 refresh of a census of ROS repositories on GitHub [19], and a ledger of all 1,567 candidate pull requests with the stage that each reached and the reason for every rejection. • A first use: 3,540 graded attempts by five coding-agent setups, with their patches, per-test verdicts, and trajectories. •

Dataset: https://huggingface.co/datasets/vharatian/Robo-SWE. Leaderboard: https://evalium.ai/robo-swe.

TABLE I C ANDIDATE FILTERING BY DOMAIN , FROM 1,567 DISTINCT PULL REQUESTS TO 236 EVALUATED TASKS . ROS 2

Sim.

Alg.

Learn.

All

Candidate PRs Queued for building Environment built Admitted by execution Evaluated tasks

640 325 291 124 103

709 458 104 66 63

140 93 50 34 32

78 52 44 38 38

1,567 928 489 262 236

Repositories with candidates Repositories with evaluated tasks Resolved in every baseline run∗ Resolved in no baseline run∗

39 17 29 39

11 4 31 21

11 5 14 7

2 2 9 14

63 28 83 81

∗

15 runs of the five primary setups (Section IV).

II. DATASET C ONSTRUCTION One pipeline builds all four domains with the same task definition, mining window, statement, and export format; Table I follows the candidates through it. A task is a merged pull request (PR) that closes an earlier issue and changes production code together with tests, so that the developers’ own tests can serve as the oracle. Its author must be human, and it must not be a backport. Build-queue counts include postponements; “environment built” denotes a successful recorded build. Release checks remove 26 admitted candidates independently of agent scores. Sim., Alg., and Learn. denote simulation and physics, robotics algorithms, and robot learning, respectively. A. Repositories ROS 2. The census of ROS-based systems by Malavolta et al. [19], [20] uses a 2019 snapshot. A recent survival study samples 810 ROS 2 repositories from the ROS Index [21], for a different research purpose. We re-implemented the census mining on the GitHub GraphQL API with the same thresholds, extended the keyword lists with ROS 2 terms, and added the rosdistro index and community lists (September 21–28, 2026). Of 61,878 candidate repositories, 2,114 pass the original filters and contain ROS 2 code; 2,043 of them are not in the 2019 list. Of the 197 that are active and have at least 100 merged PRs, GPT-5 mini classifies 171 as robot systems. The 80 largest with at least five human authors, none holding more than 80% of the PRs, form the mining frame. Simulation and physics, robotics algorithms, and robot learning. These three domains use a curated list of 42 repositories and keep the 25 that are active, have at least 30 merged PRs from five or more human authors in the window, merge through PRs, and carry an open-source license. B. Candidate Mining and Screening We mine a four-year window ending in October 2026: for ROS 2, a stratified sample of up to 500 PRs from the last two years of each of the 80 repositories, and then every PR in the window for the 39 repositories that yielded a candidate; for simulation, algorithms, and learning, every PR of the 25 repositories. Production and test changes are told apart by

content, not by file name, and size limits keep tasks reviewable (ROS 2: at most 25 changed files; elsewhere: at most 100 files, 10 of them production files). The 61,655 merged PRs yield 1,567 candidates. Large language model (LLM) screening assesses issue clarity and whether developer tests plausibly check the issue without requiring a fix-only API [22]. ROS 2 screening uses GPT-5 with issue text, PR metadata, paths, and test identifiers, without the patch. The other three domains use GPT-5 mini (2025-08-07) with issue evidence and developer test changes; a deterministic gate checks the recorded facts. These checks select candidates for building; execution decides admission. Final task statements use the original issue titles and bodies; 11 tasks close multiple issues. C. Environments and Execution-Based Admission Each candidate gets a container at the PR’s base commit, and its tests run without network access. ROS 2 images start from a digest-pinned ROS image (with a dated package snapshot for older commits), install dependencies with rosdep, and build the touched packages with colcon; the distribution follows the PR’s target branch or the continuous-integration configuration at the base commit. The other domains use one reviewed recipe per repository with pinned base images and package versions. Environments were built for 489 of the 928 queued candidates. The simulation, algorithm, and learning queues were not exhausted: 229 candidates (192 in simulation) remained marked postponed for disk space, including one with an earlier successful build; 220 more had no build attempt. Thus, queue size minus successful builds is not a count of technical build failures. Admission runs the tests in three states: (A) the base commit with its own tests (outside ROS 2 only); (B) the base commit with the PR’s tests, where the fail-to-pass (F2P) tests must fail; and (C) state B plus the developers’ fix, where the F2P tests must pass and no pass-to-pass (P2P) test may break. Each state is repeated two or three times, and unstable verdicts are never admitted: in ROS 2, a flaky test is dropped, and a task without a stable F2P test is rejected; elsewhere, any flaky verdict rejects the task. A test that does not compile before the fix does not count as failing. This rule alone rejects 96 ROS 2 candidates, an obstacle particular to compiled languages, where new tests often call the API that the fix introduces. Tasks are never edited to pass. In total, 262 candidates are admitted. D. Release Checks Admitted tasks are exported in the Harbor task format [23] and checked again by the harness: the developers’ fix must score 1, and an empty patch 0. Of the admitted tasks, 26 were withheld, each with a recorded reason: 6 at export (two flaky or already passing at the base, three whose issue contains the fix, and one whose tests need an API the issue never mentions); 9 after the harness check or inspection (two whose F2P tests pass without any change, four whose fix fails in the exported image, one whose environment no longer builds, and two whose hidden tests overwrite a build file that

ros2__rosbag2-1812/ ros2/rosbag2 PR #1812, merged 2024-09-23 instruction.md the issue as filed (234 words), then the workspace path and the build command `ros2 bag convert` ignoring messages with unknown types The `ros2 bag convert` fails to convert bag files with "unknown" ,→ message types. Unknown is the type which typesupport is ,→ not available during the bag conversion.

environment/ Dockerfile: digest-pinned ROS 2 image, rosdep, colcon build at the base commit solution/ the developers’ fix (11 changed lines in 2 files), hidden from the agent rosbag2_cpp/src/rosbag2_cpp/message_definitions/ ,→ local_message_definition_source.cpp @@ -179,7 +180,13 @@ std::string package = match[1]; - std::string share_dir = ament_index_cpp:: ,→ get_package_share_directory(package); + std::string share_dir; + try { + share_dir = ament_index_cpp::get_package_share_directory( ,→ package); + } catch (const ament_index_cpp::PackageNotFoundError & e) { + ROSBAG2_CPP_LOG_WARN("'%s'", e.what()); + throw DefinitionNotFoundError(definition_identifier. ,→ topic_type()); + }

tests/ hidden oracle: the PR’s test changes, 1 F2P and 7 P2P test identifiers, the grading script. F2P: test_local_message_definition_source.throw_ definition_not_found_for_unknown_msg task.toml time, processor, and memory limits; network mode; provenance

Fig. 1. A ROBO - SWE task and its files. The agent sees instruction.md and the built workspace; solution/ and tests/ stay hidden until grading.

an agent may legitimately edit); 10 ROS 2 tasks rejected by a final LLM check of the statement with GPT-5 mini (two contain the fix, six have tests beyond what the issue asks, and two do not pose a well-defined task); and one whose pinned nightly dependency was retired upstream. The original 237-task frozen release is retained; this last task is excluded uniformly from every setup, giving 236 evaluated tasks. Several ROS 2 test suites are sensitive to host load: with 14 tasks graded concurrently on 12 cores, the developers’ fix failed on 15 ROS 2 tasks, 11 of which pass on a quiet host. III. DATASET D ESCRIPTION A. Contents Table I gives the size of each domain. The ROS 2 tasks cover navigation and mobile robots (45 tasks), control and drivers (26), middleware and infrastructure (22), task planning (5), manipulation (4), and perception (1); the largest sources are Navigation2, rosbag2, and ros2_control. Outside ROS 2, the tasks come from simulators and physics engines that run on graphics processing units (GPUs), such as Newton and MuJoCo Warp, from algorithm libraries (python-control, Robotics Toolbox), and from robot-learning frameworks (mjlab, LeRobot). The fix touches C++ in 80 of the 103 ROS 2 tasks and is pure Python in every task outside ROS 2: no C++ task from the curated simulation, algorithm, or learning projects is in this evaluation. The developers’ fixes are small: a median of 30 changed lines in 2 files; 78 tasks change at most 15 lines and 24 more than 150. The PRs were merged between November 2022 and October 2026; 153 (65%) were merged in 2026, and every task records its merge date, so that users can exclude tasks that predate a model’s training cutoff.

TABLE II TASK - RESOLUTION RATES BY MODEL – AGENT SETUP AND DOMAIN (N OVEMBER 10, 2026). Setup (runs)

Total

Interval

ROS 2 Sim. Alg. Learn. Any Every

$

Opus 5.5 (3) Sol max (3) Sol medium (3) Astra (3) Sonnet 5.5 (3)

56.5 54.1 52.5 51.4 45.8

50.7–62.4 47.9–60.3 46.5–58.6 45.1–57.9 39.8–51.7

48.5 50.8 49.5 46.3 39.8

59.8 55.6 55.0 56.1 55.6

69.8 64.6 61.5 59.4 49.0

61.4 51.8 49.1 50.9 43.0

60.2 57.2 56.4 54.2 51.3

51.7 50.0 47.9 47.9 39.4

0.47 0.43 0.23 1.24 0.10

GLM-5.3 (1) 46.2 39.9–52.6 Kimi K3.1 Pro (2) 43.9 37.5–50.0

42.7 39.3

52.4 53.1 54.0 46.9

39.5 36.8

– – 47.0 40.7

– –

B. Storage and Schema Each task is a directory in the Harbor task format (Fig. 1). The working pipeline and review application store metadata in SQLite ledgers. The replication package organizes four tables, with comma-separated values (CSV) and JSON Lines exports: tasks, one record per task with repository, PR and issue links, base and fix commits, merge date, domain and subcategory, build system, test framework, patch size, F2P and P2P test identifiers, license, content hash, and the audit flag of Section VII-A; candidates, one record per candidate PR with the last stage it reached and the reason it was dropped; admission, the verdict of every test in every repetition of states A–C; and attempts, one record per agent attempt (Section IV) with setup, run, outcome, per-test verdicts, token use, time, and the paths of the agent’s patch and trajectory. Task versions and the uniform environment exclusion must be recorded with each score. C. Usage An agent is run on a task with harbor run -p <task> -a <agent> -m <model>. It works in the built workspace at the base commit with the tests withheld; the hidden tests are then copied in and executed. A task is resolved when every F2P test passes and no P2P test fails. Because the fixes are public, agents should run without web access; because of the load sensitivity above, we grade at most three ROS 2 tasks at a time on a 12-core host. IV. BASELINE E VALUATION As a first use and a reference point, we ran five agent setups (a setup fixes the model, the agent harness, and the reasoning effort) three times over all tasks, 3,540 graded attempts: Claude Opus 5.5 and Claude Sonnet 5.5 in Claude Code 2.1.302, and GPT-6.1 Sol (medium and max effort) and GPT-6 Astra in Codex 0.163.0. Two further setups in OpenCode, GLM-5.3 and Kimi K3.1 Pro, have 1 and 2 complete runs. Each attempt receives the issue text and the built workspace, has a one-hour budget, and reaches only the model’s API; the Claude Code setups also have a cost cap of $20 per attempt. Provider interruptions are excluded rather than scored as model failures; 14 capacity-interrupted primary attempts were replaced, including four with passing raw rewards. Ordinary graded failures retain their scores. Only complete runs are included in Table II.

Table II averages task-level resolve rates over complete runs. Any and Every denote tasks resolved in at least one and in every run, respectively. The cost column reports mean APIequivalent cost per graded attempt in complete runs; – denotes unavailable data. The 95% intervals use 2,000 task-bootstrap resamples (Wilson when task scores are binary). Across the 15 primary setup-runs, 83 tasks are always resolved, 81 never, and 72 vary by setup or run. For example, Opus resolves 60.2% at least once and 51.7% in all three runs. These are comparisons of model–harness setups, not isolated models; the primary intervals overlap. Cost means exclude missing records and interruptions and describe API-price equivalents rather than subscription bills. V. R ESEARCH O PPORTUNITIES Oracle adequacy. Developer tests can accept incorrect patches [24], [25] and can reject valid ones. The agent patches, per-test verdicts, and developers’ fixes allow both errors to be measured for robot software and compared with behavioral oracles that drive a simulated robot [18]. Reproducibility of robotics builds. Recorded build outcomes and environment exclusions support studies of dependency drift and load-sensitive tests. Postponements must be distinguished from technical build failures. Issue quality. Statements are unedited issues, and the audit flag marks tasks whose tests need identifiers that the issue never names; this supports work on under-specification and issue rewriting. Agent behavior. The 3,529 available trajectories from 3,540 primary attempts support future analysis of fault localization, build-system use, and domain reasoning; failure causes have not been labeled. Training, test generation, and contamination. The tasks can serve as training or held-out data [26], their F2P tests as targets for test generation [27], and their merge dates for contamination studies. Robotics. Per-subsystem results motivate testing whether unit-test verdicts predict behavior when repaired code is replayed in simulation. VI. R ELATED W ORK Issue resolution and reproducible bugs. SWE-bench [6] defines issue-to-patch tasks; Verified [7] adds human checks, Multi-SWE-bench [8] extends language coverage, and SWEbench-Live [9] and SWE-rebench [10] automate collection. Defects4J [28], BugSwarm [29], and GitBug-Java [30] package reproducible faults. ROBUST [31] provides 221 ROS bugs. ROBO - SWE combines mined issues, withheld developer tests, and executable environments across four domains. Robotics coding. Code as Policies [13], CaP-X [14], EmbodiedSWE [15], and RLE-Bench [16] evaluate newly written control or learning programs; ROSDevEval [17] evaluates function generation. RoboBugBench [18] evaluates 33 repairs in one ROS 2 navigation stack with simulation-based behavioral oracles, complementing our inherited tests across 28 repositories. HWE-Bench [32] applies mined fail-to-pass tasks to hardware description code.

VII. L IMITATIONS AND F UTURE W ORK A. Limitations Fairness. Execution establishes reproducibility, not alignment between an issue and its tests. An automated audit flags 26 tasks whose tests use a name absent from the issue; a patch that addresses the issue can fail such a test, so the resolve rates are provisional. Human verification, under a two-reviewer protocol modeled on SWE-bench Verified [7], has not started; we therefore do not call the dataset human-verified. Oracle strength. Containerized tests do not guarantee patch correctness or robot behavior [24], [25], and resolved patches have not been systematically audited. Concentration and selection. Five repositories hold 133 tasks (56%), mjlab provides 34 of the 38 robot-learning tasks, and 9 repositories contribute a single task, so domain-level results reflect few projects. The LLM screening has not been audited against human labels. B. Future Work Human verification. Our first priority is to review every task. Reviewers will check that the developers’ fix addresses the issue, that the hidden tests measure the requested behavior, and that the issue gives an agent enough information without revealing the fix. Tasks that fail will be removed or excluded from human-verified results. More repositories. We will relax repository thresholds for activity, merged PRs, and contributors, and measure the effect on task yield and review outcomes. GPU-dependent projects. We will support projects such as NVIDIA Isaac Sim with pinned GPU environments. Simulation-based oracles. We will add oracles that replay patched code in simulation; they remain subject to the sim-toreal gap [4], [33]. PRs without issues or tests. The current dataset requires both a linked issue and developer tests. For PRs that lack one of them, we will generate the missing part. A generated test must fail before the developer fix and pass after it [34], and human reviewers must confirm that the issue, implementation, and tests agree and that the issue does not reveal the fix. DATA AVAILABILITY Dataset: https://huggingface.co/datasets/vharatian/ Robo-SWE. Leaderboard: https://evalium.ai/robo-swe. The archive includes the task directories; the issue–PR pairs of accepted tasks and rejected candidates, with rejection reasons; agent execution results, patches, and trajectories; and census data. The Evalium pipeline source code is proprietary and is not part of the release. Each task keeps its repository’s license: 25 repositories use Apache-2.0, BSD, or MIT, Navigation2 adds LGPL-2.1 packages, and two use GPL-3.0 (8 tasks), one of which reserves commercial use. ACKNOWLEDGMENT This work was supported in part by Evalium, which provided computational resources and covered project-related costs.

R EFERENCES [1] S. Kolak, A. Afzal, C. Le Goues, M. Hilton, and C. S. Timperley, “It takes a village to build a robot: An empirical study of the ROS ecosystem,” in IEEE International Conference on Software Maintenance and Evolution (ICSME), 2020, pp. 430–440. [2] S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall, “Robot Operating System 2: Design, architecture, and uses in the wild,” Science Robotics, vol. 7, no. 66, p. eabm6074, 2022. [3] K. Zakka et al., “Demonstrating MuJoCo Playground,” in Robotics: Science and Systems (RSS), 2025. [4] J. Josifovski, S. Auddy, M. Malmir, J. Piater, A. Knoll, and N. NavarroGuerrero, “Continual domain randomization,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 4965– 4972. [5] A. Afzal, C. Le Goues, M. Hilton, and C. S. Timperley, “A study on challenges of testing robotic systems,” in IEEE International Conference on Software Testing, Verification and Validation (ICST), 2020, pp. 96– 107. [6] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” in International Conference on Learning Representations (ICLR), 2024. [7] N. Chowdhury et al., “Introducing SWE-bench Verified,” OpenAI, https: //openai.com/index/introducing-swe-bench-verified/, 2024. [8] D. Zan et al., “Multi-SWE-bench: A multilingual benchmark for issue resolving,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025. [9] L. Zhang et al., “SWE-bench goes live!” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025. [10] I. Badertdinov et al., “SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2025. [11] K. Goldberg, “Good old-fashioned engineering can close the 100,000year “data gap” in robotics,” Science Robotics, vol. 10, no. 105, p. eaea7390, 2025. [12] ——, “A tale of two cultures: Can agentic coding close the gap?” Plenary talk, IEEE International Conference on Robotics and Automation (ICRA), Vienna, Austria, Jun. 2026. [13] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 9493–9500. [14] L. Fu et al., “CaP-X: A framework for benchmarking and improving coding agents for robot manipulation,” in International Conference on Machine Learning (ICML), 2026, pp. 32 155–32 212. [15] Z. Shen et al., “EmbodiedSWE: Coding agents for long horizon dexterous robotics,” arXiv:2609.27308, 2026. [16] H. Ma, C. Gao, R. Qiang, B. Dai, and N. Li, “RLE-Bench: A qualifying exam for coding agents as robot learning engineers,” arXiv:2609.34210, 2026. [17] Y. Zhao, X. Mao, T. Zhang, T. Li, and Z. Xiao, “Evaluating LLMs in ROS robotic software code generation,” Empirical Software Engineering, vol. 32, no. 1, p. 19, 2027. [18] A. Tuzikov, “RoboBugBench: Benchmarking automated repair of real robot software under an audited behavioral oracle,” arXiv:2610.16369, 2026. [19] I. Malavolta, G. A. Lewis, B. Schmerl, P. Lago, and D. Garlan, “How do you architect your robots? State of the practice and guidelines for ROSbased systems,” in IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2020, pp. 31–40. [20] ——, “Mining guidelines for architecting robotics software,” Journal of Systems and Software, vol. 178, p. 110969, 2021. [21] J. Freitas, E. Phifer, and F. Fronchetti, “The walking packages: A survival analysis of ROS repositories,” in International Workshop on Robotics Software Engineering (RoSE), CEUR Workshop Proceedings, vol. 4243, 2026. [22] R. Aleithan, H. Xue, M. M. Mohajer, E. Nnorom, G. Uddin, and S. Wang, “SWE-Bench+: Enhanced coding benchmark for LLMs,” arXiv:2410.06992, 2024.

[23] M. A. Merrill et al., “Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces,” in International Conference on Learning Representations (ICLR), 2026. [24] B. Yu, Y. Zhu, P. He, and D. Kang, “UTBoost: Rigorous evaluation of coding agents on SWE-Bench,” in Annual Meeting of the Association for Computational Linguistics (ACL), 2025, pp. 3762–3774. [25] Y. Wang, M. Pradel, and Z. Liu, “Are “solved issues” in SWE-bench really solved correctly? An empirical study,” in IEEE/ACM International Conference on Software Engineering (ICSE), 2026, pp. 169–181. [26] J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang, “Training software engineering agents and verifiers with SWE-Gym,” in International Conference on Machine Learning (ICML), 2025, pp. 47 717–47 737. [27] N. Mündler, M. N. Müller, J. He, and M. Vechev, “SWT-Bench: Testing and validating real-world bug-fixes with code agents,” in Advances in Neural Information Processing Systems (NeurIPS), 2024, pp. 81 857– 81 887. [28] R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existing faults to enable controlled testing studies for Java programs,” in International Symposium on Software Testing and Analysis (ISSTA), 2014, pp. 437–440. [29] D. A. Tomassi et al., “BugSwarm: Mining and continuously growing a dataset of reproducible failures and fixes,” in IEEE/ACM International Conference on Software Engineering (ICSE), 2019, pp. 339–349. [30] A. Silva, N. Saavedra, and M. Monperrus, “GitBug-Java: A reproducible benchmark of recent Java bugs,” in International Conference on Mining Software Repositories (MSR), 2024, pp. 118–122. [31] C. S. Timperley, G. van der Hoorn, A. Santos, H. Deshpande, and A. Wasowski, ˛ “ROBUST: 221 bugs in the Robot Operating System,” Empirical Software Engineering, vol. 29, no. 3, p. 57, 2024. [32] F. Cui, H. Hou, Z. Luo, C. Yin, and Y. Liang, “HWE-Bench: Benchmarking LLM agents on real-world hardware bug repair tasks,” arXiv:2604.14709, 2026. [33] M. Malmir, J. Josifovski, N. Klarmann, and A. Knoll, “DiAReL: Reinforcement learning with disturbance awareness for robust Sim2Real policy transfer in robot control,” IEEE Transactions on Control Systems Technology, vol. 34, no. 2, pp. 1037–1043, 2026. [34] V. Haratian, A. Akar, B. Çakar, and E. Tüzün, “PR-aware automated unit test generation: Challenges and opportunities,” in IEEE Annual Computers, Software, and Applications Conference (COMPSAC), 2026, pp. 346–355.

Related documents

• Robo-SWE: An Executable Issue-Resolution Dataset Mined from Robot Software Repositories#1659697
• daVinci-Env: Open SWE Environment Synthesis at Scale#1447108
• OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration#1345191
• OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration#1336059
• Educational robotics beyond ROS: a custom library for direct transfer of Webots controllers in the Turtlebot3#1623928