Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages Vladimir Dobrovolskii1 , Ibragim Badertdinov1 , Alexander Golubev1* 1 *

Nebius

Correspondence to: alex_golubev@nebius.com

arXiv:2611.10304v1 [cs.SE] 6 Nov 2026

Abstract Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its construction uses a solver–judge–repairer pipeline: a solver attempts each candidate task, and a judge reviews the task together with the solver’s trajectory and test results to guide revisions to tests and task descriptions. These revisions aim to correct tests that reject valid alternative implementations or accept incorrect solutions while retaining challenging tasks and preserving their intended requirements. We evaluate nine model configurations, including frontier models, and analyze task-solving performance, consistency across repeated attempts, and resource usage. We release all benchmark instances, executable environments, complete construction and evaluation tooling, and agent trajectories, together with an additional set of easier tasks to support evaluation of smaller models.

1

Introduction

Repository-level benchmarks evaluate whether coding agents can resolve real software issues in existing codebases (Jimenez et al., 2023). SWE-bench (Jimenez et al., 2023) has introduced a pipeline for task collection for Python, and later Multi-SWE-bench (Zan et al., 2025), SWE-PolyBench (Rashid et al., 2025), and SWE-Bench Pro (Deng et al., 2025) have broadened this setting beyond it. However, many programming language ecosystems remain sparsely represented in dedicated issue-resolution evaluations. This limits the evidence available about how well agents work with different libraries, toolchains, and development practices. Extending this coverage requires executable tasks that are both challenging and reliably evaluated. Such tasks are commonly built from historical issues and pull requests (PRs), whose original fixes serve as reference patches. However, checking that a reference patch resolves test failures without regressions does not establish evaluation reliability (Yu et al., 2026). PR-specific tests may reject valid alternative implementations, and descriptions may omit requirements that agents cannot infer from the available context (Chowdhury et al., 2024), while insufficient tests may accept incorrect solutions (Yu et al., 2025). Filtering can discard repairable tasks and bias benchmark composition: the SWE-bench Multilingual release notes that requiring clear descriptions and unambiguous tests favors small code changes (Khandpur et al., 2025). Recovering such candidates is especially important in underrepresented ecosystems with limited task pools. Building on prior work on test strengthening (Yu et al., 2026), specification repair (Wang et al., 2025), and agent-informed review (Desai et al., 2026), we jointly refine descriptions and tests to retain challenging tasks. We introduce Longtail-SWE, a benchmark of 202 tasks derived from real pull requests across 13 programming languages: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. These languages span diverse programming paradigms and application domains, providing a dedicated setting for

examining agent performance beyond the ecosystems emphasized by existing issue-resolution benchmarks. We construct Longtail-SWE using a solver–judge–repairer pipeline. A solver attempts each task; a judge compares its trajectory and test results with the task description, reference patch, and tests to distinguish solver errors from task defects. The judge’s findings guide a repair agent in revising tests and appending clarifications to the description. These revisions aim to reject incorrect solutions without excluding valid alternatives or prescribing implementation details, while preserving the task’s intended engineering requirements. Our contributions are threefold: • Task recovery and refinement. Our central contribution is a solver–judge–repairer pipeline that coordinates revisions to task descriptions and tests, allowing scarce repository-level tasks with repairable defects to be retained rather than discarded. • Benchmark and companion set. Applying this method yields Longtail-SWE, a benchmark of 202 challenging tasks across 13 underrepresented programming languages, together with a companion set of easier tasks. • Empirical analysis. We quantify how task recovery and refinement improve executable candidate yield and final task retention. We evaluate nine model configurations, comparing task-solving performance across languages, consistency across repeated attempts, and resource usage.

2

Related Work

Repository-level and multilingual benchmarks. SWE-bench (Jimenez et al., 2023) established repository-level issue-resolution evaluation in Python, and Multi-SWE-bench (Zan et al., 2025), SWEbench Multilingual (Khandpur et al., 2025), and SWE-PolyBench (Rashid et al., 2025) extended it to other languages. Specialized coverage includes Kotlin repositories in the Kotlin Benchmark (JetBrains, 2026) and Kotlin/Dart mobile applications in MobileDev-Bench (Fakorede et al., 2026). Longtail-SWE complements these efforts with tasks from underrepresented language ecosystems. Challenging software engineering tasks. SWE-Bench Pro (Deng et al., 2025) targets demanding issue resolution, while SWE-EVO (Le et al., 2025), SWE-Bench ProMax (Shi et al., 2026), and SWE Refactor Bench (Hong et al., 2026) address software evolution, multilingual refactoring, and stack migration, respectively. DeepSWE (Huang et al., 2026), FrontierSWE (Kondra et al., 2026), and SWE-Marathon (Desai et al., 2026) evaluate long-horizon engineering or research tasks. ProgramBench (Yang et al., 2026) evaluates program reconstruction with behavioral tests that permit alternative implementations. Automated task collection and environment construction. SWE-rebench (Badertdinov et al., 2025) automates task collection and validation from real pull requests. SWE-rebench V2 (Badertdinov et al., 2026) scales collection to 20 languages and adds diagnostic annotations for task quality. SWE-Bench++ (Wang et al., 2025) combines environment construction with test extraction and quality checks. SWE-Factory (Guo et al., 2025) uses agents for environment setup, while SWE-Universe (Chen et al., 2026) validates verifiers on buggy and fixed states and screens for evaluation shortcuts. R2E-Gym (Jain et al., 2025) generates tests and descriptions from commits; SWE-smith (Yang et al., 2025) synthesizes bugs while reusing repository environments and tests. Terminal-Universe (Wu et al., 2026) reconstructs workspaces from agent trajectories to synthesize tasks. Longtail-SWE complements automated task collection with coordinated refinement of descriptions and tests, aiming to retain challenging candidates from scarce task pools in underrepresented language ecosystems. Task auditing and specification alignment. SWE-bench Verified (Chowdhury et al., 2024) uses expert review to exclude tasks with underspecified issues or unsuitable tests. SPICE (Oliva et al., 2025) automates annotations of issue clarity, test adequacy, and expected effort; PAIChecker (Wang et al., 2026c) detects issue–PR mismatches. Auto Benchmark Audit (Wang et al., 2026b) investigates defects in specifications,

2

environments, and grading, including evidence from agent trajectories. SWE-Bench++ supplements underspecified descriptions with interface information from reference patches and filters instances whose failed solver runs reveal task defects. SWE-Marathon (Desai et al., 2026) uses pilot trajectories to distinguish agent failures from task defects, revising and revalidating affected candidates. The Verification Horizon (Wang et al., 2026a) examines instruction clarity and alignment between instructions and tests as requirements for faithful reinforcement-learning rewards. Test adequacy and repair. Test-suite overfitting is a longstanding concern in automated program repair (Qi et al., 2015). EvalPlus (Liu et al., 2023) augments functional tests to expose errors in generated programs. For generated postconditions, nl2postcond (Endres et al., 2024) evaluates both test-set correctness on a reference implementation (accept@k) and bug completeness against incorrect mutants, reflecting the goals of accepting valid solutions and rejecting invalid ones, respectively. UTBoost (Yu et al., 2025) generates additional tests for SWE-bench to identify incorrect agent patches that passed the original suites. SWEABS (Yu et al., 2026) combines coverage-guided generation with mutation analysis, relaxing tests that reject semantically equivalent mutants and adding tests against incorrect mutants. SWE-Mutation (Sun et al., 2026) evaluates test discrimination using semantic mutants, while XREPOTEST (Quang et al., 2026) benchmarks multilingual repository-level unit test generation. Prior work on reward hackability in code RL environments (Rajan, 2026) combines reward-hackability auditing with test augmentation checked against reference solutions. Joint curation also appears in SWE-Bench ProMax (Shi et al., 2026), which revises descriptions with expert involvement and removes tests enforcing incidental implementation details or unstated requirements. Longtail-SWE uses solver trajectories and judge feedback to guide coordinated repairs to descriptions and tests, aiming to preserve intended requirements while addressing both incorrect acceptance and incorrect rejection of agent solutions.

3

Method

Longtail-SWE builds on SWE-rebench V2 for task collection and executable environment construction. We modify the candidate selection and environment installation procedures, and add a solver–judge–repairer stage that uses solution attempts to identify and repair task defects, followed by benchmark finalization based on additional solver runs for quality and difficulty assessment. The goal is to retain challenging tasks whose descriptions and tests support reliable evaluation. 3.1

Task Definition and Pipeline Overview

Longtail-SWE follows the repository-level issue-resolution setting of SWE-bench. Each task consists of a public problem statement, a repository at a pre-fix revision (the base commit), an executable environment, a reference patch, and an evaluation test patch. We obtain the problem statement from a GitHub issue and extract the reference and test patches from its associated merged pull request (PR). Our pipeline comprises four stages: collection, construction, repair, and dataset finalization (Figure 1). Throughout curation, we retain the original issue and PR data and keep the reference patch unchanged. During task construction, we identify FAIL_TO_PASS (F2P) tests by applying the same evaluation test patch at the base commit and after the reference patch: these tests fail in the first state and pass in the second. PASS_TO_PASS (P2P) tests pass in both states and serve as regression checks. The solver receives the statement and repository environment and produces a solution patch. We evaluate each solution patch in a fresh, separate container, discarding solver changes to test files before applying the evaluation test patch. We use this evaluation procedure consistently across pipeline stages. An attempt is successful if its solution passes all designated F2P and P2P tests.

3

1 Collection

2 Construction

3 Repair

4 Finalization

Find & filter repositories Collect & filter PRs Screen task semantics

Set up environment Audit task & validate tests Build & validate images

Attempt & evaluate Diagnose and repair Revalidate and repeat

Additional agent runs Difficulty calibration Final quality review

Screened candidates

Initial candidate pool

Judge-accepted tasks

Final dataset

Accept

Repair loop Solver

Judge

Attempt & evaluate

Diagnose task

Issue

Repairer

Execution check

Tests & task statement

Revalidate task Invalid

Valid: save task, then solve / reevaluate

Figure 1. Longtail-SWE pipeline. Executable candidates enter a solver–judge–repairer loop with execution revalidation; judge-accepted tasks proceed to difficulty selection and final quality review.

3.2

Candidate Collection and Screening

Source collection and prioritization. For each target language, we independently use the GitHub API to collect merged PRs linked to a single issue from eligible repositories. Candidates require a usable base commit and nonempty, separable reference and test patches. Appendix A.1 lists the repository eligibility criteria and PR filters. When processing capacity is limited, we prioritize larger reference patches by modified-file count, then patch length, both in descending order. Semantic screening. Kimi K3 (Kimi Team et al., 2026) assigns diagnostic categories inherited from SWE-rebench V2. We retain A (well-specified and solvable as-is) and two potentially repairable categories: B2 (tests require names, signatures, or values absent from the statement) and B5 (tests may depend on incidental reference-implementation details). Other categories, including tasks requiring unavailable external information, visual context, or non-English understanding, are excluded. 3.3

Environment Construction and Initial Validation

Environment preparation and initial audit. For each language, we prepare one or more versioned base Docker images to accommodate differences among repositories in runtime versions, build systems, and dependencies. Starting from the base commit, an auditor agent inspects the repository documentation, configuration, and CI, determines the project-native installation, build, and test commands, then installs task-specific dependencies and verifies the installation with existing tests. It also compares the problem statement, repository context, reference patch, and test patch, revising tests when necessary to generalize implementation-specific checks or add missing test coverage of the required behavior. The auditor may replace implementation hints in the problem statement with [HIDDEN], leaving other text unchanged. Examples of such replacements can be found in Appendix A.4. The subsequent repair procedure described in Section 3.4 is append-only. We preserve the raw issue, installation/build/test commands, and resulting test patch. Reproducibility checks. We repeat installation, building, and testing from scratch in a clean container based on the task’s selected base Docker image. We run the evaluation tests before and after applying the reference patch, requiring at least one F2P test and all designated evaluation tests to pass with the reference patch. We then build an image for each task with its repository dependencies pre-installed and repeat the checks inside it, comparing the results with the saved test groups and task configuration. Candidates enter 4

the next stage only after these checks succeed; unsuccessful construction attempts are returned for revision or excluded. 3.4

Solver-Guided Task Diagnosis and Repair

Diagnosis from solution attempts. For each executable candidate, a solver attempts to complete the task. A judge then reviews the solver trajectories, solution patches, and evaluation outcomes in the task-specific Docker image, alongside the statement, reference patch, and tests. The verdict can expose underspecification or overly restrictive checks, while test-passing solutions that are still incorrect can reveal missing test coverage for edge cases. The judge distinguishes implementation errors from description, test, or environment defects and accepts the candidate or requests repair. Once the judge accepts the current task version, the task proceeds to a judge agent using a different model configuration; if that judge identifies a new issue, the task returns to repair before judging continues. A task completes this process only after acceptance from all three judge models: GPT 5.6 Luna (xhigh), GPT 5.6 Terra (high), and Kimi K3 (high). Permissible repairs. A separate repair agent receives the active judge’s diagnosis and may revise the evaluation test patch and/or append clarifications to the problem statement. The statement produced by the initial task audit, including any [HIDDEN] replacements, remains immutable during repair. Clarifications are placed in an addendum (which may be revised during description repair) to specify the expected behavior while preserving the source issue’s meaning and keeping the reference implementation undisclosed. Test changes aim to address false positive/negative cases, where wrong solutions might pass due to missed edge cases or incomplete checks, and correct solutions might fail due to overly restrictive or incorrect expectations. For description repair we use solver attempts from Kimi K3 (high), GPT 5.6 Terra (high), and GPT 5.6 Luna (xhigh); after the complete statement and its addendum are frozen for the test-repair phase, solver attempts use GPT 5.6 Sol (high). The full model configurations for the auditor, solver, judge, and repairer are provided in Appendix A.9, and the complete agent prompts are available in our code repository. The environment remains fixed within this loop. Environment defects require returning to the construction stage. The repairer may not introduce production-code fixes through the test patch. Validation and reuse of solution attempts. Revised tests must satisfy the execution criteria above at the base commit and with the unchanged reference patch. If a revision fails, we retain the last validated version and return error logs to the repairer. Tasks with faulty reference patches are discarded. After validation, we save the updated task and return it to the solver–judge loop. Within this repair loop, test-only changes trigger reevaluation of existing solution patches, whereas changes to the problem statement require new solver attempts so that the solver can use the revised specification. The active judge reassesses the task using updated outcomes and available trajectories. Iteration and audit history. We run diagnosis–repair–validation cycles in sets of up to five rounds until all three sequential judges accept the task or we identify a faulty reference patch and discard the task. Candidate processing for a language stops once at least 30 tasks are accepted for difficulty calibration. Across the 237 difficulty-selected tasks (see Section 3.5 on difficulty calibration), repair takes 4.2 rounds on average (median 4, range 0–14). Figure A3 in Appendix A.7 shows the full distribution by language. All intermediate revisions, trajectories, and verdicts are preserved. 3.5

Benchmark Finalization and Release

After completing repairs to task statements and tests, we calibrate task difficulty. For each judge-accepted task, we run Qwen 3.8 27B (Qwen Team, 2026) with xhigh reasoning three times using a 262K-token context window to guide difficulty-based selection. We exclude tasks solved in all three attempts unless doing so would leave fewer than 15 tasks in a language. In that case, we retain enough of these tasks to meet the

5

Table 1. Overall performance and reported resource usage on Longtail-SWE. Success rates are percentages.

Model Opus 5.5 (high) GPT 6.1 Sol (high) GPT 5.6 Sol (high) GPT 5.6 Luna (xhigh) Kimi K3 (max) DeepSeek V4 Flash (high) GLM-5.3-Flash (max) Qwen 3.8 27B (xhigh) Qwen 3.6 27B (thinking)

Context limit

Pass@1 ± SEM

Pass@3

All-3

Mean total tokens

Mean steps

1M 1M 1M 1M 262K 1M 262K 262K 262K

81.52 ± 1.16 78.38 ± 0.92 75.74 ± 1.25 71.45 ± 0.44 63.70 ± 2.43 56.11 ± 1.47 48.18 ± 0.92 42.74 ± 0.92 16.67 ± 0.83

89.11 82.18 87.13 84.16 76.24 71.78 69.80 62.38 23.76

73.76 73.27 63.37 56.93 48.51 39.11 23.76 20.79 9.41

4.48M 1.75M 2.84M 13.01M 16.84M 26.50M 38.28M 23.06M 7.20M

53.2 27.4 34.3 76.5 151.4 199.5 313.7 214.4 113.4

minimum, prioritizing those with the longest average Qwen trajectory length across the three attempts. This minimum applies before post-evaluation validation; subsequent exclusions leave 12–27 tasks per language in the final 202-task benchmark. Post-evaluation validation. After evaluation, we refined the preliminary difficulty-selected pool of 237 tasks through an iterative audit, repair, and exclusion process informed by 5,688 stored solver attempts from eight model configurations, with three attempts per configuration per task (see Section 4.1). We first corrected evaluator defects in patch application and test-output parsing, then replayed the stored solution patches. We subsequently examined problem statements, tests, reference implementations, solver patches, and execution logs to identify tests that rejected valid solutions or accepted solutions that violated stated requirements. This semi-automated process combined human review with assistance from GPT 6 Luna, Claude Opus 5.5, GPT 6 Astra, and GPT 6 Sol, which also helped refine the tests and problem descriptions. We checked test changes by replaying stored solution patches and ran new solver attempts when problem statements changed. We excluded tasks that could not be repaired without changing their intent, such as those whose tests left requested functionality untested or conflicted with the reference patch. We retained only tasks that passed revalidation, excluding 35 tasks and yielding the final benchmark of 202 tasks. Table 4 (Section 5.1) presents the full selection funnel. Data and code availability. The complete release is publicly available through https://github.com/S WE-rebench/Longtail-SWE, with all materials provided either directly in the repository or through links to external resources. This includes the 202-task Longtail-SWE benchmark, the easier companion set, prompts, reproduction scripts, all benchmark instances, original and curated problem descriptions and tests, source metadata and reference patches, environment configurations, the complete data-collection, environmentconstruction, audit, repair, and evaluation tooling, all agent execution trajectories, and benchmark container images. Our original code is licensed under the MIT License, and our original dataset contributions under CC BY 4.0; third-party materials remain subject to their original licenses.

4

Experiments

4.1

Experimental Setup

Models and evaluation protocol. We evaluate nine model configurations on the 202-task LongtailSWE: Opus 5.5 (high), GPT 6.1 Sol (high), GPT 5.6 Sol (high), GPT 5.6 Luna (xhigh), Kimi K3 (max), DeepSeek V4 Flash (high), GLM-5.3-Flash (max), Qwen 3.8 27B (xhigh), and Qwen 3.6 27B (thinking). Each configuration uses a SWE-agent-like scaffold for three attempts per task. We compare these specific model–agent configurations, with reasoning modes and context limits listed in Table 1. The agent prompts are available in the repository.

6

Table 2. Pass@1 (%) by programming language, averaged over three attempts per task. Opus 5.5 GPT 6.1 GPT 5.6 GPT 5.6 Kimi K3 DeepSeek V4 GLM-5.3-Flash Qwen 3.8 27B Qwen 3.6 27B (high) Sol (high) Sol (high) Luna (xhigh) (max) Flash (high) (max) (xhigh) (thinking)

Language

N

Clojure Dart Elixir Erlang F# Haskell Julia Kotlin Lua OCaml R Scala Swift

13 14 23 12 16 14 27 12 13 13 14 18 13

82.1 71.4 75.4 100.0 66.7 92.9 86.4 77.8 89.7 84.6 78.6 87.0 69.2

74.4 61.9 81.2 100.0 62.5 95.2 93.8 63.9 89.7 76.9 78.6 66.7 64.1

51.3 81.0 72.5 86.1 81.3 83.3 85.2 72.2 89.7 64.1 76.2 66.7 69.2

46.2 81.0 65.2 83.3 68.8 83.3 77.8 61.1 74.4 76.9 69.0 70.4 69.2

64.1 66.7 53.6 83.3 47.9 76.2 76.5 50.0 66.7 56.4 81.0 53.7 51.3

35.9 61.9 34.8 75.0 41.7 78.6 58.0 66.7 66.7 53.8 69.0 50.0 56.4

30.8 40.5 31.9 50.0 43.8 66.7 59.3 58.3 59.0 43.6 57.1 37.0 53.8

38.5 38.1 30.4 58.3 29.2 52.4 38.3 47.2 51.3 28.2 69.0 44.4 46.2

10.3 11.9 8.7 25.0 10.4 26.2 16.0 22.2 17.9 5.1 42.9 14.8 12.8

Overall

202

81.5

78.4

75.7

71.5

63.7

56.1

48.2

42.7

16.7

Metrics. An attempt succeeds when its solution passes all required F2P and P2P tests under the protocol in Section 3.1. Pass@1 averages success across tasks and attempts; Pass@3 is the fraction of tasks solved at least once, and All-3 the fraction solved in all three attempts. We report success rates as percentages, run-level standard error of the mean (SEM) for Pass@1, and mean total token usage and agent step counts. Selection and evaluation. As described in Section 3.5, difficulty selection uses three Qwen 3.8 27B calibration attempts and excludes tasks solved in all three unless needed to retain at least 15 tasks per language before post-evaluation validation. The Qwen 3.8 results in Table 1 use a different set of three attempts; no attempt used for difficulty filtering is included in these reported scores. Nevertheless, because task selection depended partly on calibration runs of the same model, the reported Qwen results may still be biased by model-dependent selection and the language-coverage constraint. 4.2

Dataset Composition and Task Scope

Longtail-SWE contains 202 tasks from 129 repositories across 13 programming languages. Core feature development is the most frequent task-type annotation (67.3%), followed by major bug fixes (27.2%) and integration features; labels are not mutually exclusive. Appendix A.8 provides the full task-type distribution in Table A8, language and repository counts, and LLM-annotated difficulty. Appendix A.3 compares coverage and structural statistics with other repository-level benchmarks. 4.3

Overall Results

Task resolution. Opus 5.5 (high) has the highest point estimates on all three success metrics, with 81.52% Pass@1, exceeding runner-up GPT 6.1 Sol by 3.14 percentage points (Table 1). Even after three attempts, 10.89% of tasks remain unsolved by Opus. Qwen 3.8 27B exceeds Qwen 3.6 27B on all three success metrics. Resource usage. GPT 6.1 Sol has the lowest reported mean token usage and step count, followed by GPT 5.6 Sol and Opus. Luna uses approximately three times as many tokens as Opus. Kimi uses more tokens and steps than Luna despite a lower Pass@1, while GLM-5.3-Flash has the highest means for both resources. Repeated attempts. Figure 2 reports Qwen 3.8 27B Pass@k estimates on the final 202-task benchmark using ten attempts per task. Pass@k increases from 43.8% at k = 1 to 73.3% at k = 10, with diminishing marginal gains as k increases. Many tasks are therefore solvable by Qwen but not reliably in a single attempt, while 26.7% remain unsolved after ten attempts.

7

73.3%

80%

Estimated pass@k

61.9% 60%

40%

43.8%

20% Qwen 3.8 27B (10 runs) 0% 1

2

3

4

5

6

7

8

9

10

Number of attempts, k Figure 2. Qwen 3.8 27B pass@k on the 202-task subset, estimated from ten runs per task. Pass@1 is 43.8% and pass@10 is 73.3%.

4.4

Performance Across Languages

Variation across language subsets. Table 2 reports per-language results for all nine configurations. Averaging Pass@1 equally across configurations, Erlang has the highest mean success rate (73.5%), followed by Haskell (72.8%) and R (69.0%); Clojure, F#, and Elixir have the lowest rates (48.1%, 50.2%, and 50.4%). The highest and lowest means differ by approximately 1.5-fold. Opus 5.5 leads outright in four language subsets, ties GPT 6.1 Sol on Erlang, ties both Sol configurations on Lua, and ties GPT 5.6 Sol and Luna on Swift. GPT 6.1 Sol leads on Elixir, Haskell, and Julia. GPT 5.6 Sol leads on F# and ties Luna on Dart, and Kimi K3 leads on R (Table 2). 4.5

Human Validation of the Judge

Audit set and annotation protocol. We evaluate the judge on an audit set of 65 tasks stratified by language and pipeline stage, used exclusively for evaluation rather than judge development. Two engineers independently assessed each task against predefined criteria using the task artifacts, solver trajectory, and evaluation test patch, with optional agent assistance for code inspection. Every decision required a written justification, including concrete evidence whenever tests rejected valid solutions or accepted incorrect ones. The two annotators marked 44 and 41 of the 65 tasks as problematic, respectively, agreeing on 40 tasks (Cohen’s κ = 0.15, reflecting the difficulty of identifying subtle task defects). They then reconciled their notes to establish a consensus gold label for every task. Of the 25 initial disagreements, 21 were resolved as REJECT, typically because one annotator spotted a defect missed by the other. The remaining four received ACCEPT: one flagged test omission involved behavior outside the problem statement, one concerned a negligible corner case, and two involved non-standard test-result parsing that proved valid. The final gold set comprises 14 accepted and 51 rejected tasks. Judge setup and results. After freezing the gold labels, we ran our sequential three-judge protocol (Section 3.4) using GPT 5.6 Luna (xhigh), GPT 5.6 Terra (high), and Kimi K3 (high), which returns ACCEPT only upon unanimous acceptance and routes any REJECT to repair to prioritize sensitivity to defects. The protocol rejected 56 tasks in total, correctly identifying 49 of the 51 problematic gold tasks. Treating rejection

8

as the positive class, the judge achieves 87.5% precision and 96.1% recall: only 3.9% of human-identified defective tasks were missed, while 12.5% of rejected tasks were sent to repair unnecessarily.

5

Ablations

5.1

Construction-Pipeline Ablation

Long-tail language ecosystems offer fewer eligible pull requests, so early construction losses can leave too few tasks for meaningful per-language evaluation. We therefore compare retaining potentially repairable candidates and combining environment installation with test auditing against the SWE-rebench V2 pipeline, then track retention through iterative repair and difficulty selection. Retaining repairable annotations. SWE-rebench V2 retains only candidates labeled A, whose statements and tests appear aligned. We also retain potentially repairable B2 candidates (tests require names, signatures, or values absent from the statement) and B5 candidates (tests may enforce incidental implementation details), expanding the pool from 1,096 to 2,640 tasks. Appendix A.2 gives the category counts. Combined environment and test audit. Our Kimi K3 auditor (Kimi Team et al., 2026) combines environment installation with test-patch repair (Section 3.3). Strict execution validation (EV) requires the tests to compile and execute in both repository states, with at least one F2P transition and no regressions after the reference patch. Table 3 compares construction configurations on the same 1,096 A candidates: strict EV passes increase from 377 to 539 (34.4% to 49.2%). The auditor modifies test patches in 331 of the 539 successful A tasks (61.4%). These construction-stage revisions go beyond environment installation and precede iterative solver–judge– repair. Expanding the input pool to include B2/B5 candidates increases the yield from 539 to 1,247 passes, adding 708 passes from outside the original A pool. The combined screening and construction changes increase strict EV passes from 6 to 86 for Kotlin, 2 to 78 for Scala, and 9 to 54 for Swift. Appendix A.2 provides the remaining per-language results (Table A6) and discusses the recovered tasks. End-to-end funnel. Table 4 tracks both paths from a common candidate pool. Iterative repair and quality review yield 533 accepted tasks; difficulty selection retains 237 tasks, and the remaining tasks form the easier companion set. Post-evaluation validation reduces the 237 difficulty-selected tasks to the final 202-task Longtail-SWE benchmark (Section 3.5). Only 68 of the benchmark tasks would remain available through the original funnel. Post-repair annotation. After repair and post-evaluation validation, 196 of the 202 tasks (97.0%) receive an A annotation, compared with 66 of the 68 tasks available through the SWE-rebench V2 path. Counting only final A annotations on both paths still yields 2.97 times as many benchmark tasks (196 vs. 66; Table 4). Appendix A.2 reports results for the full accepted pool.

Table 3. Ablation of semantic-screening scope and the construction agent. Strict EV rates use the screened candidates in each row as the denominator. Pipeline configuration

Candidates

Strict EV passes

Pass rate (%)

1,096 1,096 2,640

377 539 1,247

34.4 49.2 47.2

SWE-rebench V2 install agent, A only Combined auditor, A only Combined auditor, A/B2/B5

9

Table 4. Construction funnel from a common pool of 3,261 candidates. Parentheses report the percentage of the initial pool retained at each stage. Gray entries are shown only for downstream comparison and are not part of the SWE-rebench V2 construction pipeline. Stage

SWE-rebench V2 path

Longtail-SWE

3,261 (100.0%) 1,096 (33.6%) 377 (11.6%) 364 (11.2%) 197 (6.0%) 80 (2.5%) 68 (2.1%) 66 (2.0%)

3,261 (100.0%) 2,640 (81.0%) 1,247 (38.2%) 1,201 (36.8%) 533 (16.3%) 237 (7.3%) 202 (6.2%) 196 (6.0%)

Before semantic annotation After semantic screening Strict execution validation Image build and revalidation After repair quality review Difficulty-selected pool After post-evaluation validation Final re-annotation as A

Table 5. Opus 5.5 pre- and post-repair performance on the 63 tasks with unchanged problem descriptions. Evaluation tests Pass@1 ± SEM Pass@3 All-3 Pre-repair Post-repair

29.1 ± 2.8 67.2 ± 3.5

36.5 77.8

20.6 54.0

Judge-based quality audit of the SWE-rebench V2 yield. Additionally, we evaluated all 364 tasks that passed image build and revalidation through the SWE-rebench V2 construction path using a single GPT 5.6 Luna (xhigh) judge. The judge identified at least one task-quality issue in 328 tasks (90.1%). The issue categories are not mutually exclusive: 300 tasks (82.4%) had hidden tests that required undocumented APIs or exact outputs, and 323 (88.7%) had insufficient test coverage. Only 36 tasks (9.9%) were accepted by this single judge. Because our sequential protocol accepts a task only when all three judges accept it, applying the remaining judges could only reduce this accepted set; 36 is therefore an upper bound on the number of these tasks that would pass the full protocol without repair. In contrast, every task released in Longtail-SWE was accepted by all three judges after the repair process. 5.2

Pre- versus Post-Repair Pass Rates

For 63 released tasks, the problem description did not change during repair. We therefore re-evaluated the solution patches from the same three Opus 5.5 (high) trajectories per task against both the pre-repair and post-repair tests. Table 5 shows that Pass@1 increases by 38.1 percentage points after repair, while Pass@3 increases by 41.3 points and All-3 by 33.3 points. Because the trajectories and problem descriptions are held fixed, these differences reflect changes to the evaluation tests. Appendix A.5 presents two case studies from this subset; Appendix A.6 separately illustrates a description-only repair with unchanged tests. The paired comparison also holds across ten Qwen 3.8 27B attempts on the same 63 tasks: Pass@1 rises from 15.2% to 31.0%, and Pass@10 rises from 36.5% to 68.3%.

6

Discussion

Longtail-SWE broadens repository-level evaluation across 13 language ecosystems that are underrepresented in existing benchmarks. Our results provide a starting point for studying coding agents in these ecosystems. Revising task descriptions and tests helps recover tasks from a limited candidate pool. Our fixed-patch ablation isolates the effect of test refinement on a subset with unchanged descriptions. Further ablations are

10

needed to separate the effects of other refinement steps and assess whether construction-time solver choices favor particular models. The human audit evaluates judge reliability on a small sample drawn from multiple pipeline stages. It does not establish the error rate of the final benchmark; this requires an independent audit of the released tasks. 6.1

Threats to Validity

Construct validity. Fixing the reference patch and allowing only appended clarifications limit changes to task intent, but revisions may still alter the task contract. The paired analysis of 63 tasks with unchanged descriptions supports the test revisions; description changes are not separately evaluated in that analysis. The 65-task human audit shows high recall for problematic tasks (49/51, 96.1%) but low specificity for acceptable tasks (7/14, 50.0%), alongside low initial annotator agreement. This low specificity sends some acceptable tasks through unnecessary repair, increasing the computational resources required by the construction process. Internal validity. Difficulty selection with Qwen 3.8 27B may bias its reported scores. Some evaluated models also participate in construction and may benefit from the resulting task interpretations, though models not used during construction offer a partial check. In future work, we plan to investigate how a model’s involvement in benchmark construction affects its scores. Contamination. Tasks derive from public issues and pull requests, so overlap with training data cannot be ruled out. Because training cutoffs are often undisclosed and not determined by release dates, it is difficult to ensure that a fixed task set postdates the training data of all evaluated models. However, our pipeline can collect tasks created on or after a specified date, restricting evaluation to tasks created after a given model’s training cutoff. External validity. Repository and PR filters (permissive licenses, minimum popularity, single-issue links, prioritization of larger patches) shape the sample. With 12–27 tasks per language, language-level differences reflect these tasks and the evaluated configurations rather than inherent language difficulty.

7

Conclusion

We introduced Longtail-SWE, a benchmark of 202 issue-resolution tasks from 129 repositories across 13 underrepresented programming languages. Building on SWE-rebench V2, we developed a solver–judge– repairer pipeline designed to retain challenging candidates by addressing defects in task descriptions and tests. Across nine evaluated configurations, the highest observed scores are 81.52% Pass@1 and 89.11% Pass@3. The benchmark supports evaluation across these ecosystems and further research on reliable benchmark construction; current data and code availability is described in Section 3.5.

References Badertdinov, I., Golubev, A., Nekrashevich, M., Shevtsov, A., Karasik, S., Andriushchenko, A., Trofimova, M., Litvintseva, D., and Yangel, B. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents. arXiv preprint arXiv:2505.20411, 2025. URL https://arxiv.org/abs/2505.20411. Badertdinov, I., Nekrashevich, M., Shevtsov, A., and Golubev, A. SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale. arXiv preprint arXiv:2602.23866, 2026. doi: 10.48550/arXiv.2602.23866. URL https://arxiv.org/abs/2602.23866. Chen, M., Zhang, L., Feng, Y., Wang, X., Zhao, W., Cao, R., Yang, J., Chen, J., Li, M., Ma, Z., Ge, H., Zhang, Z., Cui, Z., Liu, D., Zhou, J., Sun, J., Lin, J., and Hui, B. SWE-Universe: Scale Real-World

11

Verifiable Environments to Millions. arXiv preprint arXiv:2602.02361, 2026. URL https://arxiv.org/ab s/2602.02361. Chowdhury, N., Aung, J., Shern, C. J., Jaffe, O., Sherburn, D., Starace, G., Mays, E., Dias, R., Aljubeh, M., Glaese, M., Jimenez, C. E., Yang, J., Ho, L., Patwardhan, T., Liu, K., and Madry, A. Introducing SWE-bench Verified. OpenAI, August 2024. URL https://openai.com/index/introducing-swe-bench -verified/. Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., Sampath, K., Krishnan, M., Kundurthy, S., Hendryx, S., Wang, Z., Bharadwaj, V., Holm, J., Aluri, R., Zhang, C. B. C., Jacobson, N., Liu, B., and Kenstler, B. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv preprint arXiv:2509.16941, 2025. URL https://arxiv.org/abs/2509.16941. Desai, R., Hu, J., Cabezas, J., Harsola, N., Shukla, P., Chaim, R. B., Assadi, A. E., Kamath, O. M., Faldu, F., Hebbar, P., Sun, J., Li, Y., Srinivasan, P., Gupta, I., Settles, C., Wang, D., Chen, D., Raja, P., Liu, A., Šuppa, M., Sasikumar, N., Kong, L., Quintanilla, E., Li, X., Bercovich, I., and Dillmann, S. SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? arXiv preprint arXiv:2606.07682, 2026. doi: 10.48550/arXiv.2606.07682. URL https://arxiv.org/abs/2606.07682. Endres, M., Fakhoury, S., Chakraborty, S., and Lahiri, S. K. Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions? Proceedings of the ACM on Software Engineering, 1 (FSE):1889–1912, 2024. doi: 10.1145/3660791. URL https://doi.org/10.1145/3660791. Fakorede, M. A., Upadhyay, K., Siddique, A. B., and Farooq, U. MobileDev-Bench: A Comprehensive Benchmark for Evaluating Language Models on Mobile Application Development. arXiv preprint arXiv:2603.24946, 2026. doi: 10.48550/arXiv.2603.24946. URL https://arxiv.org/abs/2603.24946. Guo, L., Wang, Y., Li, C., Tao, W., Yang, P., Chen, J., Song, H., Tang, D., and Zheng, Z. SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarks. arXiv preprint arXiv:2506.10954, 2025. URL https://arxiv.org/abs/2506.10954. Hong, D., Chi, Y., Li, W., Wang, X., Gao, M., Yang, K., He, B., Zheng, Y., Xiao, C., and Na, Q. SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration? arXiv preprint arXiv:2608.23564, 2026. doi: 10.48550/arXiv.2608.23564. URL https://arxiv.org/abs/2608.23564. Huang, W., Lee, C., Tng, L., and Ge, S. DeepSWE: Measuring Frontier Coding Agents on Original, LongHorizon Engineering Tasks. arXiv preprint arXiv:2607.07946, 2026. doi: 10.48550/arXiv.2607.07946. URL https://arxiv.org/abs/2607.07946. Jain, N., Singh, J., Shetty, M., Zheng, L., Sen, K., and Stoica, I. R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents. arXiv preprint arXiv:2504.07164, 2025. doi: 10.48550/arXiv.2504.07164. URL https://arxiv.org/abs/2504.07164. JetBrains. The Kotlin Benchmark. Benchmark website, 2026. URL https://kotlinlang.org/benchmark/. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv preprint arXiv:2310.06770, 2023. doi: 10.48550/arXiv.2310.06770. URL https://arxiv.org/abs/2310.06770. Khandpur, K., Lieret, K., Jimenez, C. E., Press, O., and Yang, J. SWE-bench Multilingual. SWE-bench release notes, May 2025. URL https://www.swebench.com/multilingual.html. Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., Charles, Y., Che, H. S., Chen, G., Chen, G., Chen, G., Chen, H., Chen, J., Chen, J., Chen, J., Chen, K., Chen, P., Chen, R., Chen, W., Chen, X., Chen, Y., Chen, Y., Chen, Y., Chen, Y., Chen, Y., Chen, Y., Chen, Y., Chen, Z., Cheng, D., Cheng, Y., Cui, J., Cui, J., Dai, A., Deng, J., Ding, H., Ding, R., Ding, S., Dong, M., Dong, M., Dong, Y., Dong, Y., Du, A., Du, C., Du, D., Du, J., Du, Y., Fan, Y., Feng, J., Feng, Q., Feng, Y., Fu, K., Fu, Q., Gao, F., Gao, H., Gao, J., Gao, T., Gao, W., Geng, S., Gong, J., Gong, L., Gong, S., Gong, X., Gu, Q., Gu, Y., Guan, S., Guo, H., Guo, S., Guo, X., Guo, Z., Hao, B., Hao, W., Hao, X., He, D., He, H., 12

He, L., He, Q., He, W., He, X., He, X., He, Y., He, Y., Hong, C., Hong, T., Hu, H., Hu, J., Hu, R., Hu, W., Hu, Y., Hu, Z., Hua, L., Huang, J., Huang, K., Huang, R., Huang, S., Huang, W., Huang, Y., Huang, Z., Huang, Z., Hui, Y., Jia, C., Jiang, Y., Jiang, Z., Jiang, Z., Jin, W., Jin, X., Jing, Y., Kong, H., Lai, G., Li, A., Li, C., Li, C., Li, C., Li, F., Li, G., Li, H., Li, J., Li, J., Li, L., Li, L., Li, L., Li, W., Li, W., Li, X., Li, Y., Li, Y., Li, Y., Li, Y., Li, Z., Li, Z., Li, Z., Li, Z., Li, Z., Lin, J., Lin, X., Lin, Y., Lin, Z., Lin, Z., Liu, B., Liu, B., Liu, C., Liu, L., Liu, S., Liu, S., Liu, S., Liu, T., Liu, W., Liu, Y., Liu, Y., Liu, Y., Liu, Y., Liu, Z., Liu, Z., Lu, E., Lu, H., Lu, L., Lu, T., Lu, Z., Luo, A., Luo, G., Luo, J., Luo, Y., Lyu, B., Lyu, W., Mao, S., Mei, Y., Men, X., Ni, M., Niu, Y., Pan, S., Peng, S., Qi, Z., Qin, R., Qin, Z., Qin, Z., Qiu, H., Qiu, J., Qiu, J., Qu, B., Qu, Y., Shang, Z., Shao, Y., Shen, H., Shi, J., Shi, J., Shi, L., Shi, S., Siu, W., Song, P., Song, X., Su, J., Su, Y., Su, Z., Sui, L., Sun, J., Sun, J., Sun, S., Sun, S., Sun, T., Sun, Y., Tai, Y., Tang, C., Tang, H., Tang, S., Tang, Z., Tian, C., Tian, R., Tian, Y., Tu, W., Wang, C., Wang, C., Wang, C., Wang, D., Wang, F., Wang, H., Wang, H., Wang, H., Wang, H., Wang, H., Wang, H., Wang, J., Wang, J., Wang, J., Wang, J., Wang, L., Wang, S., Wang, S., Wang, S., Wang, S., Wang, S., Wang, T., Wang, W., Wang, X., Wang, X., Wang, X., Wang, X., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Y., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wang, Z., Wei, C., Wei, M., Wei, S., Wen, Z., Wu, F., Wu, H., Wu, R., Wu, W., Wu, X., Wu, Y., Wu, Y., Wu, Y., Wu, Z., Xian, X., Xiang, C., Xiang, Y., Xiao, B., Xiao, C., Xiao, X., Xie, J., Xie, X., Xie, Y., Xie, Z., Xing, B., Xiong, Y., Xu, B., Xu, B., Xu, J., Xu, J., Xu, J., Xu, J., Xu, L. H., Xu, Q., Xu, S., Xu, S., Xu, T., Xu, T., Xu, W., Xu, X., Xu, Y., Xu, Y., Xu, Y., Xu, Z., Xue, H., Yan, J., Yan, Y., Yang, F., Yang, G., Yang, H., Yang, J., Yang, R., Yang, W., Yang, X., Yang, X., Yang, Y., Yang, Y., Yang, Y., Yang, Y., Yang, Z., Yang, Z., Yang, Z., Yang, Z., Yao, H., Ye, D., Ye, H., Ye, W., Ye, Z., Yin, B., Yin, H., Yin, X., Yu, C., Yu, H., Yu, L., Yu, S., Yu, S., Yu, T., Yuan, E., Yuan, M., Yue, T., Yue, W., Yue, Y., Zha, D., Zhan, H., Zhang, B. H., Zhang, D., Zhang, F., Zhang, H., Zhang, H., Zhang, H., Zhang, J., Zhang, J., Zhang, J., Zhang, K., Zhang, M., Zhang, P., Zhang, Q., Zhang, R., Zhang, R., Zhang, S., Zhang, S., Zhang, X., Zhang, X., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Y., Zhang, Z., Zhang, Z., Zhao, B., Zhao, C., Zhao, F., Zhao, J., Zhao, J., Zhao, S., Zhao, W., Zhao, X., Zhao, X., Zhao, Y., Zhao, Z., Zheng, H., Zheng, H., Zheng, R., Zheng, S., Zheng, T., Zhong, H., Zhong, L., Zhong, L., Zhou, M., Zhou, Q., Zhou, R., Zhou, R., Zhou, X., Zhou, Y., Zhou, Z., Zhu, J., Zhu, L., Zhu, X., Zhu, Y., Zhu, Y., Zhu, Z., Zhuang, C., Zhuang, W., and Zu, X. Kimi K3: Open Frontier Intelligence. arXiv preprint arXiv:2607.24653, 2026. doi: 10.48550/arXiv.2607.24653. URL https://arxiv.org/abs/2607.24653. Kondra, R., Mhatre, S., Kumar, A., Chu, E., Ahmad, B. B., Nangia, A., Agarwal, R., Dasgupta, A., Sinha, A., Sridharan, B., Dave, K., Graham, B., Song, G., Rahul, A., Lim, W. H., Thangamuthu, A., Singh, R., Liu, D., Pour, N., Chen, C., and Mattern, J. FrontierSWE v2. Proximal Blog, September 2026. URL https://www.frontierswe.com/blog/v2. Le, T., Thai, M. V. T., Manh, D. N., Nhat, H. P., and Bui, N. D. Q. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios. arXiv preprint arXiv:2512.18470, 2025. doi: 10.48550/arXiv.2512.18470. URL https://arxiv.org/abs/2512.18470. Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 21558–21572. Curran Associates, Inc., 2023. doi: 10.52202/075280-0943. URL https: //openreview.net/forum?id=1qvx610Cu7. Oliva, G. A., Rajbahadur, G. K., Bhatia, A., Zhang, H., Chen, Y., Chen, Z., Leung, A., Lin, D., Chen, B., and Hassan, A. E. SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation. arXiv preprint arXiv:2507.09108, 2025. URL https://arxiv.org/abs/2507.09108. Qi, Z., Long, F., Achour, S., and Rinard, M. An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems. In Proceedings of the 2015 International Symposium on Software Testing and Analysis, pp. 24–36. ACM, 2015. doi: 10.1145/2771783.2771791. URL https://people.csail.mit.edu/zichaoqi/papers/kali-issta15/paper.pdf. Quang, D. L., Van, D. C., Hai, N. L., Van, L. N., Bui, A. M. T., and Nguyen, P. T. XREPOTEST: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language Models. arXiv 13

preprint arXiv:2608.25939, 2026. doi: 10.48550/arXiv.2608.25939. URL https://arxiv.org/abs/2608.2 5939. Qwen Team. Qwen3.8-27B. Hugging Face model card, August 2026. URL https://huggingface.co/Qwen/ Qwen3.8-27B. Rajan, S. Auditing reward hackability in code RL training environments. arXiv preprint arXiv:2606.16062, 2026. URL https://arxiv.org/abs/2606.16062. Rashid, M. S., Bock, C., Zhuang, Y., Buchholz, A., Esler, T., Valentin, S., Franceschi, L., Wistuba, M., Sivaprasad, P. T., Kim, W. J., Deoras, A., Zappella, G., and Callot, L. SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents. arXiv preprint arXiv:2504.08703, 2025. doi: 10.48550/arXiv.2504.08703. URL https://arxiv.org/abs/2504.08703. Shi, Y., Xu, J., Fu, K., Zeng, W., He, S., Zhang, L., Liu, Y., Zhao, Z., Zhuo, T. Y., Cao, J., Ye, S., Liu, T., Cai, K., Cheung, S.-C., and Gu, X. SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring. arXiv preprint arXiv:2608.09802, 2026. doi: 10.48550/arXiv.2608.09802. URL https://arxiv.org/abs/2608.09802. Sun, Y., Zhao, Y., Wang, Y., Du, Y., Ma, Z., Wang, J., Zhang, M., Zhang, K., and Huang, Z. SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering? arXiv preprint arXiv:2605.22175, 2026. doi: 10.48550/arXiv.2605.22175. URL https://arxiv.org/abs/2605.22175. Wang, B., Zhang, C., Liu, D., Zhang, J., Chen, J., Chen, M., Fang, R., Zhang, S., Wang, X., Jing, Y., Ma, Z., and Cui, Z. The Verification Horizon: No Silver Bullet for Coding Agent Rewards. arXiv preprint arXiv:2606.26300, 2026a. doi: 10.48550/arXiv.2606.26300. URL https://arxiv.org/abs/2606.26300. Wang, J., Bianchi, F., Zhu, S., Nie, F., Kwon, Y., Dhingra, B., and Zou, J. Automated Benchmark Auditing for AI Agents and Large Language Models. arXiv preprint arXiv:2605.26079, 2026b. doi: 10.48550/arXiv.2605.26079. URL https://arxiv.org/abs/2605.26079. Wang, L., Ramalho, L., Celestino, A., Pham, P. A., Liu, Y., Sinha, U. K., Portillo, A., Osunwa, O., and Maduekwe, G. SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories. arXiv preprint arXiv:2512.17419, 2025. doi: 10.48550/arXiv .2512.17419. URL https://arxiv.org/abs/2512.17419. Wang, M., Xu, J., and He, P. PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWEBench-Like Benchmarks. arXiv preprint arXiv:2607.28587, 2026c. doi: 10.48550/arXiv.2607.28587. URL https://arxiv.org/abs/2607.28587. Wu, J., Zhang, Z., Zhang, B., Wang, X., Su, Y., Chen, M., Wang, P., Wang, Z., Shen, Q., Zhou, H., Yang, A., Huang, F., Yang, Y., and Liu, D. Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments. arXiv preprint arXiv:2609.04148, 2026. doi: 10.48550/arXiv.2609.04148. URL https://arxiv.org/abs/2609.04148. Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., and Yang, D. SWE-smith: Scaling Data for Software Engineering Agents. arXiv preprint arXiv:2504.21798, 2025. doi: 10.48550/arXiv.2504.21798. URL https://arxiv.org/abs/2504.21798. Yang, J., Lieret, K., Ma, J., Thakkar, P., Pedchenko, D., Sootla, S., McMilin, E., Yin, P., Hou, R., Synnaeve, G., Yang, D., and Press, O. ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv preprint arXiv:2605.03546, 2026. doi: 10.48550/arXiv.2605.03546. URL https: //arxiv.org/abs/2605.03546. Yu, B., Zhu, Y., He, P., and Kang, D. UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3762–3774, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.189. URL https://aclanthology.org/2025.acl-long.189/.

14

Yu, B., Cao, Y., Zhang, Y., Lin, L., Xu, J., Zhong, Z., Xu, Q., Wang, G., Cao, J., Cheung, S.-C., He, P., and Briand, L. SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark. arXiv preprint arXiv:2603.00520, 2026. doi: 10.48550/arXiv.2603.00520. URL https://arxiv.org/abs/2603.00520. Zan, D., Huang, Z., Liu, W., Chen, H., Zhang, L., Xin, S., Chen, L., Liu, Q., Zhong, X., Li, A., Liu, S., Xiao, Y., Chen, L., Zhang, Y., Su, J., Liu, T., Long, R., Shen, K., and Xiang, L. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. arXiv preprint arXiv:2504.02605, 2025. doi: 10.48550/arXiv.2504.02605. URL https://arxiv.org/abs/2504.02605.

15

A

Appendix

A.1

Detailed Collection and Filtering Rules

Repository-level collection and filtering rules. Repository collection uses the following criteria and search procedures: • Repositories are searched separately for each target language using the GitHub language label. • Repositories must have at least 10 GitHub stars. • Repositories must have been updated on or after January 1, 2026. • Archived repositories are excluded. • Disabled repositories are excluded. • Only licenses on the permissive SPDX allowlist are accepted: 0BSD, Apache-2.0, BSD-2-Clause, BSD-3-Clause, BSL-1.0, CC0-1.0, ISC, MIT, PostgreSQL, Python-2.0, Unlicense, and Zlib. • Missing or unknown licenses are rejected unless an exception is explicitly recorded. • No filter is specified in the default configuration for total file count, lines of code, or checkout size. • Searches reaching the 1,000-result cap are split into nonoverlapping date or star-count ranges; results are then deduplicated. Pull-request-level collection and filtering rules. PR collection uses the following criteria, with processing order noted separately from eligibility: • Only merged PRs are accepted. • Each PR must be linked to an issue. • PRs linked to multiple distinct issues are excluded; duplicate records of the same PR–issue pair do not count as multiple issues. • PRs must fall within the configured inclusive date window. • Bot-generated PRs are excluded. • Automated dependency updates are excluded. • PRs may change at most 1,000 files. • The production/reference patch must modify at least one supported file. • The test patch must modify at least one supported source or test-bearing file. • The production patch must contain at least five added lines by default. • The combined size of recognized diff sections for code files must be less than 100,000 characters. • The problem statement must be less than 100,000 characters. • No upper limit on test patch size is imposed by the default language-specific procedure. • Both the reference patch and the test patch must be nonempty. • A usable base commit must exist. • Candidates are rejected if tests are detected inside production (non-test) files. • Some language batches are sorted by change size; this determines processing order rather than serving as a universal acceptance filter. 16

Table A6. Execution-validation (EV) passes by language. SWE-rebench V2 uses the A-only pool; LongtailSWE uses the expanded A/B2/B5 pool and combined auditor. The comparison therefore includes both screening and construction changes. Language

A.2

SWE-rebench V2

Longtail-SWE

Clojure Dart Elixir Erlang F# Haskell Julia Kotlin Lua OCaml R Scala Swift

21 20 59 36 62 49 27 6 42 16 28 2 9

55 122 109 57 74 126 123 86 153 114 96 78 54

Total

377

1,247

Additional Construction-Pipeline Results

Semantic-screening counts. The common pre-screening pool contains 3,261 candidates across 13 languages. Of these, 1,096 receive label A, 1,408 receive B2, and 136 receive B5, as defined in Section 5.1. Retaining all three categories adds 1,544 candidates to the A-only pool, a 140.9% increase. Test-patch revisions during construction. Among the 539 A candidates that pass strict execution validation with the combined auditor, 331 (61.4%) have test patches modified by the auditor. These revisions occur during construction, before iterative solver–judge–repair. Execution-validation passes by language. Table A6 breaks down strict EV passes by language for the construction configurations in Section 5.1. Relative increases are particularly large for Kotlin, Scala, and Swift. Audit logs indicate that repairing test patches that failed to compile at the base revision recovered tasks rejected by the installation-only baseline. Post-repair annotation of the accepted pool. Of the 533 tasks accepted after iterative repair and quality review, 517 (97.0%) receive a final A annotation; annotation cannot be completed for five additional tasks because they exceed the annotator’s context limit. Difficulty selection retains a preliminary pool of 237 tasks; the remaining tasks form the easier companion set. Post-evaluation validation then reduces this pool

Table A7. Coverage and structural statistics for selected repository-level benchmarks. Tasks, repositories, and languages are counts; all remaining columns are per-task means. Words denotes statement words, files denotes changed solution files, and reference/test additions count added lines in the respective patches. Dashes indicate unavailable measurements.

Dataset Longtail-SWE SWE-Bench ProMax SWE-Bench Pro (Public) SWE-bench Multilingual

Reference Test Tasks Repos. Langs. Words Files additions additions 202 170 731 300

129 70 11 41

13 513.25 13.14 7 481.45 12.38 4 178.88 5.07 9 234.83 1.74

17

414.82 342.05 120.34 28.87

F2P

P2P

645.87 23.97 42.91 236.52 — — 105.73 14.43 57.46 39.35 1.44 34.02

Listing A1. Hidden implementation hint in livekit__client-sdk-android-874; the replaced content is given after three dashes. Library plants Timber.DebugTree() globally, affecting host app logging Hi! LiveKit currently plants Timber.DebugTree() internally when loggingLevel != OFF (if no DebugTree is present in the forest). Since Timber is a global logger, this affects the entire host application, not just LiveKit logs, and removes control over logging configuration (e.g. custom Trees like Crashlytics-only logging). Library code generally should not plant Timber trees, as this is an application-level concern. [HIDDEN] Thanks! --Suggested alternatives: - Do not plant any Timber Tree, only log via Timber with a specific tag - Provide a logger interface / callback so the host app decides where logs go - Expose a flag or API to opt out of planting DebugTree

Listing A2. Hidden implementation hint in amazon-mq__rabbitmq-stream-s3-205; the replaced content is given after three dashes. AWS API: Use a ‘gen_server‘ for short-lived credentials Currently ‘rabbitmq_stream_s3_api_aws‘ uses a lock (‘global:set_lock/3‘) to prevent multiple processes from attempting to refresh short-lived credentials simultaneously. We should replace this with a simple ‘gen_server‘ which is responsible for holding and refreshing short-lived credentials. Using a ‘gen_server‘ would avoid the locking (which is unnecessarily complicated) and it would let us eagerly refresh the short-lived credentials in the background when the credentials are close to expiration. [HIDDEN] --It can start up and return ‘ignore‘ from ‘c:gen_server:init/1‘ when long-lived credentials are configured.

to the final 202-task benchmark (Section 3.5). A.3

Structural Comparison with Other Benchmarks

Among the benchmarks in Table A7, Longtail-SWE covers the most repositories and languages and has larger reference changes and test additions on average. A.4

Implementation Hints Hidden by the Auditor Agent

In the final 202-task dataset, the initial task audit replaced parts of the problem description with [HIDDEN] in 56 tasks (113 occurrences in total). We sampled two examples of such replacements, which are given in Listings A1 and A2.

18

A.5

Case Studies of Test Repair

We sampled two illustrative cases from the 63 released tasks whose problem statements did not change during repair. This restriction allows the same solver trajectories and solution patches to be evaluated against the pre-repair and post-repair tests: for each corresponding run, the Opus 5.5 (high) trajectory and submitted patch are identical across the two evaluations. The cases illustrate two distinct test defects and are not intended to estimate their prevalence in the dataset. Removing an unstated representation constraint. The esl__amoc-193 task asks that a required module attribute accept a binary description, making Elixir strings usable alongside the existing Erlang charlist form. All three Opus patches add a binary clause to check_mandatory_fields while retaining the existing charlist clause. The pre-repair tests correctly check that a binary description is accepted, but also require a charlist description to be stored as a binary and make additional exact-value assertions about nested character data and Unicode. The problem statement does not require conversion of existing charlists. All three patches fail the pre-repair suite at the charlist assertion even though they implement the requested binary-acceptance behavior. The post-repair tests continue to require the binary description to be preserved exactly. For the legacy charlist, however, they check successful processing without prescribing its stored representation; they also remove the nested-character-data and Unicode-normalization assertions. The test identifiers and command remain unchanged. Under the post-repair tests, the same three Opus patches pass. This repair prevents rejection of solutions that meet the stated requirement but do not reproduce the reference patch’s normalization choice. Adding coverage for an incomplete implementation. The intersectmbo__cardano-ledger-5671 task asks for NoThunks instances to be removed from predicate failures throughout the codebase and from ContextError types in every era. The run 2 Opus patch removes many such instances but leaves five in the TPraos protocol code. The reference patch removes all five. The pre-repair tests check only five Shelley predicate-failure types and a Void control case, so they do not exercise the remaining TPraos types and incorrectly accept the run 2 patch. The post-repair tests expand the checks across eras and protocol rules while retaining a control for the detection mechanism. When the identical run 2 patch is evaluated against the post-repair suite, 5 of 74 examples fail—exactly the five remaining TPraos instances. The test command is unchanged. Thus, the post-repair tests turn a false positive into a failure by covering behavior explicitly required by the problem statement. A.6

Case Study of Problem-Statement Repair

We additionally sampled one of the five tasks for which repair changed only the problem statement, leaving the evaluation test_patch byte-for-byte unchanged. This complements the preceding test-repair cases by illustrating how statement repair can expose requirements that the tests already enforce. The selected task, penelopeysm__flexichains.jl-229, asks for merging FlexiSummary objects. Its original 68-word statement describes the central use case, shown in Listing A3. The repaired 401-word statement in Listing A4 retains that text and makes the tested contract explicit. The unchanged tests cover rightmost-value precedence, missing entries for absent key–statistic combinations, retained- and collapsed-axis behavior, mixed key types and their warning, and merges of three or more summaries. The original statement communicates the main operation, whereas the expanded statement surfaces edge cases and design choices that a solver could not reliably infer. A.7

Repair-Round Distribution

Figure A3 shows the distribution of repair rounds overall and by language.

19

Overall (n=237) Clojure (n=15) Dart (n=15) Elixir (n=28) Erlang (n=15)

Language

F# (n=18) Haskell (n=18) Julia (n=31) Kotlin (n=15) Lua (n=15) OCaml (n=15) R (n=15) Scala (n=22) Swift (n=15) 0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

Accepted repair cycles before convergence Density

Interquartile range

Minimum–maximum

Mean

Median

Individual task

Figure A3. Distribution of accepted repair rounds before convergence for the 237 difficulty-selected tasks, overall and by language (overall mean: 4.22 rounds).

A.8

Detailed Dataset Composition

Table A8 reports the full distribution of task-type annotations. Labels are not mutually exclusive: a task contributes to every category assigned to it. Table A9 breaks down the final sample by language and repository count. Difficulty labels are LLM annotations rather than measurements of human resolution time. A.9

Agent Model Configurations

This section summarizes the model configurations and token usage for the agents used during task construction and repair (Sections 3.3 and 3.4). Evaluation configurations are listed in Table 1 of the main paper; the complete agent prompts are provided in our public repository (Section 3.5). The auditor and repairer use Kimi K3 with temperature 1, top_p 1, high reasoning, and a 1M-token context window. The sequential judges use GPT 5.6 Luna (xhigh), GPT 5.6 Terra (high), and Kimi K3 (high). During description repair, solver attempts use Kimi K3 (high), GPT 5.6 Terra (high), and GPT 5.6 Luna (xhigh). After the complete statement and its addendum are frozen for the test-repair phase, solver attempts use GPT 5.6 Sol (high). Kimi K3 uses the same configuration across its auditor, repairer, judge, and construction-time solver roles. The auditor, judge, and repairer average 2.8M, 0.7M, and 3.3M tokens per trajectory, respectively. Mean solver token usage is reported in Table 1 of the main paper.

20

Table A8. Task-type annotations for the 202 Longtail-SWE tasks. A task may have multiple labels; percentages use all 202 tasks as the denominator and do not sum to 100%. Task type

Tasks

Share (%)

136 55 25 20 19 11 9 8 5 4 4 1

67.3 27.2 12.4 9.9 9.4 5.4 4.5 4.0 2.5 2.0 2.0 0.5

Core features Major bugs Integration features Edge-case bugs Minor bugs UI/UX features DevOps enhancements Performance bugs Security bugs Critical bugs Documentation enhancements Regression bugs

Table A9. Distribution of Longtail-SWE tasks and repositories by language and difficulty. Language

Tasks

Repos

Easy

Medium

Hard

Clojure Dart Elixir Erlang F# Haskell Julia Kotlin Lua OCaml R Scala Swift

13 14 23 12 16 14 27 12 13 13 14 18 13

10 11 13 7 8 8 22 12 8 2 9 11 8

0 1 0 2 0 1 0 0 1 0 1 0 1

3 3 7 4 13 5 6 1 8 6 11 4 2

10 10 16 6 3 8 21 11 4 7 2 14 10

Total

202

129

7

73

122

21

Listing A3. Problem statement before repair for the FlexiSummary merge task. merge on FlexiSummary If two ‘FlexiSummary‘s have the same iter_indices and chain_indices, but different stat indices, we should be able to merge them. In principle one may want to merge along different axes, but I think this is the main use case. For example, you can do ‘‘‘julia chain = ... merge(mean(chain), hdi(chain; prob=0.95, split_interval=true)) ‘‘‘ and that would easily replicate the behaviour of tidybayes’ (for example) ‘mean_hdi‘.

Listing A4. Problem statement after repair for the FlexiSummary merge task, which shows additional requirements appended to the original problem statement. <THE ORIGINAL PROBLEM STATEMENT VERBATIM> ### Additional contract requirements ## Merging ‘FlexiSummary‘ objects ‘Base.merge‘ is extended to ‘FlexiSummary‘, with the following observable behavior: - Two summaries can be merged provided that, for every collapsed dimension axis that both summaries retain, their iteration indices are equal and their chain indices are equal. If either the retained iteration indices or the retained chain indices differ, ‘merge‘ throws a ‘DimensionMismatch‘. A summary collapsed along an axis has no indices for that axis, so such a collapsed axis places no equality constraint on the other summary for that axis. - The merged summary contains the union of the keys of the input summaries and the union of their statistic names. - Where a key is present in both summaries with the same statistic, the value from the later (rightmost) summary takes priority. Likewise, when a statistic name appears in both summaries, the later summary’s values for that statistic take priority. - When a key/statistic combination is present in only one of the input summaries, indexing the merged summary at that combination for the summary that lacks it yields ‘missing‘. - If both summaries have a statistic dimension marked as dropped (e.g. each summary was produced with a single statistic whose statistic dimension was dropped), then: if both summaries contain exactly the same statistic name, the merged summary also has its statistic dimension dropped; if the statistic names differ, the merged summary retains its statistic dimension over the combined statistic names. - For a summary collapsed over iterations, the merged result likewise has ‘iter_indices(merged) === nothing‘; for a summary collapsed over chains, ‘chain_indices(merged) === nothing‘. Retained indices are preserved in the merged result. - When the input summaries have different key types (for example ‘FlexiSummary{Symbol}‘ and ‘FlexiSummary{String}‘), ‘merge‘ emits a warning mentioning "different key types" and returns a summary whose keys are promoted so that all original keys (with their original types) can still be accessed on the result. - ‘merge‘ accepts three or more summaries, combining them as by successive pairwise merges.

22

Related documents

• Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks#1343491
• CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph#1326595
• RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models#411120
• ASA: Backbone-Training-Free Representation Engineering for Tool-Calling Agents#785110
• ASA: Backbone-Training-Free Representation Engineering for Tool-Calling Agents#783520