ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases

arXiv:2611.13246v1 [cs.AI] 9 Nov 2026

Technical report November 2026

ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases Josip Vrdoljak1

Akash Singh1

John Lorenzo1

Bas Steunebrink1

Achille Nicoletti1

Alex Yarotsky1

Sam Pigott1

1 Toptal ExpertCrowd

Abstract ExpertCrowd-SWE (EC-SWE) is a benchmark of 100 expert-authored, multi-file software engineering tasks for evaluating autonomous coding agents in large, unfamiliar codebases. Each task is a change request written by a practising engineer against a pinned commit of a real repository, paired with a sealed reference solution and a task-specific grader whose weighted, mostly executable criteria award partial credit; must-pass blockers and pass/fail gates encode what a maintainer would refuse to merge. The tasks span 87 open-source projects and 11 languages. The median ticket is 148 words, while the median reference solution changes 471 lines. Every task image is calibrated before release so that the reference solution scores 1.0 and both the untouched repository and an inert change score 0; nearly half of the tasks also ship alternative, lazy and adversarial control patches that the grader must classify correctly. The final 100 tasks were selected for difficulty against current frontier models. We evaluate seven frontier models, five runs each (3,495 scored runs), in a single platform harness that gives each model its provider-native tool interface. Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) lead without a statistically significant gap, followed by Claude Fable 5.1 (46.5%), Gemini 3.8 Flash (40.0%), and a statistically indistinguishable group of Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 (32–34%). No task is fully solved by every model, and on 80 tasks no model earns full credit on all five runs. Scores span 35 points. Depending on the model, agents take a median of 33 to 166 steps per run, with little relation to score. We release 24 tasks with their environments, graders and results, and hold out 76 to limit contamination. Dataset

huggingface.co/datasets/ExpertCrowd/ec-swe

Leaderboard toptal.com/expertcrowd/benchmarks/expertcrowd-swe-benchmark

1 Introduction Large language models (LLMs) now act as autonomous software engineers: given a request, they explore an unfamiliar repository, edit across files, run builds and tests, and iterate until the change works. Benchmarks have become the standard way to measure these systems and to decide which of them to trust. They can only

1

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

play that role if their tasks resemble real engineering work, their grading accepts every correct solution and rejects incorrect ones, and their solutions are not already present in the models’ training data. The dominant recipe, introduced by SWE-bench [Jimenez et al., 2024], mines merged fixes from public GitHub repositories and grades each task with the tests that shipped with its fix. The recipe scales, but it has two limitations. First, the fixes, their discussion and their tests are public and plausibly in pretraining corpora, so a high score can reflect recall rather than problem solving [Liang et al., 2025]. Second, inherited tests were written to confirm one particular patch, not to grade arbitrary new solutions, so they can reject valid alternatives or accept incomplete ones. In 2026 both limitations became impossible to ignore. In February, OpenAI stopped reporting SWE-bench Verified after an audit found material issues in the tests or problem statements of 59.4% of 138 audited problems, together with evidence that frontier models had memorized gold patches [OpenAI, 2026a]. In July, a follow-up audit estimated that roughly 30% of SWE-Bench Pro tasks were broken; OpenAI retracted its earlier recommendation of that benchmark and called for new evaluations built with experienced software developers and human oversight throughout [OpenAI, 2026b]. Benchmarks with authored rather than mined tasks address the contamination problem [Huang et al., 2026; Merrill et al., 2026; Kottamasu et al., 2026], but the strongest agents now crowd the top of those leaderboards as well. On the DeepSWE v1.1 leaderboard, Claude Opus 5.5, GPT-6 Astra, DeepSeek V4.1 Flash and Gemini 3.8 Flash sit within 0.9 points of one another (72.3% to 71.4%) [DeepSWE leaderboard, 2026]. A benchmark whose top scores are this close cannot tell configurations apart, however well its tasks are built. What is missing is a benchmark of changes no model has seen solved: written for the benchmark by engineers who know the codebase, never published as fixes, graded the way a maintainer reviews a pull request, and kept mostly private. This paper introduces ExpertCrowd-SWE (EC-SWE), a benchmark of 100 multi-file software engineering tasks authored by practising engineers from Toptal’s ExpertCrowd network. The tasks are built and evaluated on HUD [HUD, 2026], a platform for building and evaluating AI agents: each task ships as a container image with its grader, agents run against it through a hosted runner and inference gateway, and every run is recorded as a full trace (Sec. 3.8). Our contributions are: • Expert-authored change requests in codebases the authors know. Each task is a ticket that an engineer

who knows the codebase well writes against a pinned commit of a real repository. Most tasks are new changes on top of recent public commits; a few build on code the expert added or owns. Authors are forbidden to base a task on a bug that has already been fixed in public (Sec. 3.2–3.3). • Rubric grading with partial credit. Each task has a weighted grader written by its author. Executable

criteria are checked through public interfaces, and must-pass blockers and pass/fail gates enforce conditions a maintainer would insist on, such as leaving the repository’s own tests untouched. LLM-judged criteria play a deliberately small role (Sec. 3.5). • Calibrated, expert-checked graders. A task image only builds if the reference solution scores exactly

1.0 and both the untouched repository and an inert change score 0. Nearly half of the tasks also ship explicit alternative, lazy and adversarial control patches that the grader must classify correctly, and every task is quality-checked by a senior engineer before release (Sec. 3.6). • Contamination controls. Each task’s change is unpublished, 76 of the 100 tasks are held out, and every

task was screened for overlap with 93 public benchmark datasets (Sec. 3.7). • A seven-model evaluation with reliability and effort data. We report 3,495 scored runs of seven frontier

models, with task-level and run-to-run uncertainty, paired comparisons, full-credit rates, and per-run step counts (Sec. 4).

2

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

Scope. EC-SWE measures one capability: given a short ticket, can an agent working autonomously in a sandboxed, offline repository produce a change that an expert maintainer’s grader accepts? All models run through one platform harness that exposes each provider’s native tool interface (Sec. 3.8). The leaderboard therefore compares models under shared task, prompt, sandbox and budget conditions. It is not a ranking of coding products or of overall model quality.

2 Related Work The SWE-bench lineage. SWE-bench framed repository-level evaluation as resolving GitHub issues, with 2,294 instances mined from merged pull requests in twelve Python projects [Jimenez et al., 2024]. SWEbench Verified distilled it to 500 human-screened instances and became the de facto industry leaderboard [OpenAI, 2024]. Later work extended the format to more languages [Zan et al., 2025], to visual software domains [Yang et al., 2025a], and to longer, enterprise-style problems with public, held-out and commercial splits [Deng et al., 2025]. SWE-smith synthesizes task instances at scale for training and evaluation [Yang et al., 2025b]. EC-SWE keeps this lineage’s framing of repository-level, execution-graded tasks, but it authors its tasks rather than mining them and grades them with author-written rubrics rather than inherited tests. Authored, non-mined tasks. A growing line of work writes tasks directly. Aider’s polyglot benchmark curates 225 short exercises [Gauthier, 2024]. Terminal-Bench authors terminal tasks across many domains, software engineering among them [Merrill et al., 2026]. APEX-SWE has experts author integration and observability tasks [Kottamasu et al., 2026]. FrontierSWE curates a small number of very large, open-ended problems graded with partial credit [Chu et al., 2026]. DeepSWE is the closest neighbour: 113 original, never-merged tasks across 91 repositories, graded by hand-written functional verifiers under a single bashonly harness [Huang et al., 2026]. EC-SWE shares DeepSWE’s emphasis on original tasks at repository scale. It differs in three ways: grading is rubric-based with partial credit rather than pass/fail; the authors are domain experts working in codebases they know; and the released set is explicitly selected for difficulty against current frontier models. Contamination and decontamination. Direct probing shows that frontier models reproduce SWE-bench gold patches far more often than matched off-benchmark code [Liang et al., 2025], and OpenAI’s red-teaming found every tested frontier model able to reproduce gold patches or verbatim problem details for some SWE-bench Verified tasks [OpenAI, 2026a]. Defences fall into three families. Continuous collection restricts evaluation to tasks created after a model’s training cutoff [Zhang et al., 2025; Badertdinov et al., 2025; Jain et al., 2025]. Access control holds out private or commercial splits [Deng et al., 2025]. Original authorship keeps reference solutions out of the public record [Huang et al., 2026]. EC-SWE combines original authorship with a held-out split (76 of 100 tasks are private) and an explicit similarity check against 93 public benchmark datasets (Sec. 3.7). Verification and grading. Most mined benchmarks grade with the resolving commit’s FAIL_TO_PASS and PASS_TO_PASS tests [Jimenez et al., 2024; OpenAI, 2024]. Audits find these tests too strict, rejecting valid alternatives, and too loose, accepting incomplete fixes [OpenAI, 2026a,b]. DeepSWE replaces them with functional verifiers written from the task specification and audits them with an independent LLM judge [Huang et al., 2026]. EC-SWE likewise writes its graders from the specification, but it decomposes each task into weighted criteria. Partial credit separates an agent that solves most of a change from one that solves little, and blockers and gates encode the conditions under which a maintainer would reject a change outright. Harnesses and judges. The scaffold between model and repository varies widely, from structured agentcomputer interfaces [Yang et al., 2024] and general agent platforms [Wang et al., 2025] to fixed localize-repair pipelines [Xia et al., 2024] and minimal bash-only loops [SWE-agent contributors, 2025]. Because the same model can score differently across scaffolds, harness choice confounds cross-model comparison. DeepSWE 3

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

removes this confound by giving every model a single bash tool [Huang et al., 2026]. EC-SWE instead runs every model through one platform harness that exposes each provider’s native tool interface, as discussed in Sec. 3.8 and Sec. 5.2. LLM-as-a-judge methods [Zheng et al., 2023] and agent-as-a-judge methods [Zhuge et al., 2024] are increasingly used to grade qualities that tests cannot capture. EC-SWE uses them sparingly, for maintainer-style review criteria in a minority of tasks. Finally, we follow recent calls to report uncertainty from task sampling as well as run-to-run noise [Madaan et al., 2024; Merrill et al., 2026].

3 Methodology EC-SWE is built in five stages. Engineers propose and build tasks in codebases they know (Sec. 3.2–3.4). Each task receives a rubric grader (Sec. 3.5), which is calibrated and reviewed (Sec. 3.6) and checked for contamination (Sec. 3.7). Tasks are then selected for difficulty and evaluated under a fixed protocol (Sec. 3.8–3.9).

3.1 Design principles Mined benchmarks (Sec. 2) grade a change by whether its tests pass [Jimenez et al., 2024; OpenAI, 2024]. Our authoring brief instead asks the question a maintainer asks of a pull request, “would you merge it?”, which takes in correctness, regressions, scope, test quality and maintainability. Every task must be: • Realistic: a change that a real team would ask for, written the way a senior engineer would assign it,

saying what is wanted and never how to do it; • Solvable: the author’s own reference solution proves the task can be done; • Gradable: a machine can tell a right answer from a wrong one; • Reproducible: the same environment gives the same result.

Difficulty must come from reasoning, never from withheld information. We have used four valid sources of difficulty: cross-module synthesis, real trade-offs (going faster costs something the task says must be kept), unwritten but discoverable conventions of the repository, and chained work in which an early decision shapes later steps. We have also specified four invalid ones: facts that appear nowhere in the repository, a single acceptable implementation that the prompt does not signal, a single file that gives the whole answer away, and graders or reference solutions the agent can read. Authors are encouraged to combine two kinds of work, for example performance × error handling, concurrency × logging, migration × backwards compatibility, or refactoring × test quality, because a task that exercises only one is usually too easy.

3.2 Expert recruitment and authoring workflow Task authors are engineers from Toptal’s talent network (more than 30,000 vetted and specialized experts). Authors apply to one of four areas: compilers and language tooling; systems, backend and networking; database and storage systems; and SRE, DevOps and infrastructure. Thirty-nine experts contributed to task authoring. After QC, verification and selection, the final 100 tasks come from 17 of them. Authoring follows a two-stage, draft-first workflow: • Draft. The author submits a short, human-readable draft with three parts: the repository and the exact

commit hash it is pinned to (never a branch name), the prompt written as a ticket, and a grading plan naming the checks and which of them must pass. A reviewer reads the draft and either approves it or sends it back. Nothing is built before approval, and “almost” counts as not approved. Drafts are rejected

4

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

for being unreadable, for prescribing an implementation, for lacking a grading plan or relying only on an AI judge, or for proposing a problem the author has not solved. • Build. After approval, the author builds the task on the HUD coding template (environment, reference

solution and grader), applies their own reference solution, confirms that it scores 100% while an unchanged checkout scores 0%, and runs the task five times against a frontier model (Claude Opus 5, step limit 100). A result of 0% must be checked, since five zero runs also look like a broken grader. AI assistance is permitted for environment wiring but not for the task idea or the reference solution. Recorded reasons for rejecting submitted tasks include reference solutions that were never tested (4), weak, broken or opinion-only graders (3), tasks or repositories that were too small (3), incomplete submissions (2) and one impossible task.

3.3 Task sources Tasks come from four routes, all governed by the same contamination rules: never base a task on a bug that has already been fixed in public, never use employer or client code in any form, keep the task’s own change unpublished, and use proprietary code only when the author owns it and grants the rights. • Expert experience on public repositories. The author proposes a change they have shipped, or wished

someone had, in a public project they know well, pinned to a recent upstream commit. • Long-standing unresolved issues. The task is motivated by an open, unresolved upstream issue, but the

reference solution is the author’s own and does not exist upstream. • Forks with expert-added problems. The author forks a public repository and adds unpublished commits

or a substrate patch on top. The task is then defined against that added code. • Codebases the expert owns. The task is built on a codebase that the author wrote and controls.

In the released set, every task’s base commit could be fetched from the GitHub repository named in its environment. Ninety-eight tasks are new changes on public open-source projects with no added substrate. One task (a PostgreSQL foreign-key migration preflight in SQLAlchemy) adds an expert-authored substrate patch of about 250 lines on top of the public project, and one (a payments webhook pipeline) is built on a codebase owned by its author. Base commits are recent: 82 of the 100 were committed in August or September 2026, with a median date of 4 September 2026, and at least 97 lie on the upstream default branch.

3.4 Task anatomy Each task ships four artifacts: • Prompt. A ticket written by the author, prefixed with a short environment preamble giving the repository

location, resource limits (about two vCPUs and a hard memory limit), the fact that the sandbox is offline, how to start any backing service the repository’s own tests need, and the instruction that hidden tests will grade the work and that existing tests must not be modified (Appendix A). • Environment. A Docker image containing the repository at the pinned commit, with its history vaulted

so that later upstream commits cannot be read, the toolchains, and a pre-populated dependency cache. Agent shells run as an unprivileged user. Everything answer-shaped, including the reference solution, hidden grader, specification files and a pristine checkout, sits under a root-only directory that the agent cannot read. • Reference solution. The author’s own patch (task/gold.diff), sealed inside the image and used only

for calibration and review. 5

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

Table 1. Corpus statistics for the 100 released tasks. Prompt length counts the author-written ticket only, excluding the

shared environment preamble. Changed lines count added plus removed lines in the reference patch. Property

Value

Tasks (public / held-out) Open-source projects / languages Ticket length, words: median (IQR; range) Reference solution, changed lines: median (IQR) Reference solution, files: median (IQR) Agent steps per run: per-model median (range across models) Weighted criteria per task: median (range) Tasks with gates Tasks with an LLM-judged criterion (share of task weight) Base commits dated Aug–Sep 2026

b

Language

7 6

C

5

C++

5

Rust

4

20

Tasks

Java JavaScript

15

25

15

Python

15 10 5

Elixir 1 Ruby 1

0 0

10

5

2

C#

Grader size median 10

median 471

18

TypeScript

c

30

36

Go

Reference solution size

Tasks

a

100 (24 / 76) 87 / 11 148 (69–261; 39–768) 471 (270–808) 4.5 (3–7) 33–166 10 (1–26) 14 10 (2%–25%) 82

20

Tasks

40

0 10

100

1,000

Changed lines (log scale)

0

10

20

Weighted criteria per task

Figure 1. Composition of the 100 tasks. (a) Primary language of each task’s reference change. (b) Size of the reference

solution in changed lines (log scale). (c) Number of weighted criteria in each task’s grader. Green lines mark medians. • Grader. Weighted criteria, blockers and gates (Sec. 3.5), executed hermetically: no network, no wall-

clock dependence and no randomness in any verdict. Where a criterion needs a backing service, such as Redis or a database, the grader starts its own instance inside the image on a private port. Table 1 summarizes the released corpus. Tickets are short (median 148 words) relative to the changes they request: the median reference solution changes 471 lines across 4.5 files, and agents take a median of 33 to 166 steps per run, depending on the model (Sec. 4.6). The released set spans 87 open-source projects and 11 languages, and its tasks run from database engines, query builders and storage layers to schedulers, proxies, authentication servers and compilers (Figure 1).

3.5 Grading Each task’s grader is a list of named criteria with weights that sum to exactly 1.0; this holds for all 100 released tasks. Most criteria are executable checks of observable behaviour through the library’s or service’s own public interface: work enqueued and removed through the public API, documents written and read back, 6

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

queries transpiled and executed. Implementation-internal checks are avoided, so that any reasonable solution can earn full credit. Graders combine three kinds of rule: • Weighted criteria. Partial credit is the weighted sum of criterion scores. • Blockers. Must-pass criteria marked by the author. If a blocker fails, the run’s score is capped below full

credit (for example at 0.60) rather than set to zero, so that a change missing an essential property cannot score well on its other criteria. • Gates. Pass/fail preconditions that carry no weight and cannot be traded against the criteria, for example

“the repository’s own test suite is green” and “the repository’s own tests are untouched”. If any gate fails, the run scores 0. Fourteen released tasks use gates. Formally, for run j of model m on task t, with criterion scores ci ∈ [0, 1], weights wi with ∑i wi = 1, blocker cap κt ∈ (0, 1) and gate set Gt :   0, if any gate in Gt fails;     r(m,t, j) = min ∑i wi ci , κt , otherwise, if a blocker fails; (1)    otherwise. ∑i wi ci , Why partial credit, blockers and gates. Pass/fail grading would compress most of the field into a narrow band. Only 1.0% to 3.8% of the runs of Gemini 3.8 Flash, Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 earn full credit (Table 3), so a binary score would place all four between 1% and 4%. With partial credit they span 32.2% to 40.0%, and Gemini 3.8 Flash is clearly ahead of the other three (Sec. 4.2). Partial credit on its own, however, would let a change that misses an essential property score well on its other criteria; blockers cap such runs below full credit. Gates encode conditions a maintainer would not trade against anything else, such as leaving the repository’s own tests untouched, so a run that violates one scores zero. LLM-judged criteria. These criteria carry between 2% and 25% of their task’s weight, 8.5% on average among the ten, and 0.85% of total benchmark weight. The remaining 90 tasks are graded entirely by deterministic checks. We kept the LLM-judged criteria to under 10% of the total grade due to potential bias and reliability issues.

3.6 Quality assurance and calibration Quality is enforced at build time, through control patches, and through senior-engineer quality control (QC). • Build-time calibration. The image build grades the reference solution, the untouched checkout and an

inert (comment-only) change with the real grader. The build fails unless the reference scores 1.00 with every criterion passing individually, and both the untouched checkout and the inert change score 0.00 with every criterion failing individually. • Control patches. We have also used explicit control patches that the grader must classify correctly. These

are alternative valid solutions that must pass (35 tasks), adversarial or “cheat” patches that must fail, for example ones that edit tests, special-case the graded inputs or disable the feature under test (30 tasks), and lazy or no-op patches that must score low (42 tasks). • Senior-engineer QC. Every task was quality-checked by a senior engineer, who examined its prompt,

environment, reference solution, grader and diagnostic frontier-model runs. Tasks that fell short were returned to their authors for revision, and only tasks that passed QC received the platform status “verified”.

7

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

Table 2. Evaluated configurations. All models run in the HUD harness, which calls the provider API through the HUD

inference gateway and exposes the provider’s native tool interface. Reasoning effort and output-token limits are the platform defaults. Model

Agent / API

Native tools exposed

Reasoning

Max output

Claude Opus 5.5

Anthropic Messages

bash, str_replace_based_edit_tool

effort max

128,000

Claude Fable 5.1

Anthropic Messages

bash, str_replace_based_edit_tool

effort max

128,000

GPT-6 Astra Gemini 3.8 Flash

OpenAI Responses Gemini generateContent

shell (patch-based edits)

effort max thinking high

64,000 65,536

Qwen 3.8 Max

OpenAI-compatible Chat OpenAI-compatible Chat OpenAI-compatible Chat

bash, read, edit, write, grep, glob

effort xhigh

131,072

bash, read, edit, write, grep, glob

effort xhigh

131,072

bash, read, edit, write, grep, glob

effort xhigh

provider default

DeepSeek V4.1 Flash Grok 4.6

run_shell_command, read_file, replace, write_file, grep_search, glob, list_directory

3.7 Contamination analysis The authoring rules keep each task’s own change out of the public record. As an additional check, we compared every task with 499,647 task records from 93 public software engineering benchmark datasets, by matching repository commits and task identifiers and by the semantic similarity of prompts and code. An LLM judge reviewed the closest pairs. No task duplicated a public task or graded the same behaviour as one; a minority touch the same component as a public task with a different change. We also monitor the upstream repositories for changes that overlap a task’s reference solution.

3.8 Evaluation protocol Models and settings. We evaluate seven models (Table 2), each with extended reasoning at the platform’s default high-effort setting and no other agent configuration. All evaluation jobs ran with empty agent parameters, and no system prompt was set. Harness. Every run uses the HUD hosted runner and the HUD SDK (hud-python, v0.6 series; latest release v0.6.21). We chose HUD because it runs every provider’s models through one harness against the same container and grader, enforces the build-time calibration described in Sec. 3.6, and records a full trace of every run for audit. The SDK’s agent loop calls the model’s API directly; no vendor product such as Claude Code, Codex CLI or Gemini CLI is involved. The loop exposes the tool schema each provider’s models are trained on (Table 2), and every tool call executes in the same task container. All models therefore share the task, prompt, container, resources, step budget and grader, and differ only in tool interface and agent loop, which the SDK fixes per provider. This differs from DeepSWE’s choice of a single bash tool for every model [Huang et al., 2026]; we discuss the trade-off in Sec. 5.2. Budget and sampling. Each run is limited to 1,000 agent steps. Each model attempts each task five times. Only the five most recent valid, completed runs per model and task count toward the score. Runs are excluded and rerun if they were invalidated on the platform, failed for infrastructure reasons, were smoke checks, or hit the step limit or output-token limit. Limit-hit exclusions were concentrated in two models: 27 Claude Opus 5.5 runs and 55 Gemini 3.8 Flash runs, plus 2 Grok 4.6 runs. Gemini 3.8 Flash has no valid 8

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

run on one released task and is scored over 99 tasks. The final dataset contains 3,495 scored runs, collected between 1 and 10 October 2026.

3.9 Metrics and uncertainty A run’s score r(m,t, j) ∈ [0, 1] is defined in Eq. (1) (Sec. 3.5). A model’s task score is the mean over its nm,t scored runs on that task (nm,t = 5 for every model and task), and its benchmark score is the macro-average over the Nm tasks it covers (Nm = 100, or 99 for Gemini 3.8 Flash), so that every task counts equally: s(m,t) =

1

nm,t

r(m,t, j), nm,t ∑ j=1

S(m) =

1 s(m,t). Nm ∑ t

(2)

We report three kinds of uncertainty metrics, because they answer different questions: • Task-sampling intervals (headline): S ± 1.96 · SE, where SE is the standard error of the task scores. This

asks how much the score would move on a different but equally reasonable set of tasks, and it is the interval shown on the public leaderboard. • Paired tests for differences between models. Tasks are resampled 2,000 times to give a bootstrap interval

for the mean per-task difference, and a sign-flip permutation test (20,000 permutations, Holm-adjusted across comparisons) gives p-values against the top model. Because both models are scored on the same tasks, paired tests are the appropriate basis for a ranking claim. • Run-to-run intervals, for comparability with DeepSWE and Terminal-Bench [Huang et al., 2026;

Merrill et al., 2026]: the √ benchmark is scored separately on each of the five run slots, and the interval is 1.96 · std(S1 , . . . , S5 )/ 5. This captures only sampling noise in the agent’s behaviour on a fixed task set. We also report reliability measures: the mean of each model’s best run out of five, the share of runs that earn full credit, the number of tasks with at least one full-credit run out of five (an analogue of pass@5), the number with five out of five, and the mean within-task standard deviation of run scores. Steps are agent steps, meaning model turns.

4 Results 4.1 Leaderboard Table 3 and Figure 2 report each model’s mean task score on the 100 released tasks. Claude Opus 5.5 scores 67.3% and GPT-6 Astra 63.7%, followed by Claude Fable 5.1 (46.5%) and Gemini 3.8 Flash (40.0%, 99 tasks). Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 form a lower group between 32.2% and 33.5%. The mean across all seven models is 45.1%. If only each model’s best run out of five counted, the scores would rise to 80.9% for Claude Opus 5.5, 71.4% for GPT-6 Astra and 66.7% for Claude Fable 5.1 (dotted outlines in Figure 2a); the gain is largest for the least consistent models (Sec. 4.4). These are the scores published with the benchmark release. The task-sampling intervals (±3.3 to ±5.5 points) are two to ten times wider than the run-to-run intervals (±0.4 to ±1.9). On a fixed set of 100 tasks, rerunning the agents barely moves the scores; most of the uncertainty comes from which tasks are in the benchmark. Leaderboards that report only run-to-run intervals therefore understate how much a ranking depends on the particular task sample [Madaan et al., 2024].

9

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

Table 3. Main results on the 100 released tasks, 3,495 scored runs. Score is the macro-average of task scores, with a

95% interval over tasks; the run-to-run interval covers agent sampling noise only (Sec. 3.9). “Full-credit runs” is the share of runs scoring 1.0. “≥1 of 5” and “5 of 5” count tasks with at least one, or all five, full-credit runs. “Within-task SD” is the mean standard deviation of run scores within a task, in points. Model

Claude Opus 5.5 GPT-6 Astra Claude Fable 5.1 Gemini 3.8 Flasha Qwen 3.8 Max DeepSeek V4.1 Flash Grok 4.6

Score (95% CI)

Run-to-run CI

Full-credit runs

Tasks ≥1 of 5 / 5 of 5

Within-task SD

67.3 ± 4.7 63.7 ± 5.5 46.5 ± 3.7 40.0 ± 4.3 33.5 ± 3.3 32.5 ± 3.7 32.2 ± 3.5

±1.9 ±1.5 ±0.4 ±1.5 ±1.4 ±1.4 ±1.2

26.6% 23.4% 8.6% 3.8% 1.0% 3.0% 1.0%

47 / 9 36 / 15 25 / 1 9/1 5/0 9/0 4/0

13.6 7.3 16.9 11.0 10.9 13.0 10.1

a Scored over 99 tasks; Gemini 3.8 Flash has no valid run on one released task.

a

Mean task score, n = 100 tasks

vs Opus 5.5 (Holm)

Claude Opus 5.5 67.3

Tasks with full credit

—

GPT-6 Astra 63.7

p = 0.20

Claude Fable 5.1 46.5

p < 0.001

Gemini 3.8 Flash 40.0

p < 0.001

Qwen 3.8 Max 33.5

p < 0.001

DeepSeek V4.1 Flash 32.5

p < 0.001

Grok 4.6 32.2

p < 0.001

0

b

25

50

75

100

Score, % Mean task score (a) · ≥1 of 5 runs (b)

47

9 36

15 25

1 9

1 0

5 9

0 0

0

4

10

20

30

40

50

Tasks (of 100) All 5 runs (b)

Best of five runs (a)

95% CI over tasks

Figure 2. EC-SWE leaderboard. (a) Mean task score on the 100 tasks (solid bars), with 95% intervals over tasks.

Dotted outlines show the mean of each model’s best run out of five. The right-hand column gives Holm-adjusted p-values from paired sign-flip permutation tests against Claude Opus 5.5. (b) Number of tasks on which a model earned full credit on at least one of its five runs (solid) and on all five runs (hatched). Every model was run five times per task with a 1,000-step limit.

4.2 Statistical separation Because all models are scored on the same tasks, paired comparisons are much sharper than overlapping marginal intervals suggest (Table 4). They resolve four tiers. Against the top model, a paired permutation test gives p = 0.20 for GPT-6 Astra and p < 0.001 for every other model (Holm-adjusted; Figure 2a), i.e., Claude Opus 5.5 outperforms those models in a statistically significant manner. The top two models are not separable: Claude Opus 5.5 leads GPT-6 Astra by 3.6 points, but the paired 95% interval [−1.4, +9.7] includes zero, and each model wins a large share of tasks (53 against 41, with 6 ties). Both are clearly ahead of Claude Fable 5.1, which Claude Opus 5.5 outscores on 82 of 100 tasks. Fable is in turn ahead of Gemini 3.8 Flash (+6.5 10

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

Table 4. Paired differences between models on shared tasks. ∆ is the difference in mean task score in points (first model

minus second), with a 95% paired bootstrap interval over tasks (2,000 resamples). W / T / L counts tasks where the first model scores higher, ties, or scores lower. Comparison

∆ (points)

95% paired CI

W/T/L

+3.6 +20.8 +17.2 +6.5 +6.5 +1.0 +0.3 +1.3

[−1.4, +9.7] [+16.9, +25.3] [+11.8, +22.1] [+1.6, +10.5] [+3.0, +11.0] [−2.2, +3.5] [−3.4, +3.8] [−2.5, +4.7]

53 / 6 / 41 82 / 5 / 13 70 / 2 / 28 63 / 3 / 33 50 / 6 / 43 64 / 4 / 32 45 / 6 / 49 53 / 5 / 42

Claude Opus 5.5 − GPT-6 Astra Claude Opus 5.5 − Claude Fable 5.1 GPT-6 Astra − Claude Fable 5.1 Claude Fable 5.1 − Gemini 3.8 Flash Gemini 3.8 Flash − Qwen 3.8 Max Qwen 3.8 Max − DeepSeek V4.1 Flash DeepSeek V4.1 Flash − Grok 4.6 Qwen 3.8 Max − Grok 4.6 Score on every task, mean of five runs

100

Claude Opus 5.5

Claude Fable 5.1

60

Gemini 3.8 Flash 40

Qwen 3.8 Max DeepSeek V4.1 Flash

Task score, %

80

GPT-6 Astra

20

Grok 4.6 0

1

25

50

75

100

Task rank by seven-model average (highest to lowest)

Figure 3. Task-by-model score matrix for the 100 tasks, ordered left to right by seven-model average. Each cell is

the mean of five runs. The empty cell in the Gemini 3.8 Flash row is the one task without a valid Gemini run. Colour swatches identify models consistently across all figures.

[+1.6, +10.5]), and Gemini is ahead of the lower three (+6.5 over Qwen 3.8 Max [+3.0, +11.0]). Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 cannot be distinguished from one another; every pairwise interval includes zero.

4.3 Task-level difficulty and headroom Figure 3 shows every model’s score on every task. The benchmark has substantial headroom. No task is fully solved by all seven models. On 80 tasks, no model earns full credit on all five runs, and the two leading models both score 100% on only four tasks while both falling below 60% on 20. Task averages across the seven models range from 2.5% to 69.5% (median 46.2%), and on 9 tasks even the best model averages below 50%. Of the 699 model–task pairs, only 26 earn full credit on every run and 14 earn zero on every run. Difficulty is far from uniform across models, and the two leaders are complementary rather than ordered. Their task scores correlate only moderately (Pearson r = 0.42), and on 14 tasks they differ by 40 points or more, 8 in Claude Opus 5.5’s favour and 6 in GPT-6 Astra’s. A few tasks are hard for every model. In an OAuth2 Proxy task on session and CSRF cookie accounting, no model averages above 21.8%. In a Harbor quota-ownership task, the best model averages 19.7%. On the 11

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

Distribution of run scores, 3,495 runs 10

Claude Opus 5.5

18

27

13

GPT-6 Astra

19

10

Claude Fable 5.1 8

Gemini 3.8 Flash

12 23

26 14

28 34

9

23

Grok 4.6

10

22

23 16

28

DeepSeek V4.1 Flash

0

13 30

22

Qwen 3.8 Max

27

15 23

36

40

11 19

31

20

9

23

60

12

80

100

Share of scored runs, % Run score 0

(0, 0.2)

[0.2, 0.4)

[0.4, 0.6)

[0.6, 0.8)

[0.8, 1)

1 (full credit)

Figure 4. Distribution of run scores across all scored runs for each model. Segments are labelled with their percentage where space allows.

hardest task, a SystemVerilog export for the CirC compiler, all seven models land between 2.3% and 2.7%.

4.4 Run-level outcomes and reliability Partial credit shows how close agents get when they fall short (Figure 4). Even for the two strongest models, most runs earn at least 0.4 (83% for Claude Opus 5.5, 78% for GPT-6 Astra), but only 26.6% of Claude Opus 5.5 runs and 23.4% of GPT-6 Astra runs earn full credit, against at most 8.6% for every other model. The lower group rarely earns full credit at all (1.0% of Qwen 3.8 Max and Grok 4.6 runs); its typical run collects between a fifth and a half of a task’s weight. The leaders differ in consistency as well as in level. GPT-6 Astra’s runs vary less within a task (mean within-task SD 7.3 points against 13.6 for Claude Opus 5.5). It reaches full credit on all five runs for 15 tasks against Claude Opus 5.5’s 9, while Claude Opus 5.5 reaches full credit at least once on more tasks (47 against 36). A user who can sample several attempts and verify them would favour Claude Opus 5.5’s breadth; a user who needs a single attempt to be right would value GPT-6 Astra’s reliability. Claude Fable 5.1 is the least consistent model (within-task SD 16.9) even though its run-to-run benchmark score is the most stable (±0.4). Its variability averages out across tasks.

4.5 Where credit is lost: blockers and gates Blockers and gates encode the conditions under which a maintainer would reject a change regardless of its other merits. Across all 3,495 scored runs, 274 runs (7.8%) were capped by a failed blocker and 80 runs (2.3%) were zeroed by a failed gate (Table 5). These rules cost the strongest models the most, because their uncapped criteria scores are highest. Without blocker caps and gates, Claude Opus 5.5 would gain 3.7 points and GPT-6 Astra 2.9, while the lower group would gain between 1.1 and 2.9. The ranking would not change. The gates catch behaviour that test-only grading would reward. The clearest case is an Asynq task on how task cancellation interacts with handler outcomes. Claude Opus 5.5 and Grok 4.6 score zero on every run, while DeepSeek V4.1 Flash averages 75.6%.

12

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

Table 5. Credit lost to blockers and gates. “Capped” counts runs whose score was reduced by a failed blocker; “zeroed”

counts runs set to zero by a failed gate. “Mean loss” is the average difference between the uncapped criteria score and the final score for those runs. “Points lost” is the total effect on the benchmark score. Model

Capped runs

Mean loss (capped)

Zeroed runs

Mean loss (zeroed)

Points lost

Claude Opus 5.5 GPT-6 Astra Claude Fable 5.1 Gemini 3.8 Flash Qwen 3.8 Max DeepSeek V4.1 Flash Grok 4.6

62 (12.4%) 42 (8.4%) 37 (7.4%) 31 (6.3%) 29 (5.8%) 24 (4.8%) 49 (9.8%)

0.22 0.26 0.27 0.25 0.21 0.23 0.22

11 (2.2%) 12 (2.4%) 12 (2.4%) 8 (1.6%) 10 (2.0%) 9 (1.8%) 18 (3.6%)

0.43 0.30 0.18 0.00 0.00 0.03 0.21

3.7 2.9 2.4 1.6 1.2 1.1 2.9

Score vs. effort 70

Claude Opus 5.5 GPT-6 Astra

Score, %

60

50 Claude Fable 5.1 Gemini 3.8 Flash

40 Qwen 3.8 Max Grok 4.6

30 0

50

DeepSeek V4.1 Flash

100

150

200

250

Agent steps per run, median (IQR)

Figure 5. Score against median agent steps per run; whiskers show the interquartile range of steps across each model’s

runs.

4.6 Effort Agents take very different routes to their scores (Table 6, Figure 5). The median number of agent steps per run varies five-fold, from 33 for GPT-6 Astra to 166 for Gemini 3.8 Flash, and five of the seven models take a median of 127 steps or more (160 for Claude Opus 5.5). Step count is a poor predictor of success: GPT-6 Astra reaches 95% of Claude Opus 5.5’s score with about a fifth of its steps, and Gemini 3.8 Flash takes the most steps of any model while scoring fourth. Within each model, the correlation between a run’s step count and its score is weak (r between −0.15 and +0.17). Wall-clock time follows yet another pattern: Qwen 3.8 Max has the longest median container lifetime (100 minutes) despite a mid-range step count, and GPT-6 Astra the shortest (36 minutes).

13

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

Table 6. Effort per scored run. Steps are agent steps (model turns). Container time is the wall-clock lifetime of the task

container, including grading. Model

Score

Median steps (IQR)

Mean steps

Median container time

Claude Opus 5.5 GPT-6 Astra Claude Fable 5.1 Gemini 3.8 Flash Qwen 3.8 Max DeepSeek V4.1 Flash Grok 4.6

67.3 63.7 46.5 40.0 33.5 32.5 32.2

160 (128–205) 33 (26–44) 127 (94–166) 166 (127–217) 137 (103–178) 160 (114–199) 73 (55–93)

171 37 134 182 147 162 80

80 min 36 min 56 min 40 min 100 min 51 min 69 min

5 Discussion and Conclusion 5.1 What the results show Expert-authored, rubric-graded tasks separate frontier agents. On EC-SWE, seven current models span 35 points and fall into four statistically distinct tiers. The same models sit much closer together on DeepSWE v1.1, where four of them (Claude Opus 5.5, GPT-6 Astra, DeepSeek V4.1 Flash and Gemini 3.8 Flash) fall within 0.9 points of one another [DeepSWE leaderboard, 2026]; on EC-SWE those four span 34.8 points. The two benchmarks agree on the top two models but diverge below them, most clearly for DeepSeek V4.1 Flash, which ranks third on DeepSWE and sixth here. The benchmarks differ in metric, harness and task population, so this comparison is indicative rather than controlled, but it suggests that EC-SWE resolves differences that existing leaderboards do not. The likely sources are the task design itself (changes that combine two kinds of work in a codebase the agent has never seen), partial credit, which distinguishes “most of the change” from “little of it”, and maintainer-style rules that refuse to reward changes a reviewer would reject. The two leading models are complementary. Claude Opus 5.5 and GPT-6 Astra are statistically tied on average but often disagree on individual tasks: their task scores correlate at only r = 0.42, and on 14 tasks they differ by 40 points or more. They also trade breadth for consistency, and they differ greatly in effort. A single leaderboard number hides all of this. We therefore recommend reading EC-SWE alongside its task-level, reliability and effort views. Effort is not capability. Step count does not track score: the model that takes the fewest steps is the second-best overall, and the one that takes the most scores fourth. For deployments where latency matters, effort should be reported next to accuracy, as recent benchmarks also argue [Huang et al., 2026]. Maintainer rules matter. About one run in ten loses credit to a blocker or gate, and the strongest models lose the most. Some of these losses are exactly the behaviour test-only grading rewards: editing the repository’s own tests to make a suite pass, or shipping a change that breaks a property the ticket said must be kept.

5.2 Limitations • Harness. Each model gets its provider’s native tool interface through the HUD SDK. This is closer to

how the models are trained and deployed than a shared bash tool, but it means the agent loop and tools are not identical across providers. Part of a gap between models could therefore reflect tool design rather than model capability. A controlled comparison against a single shared harness is future work. • Reasoning settings. Each model runs at its platform default high-effort setting (max, xhigh or high,

depending on the provider). We did not sweep reasoning effort, so other settings could change both scores 14

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

and effort. • Sample size. One hundred tasks give task-sampling intervals of ±3.3 to ±5.5 points. Differences smaller

than about 5 points between adjacent models should not be over-interpreted. • Contamination. Task changes are unpublished and their similarity to 93 public benchmark datasets was

checked, but the repositories themselves are public and their code is likely in training data. EC-SWE tests problem solving in familiar-looking code, not in code no model has ever seen. Released public tasks will eventually enter training corpora; the 76 held-out tasks mitigate this.

5.3 Future work We plan to (i) audit graders with an independent judge on a sample of agent runs and publish disagreement rates; (ii) rerun all models under a shared single-tool harness to separate tool effects from model effects; (iii) sweep reasoning effort to map score–effort trade-offs; (iv) grow the corpus through further authoring waves and periodically retire public tasks; and (v) extend the same expert-authored, rubric-graded methodology to other professional domains, starting with management consulting.

5.4 Conclusion EC-SWE asks frontier coding agents to make changes that experienced engineers would ask for, in codebases those engineers know, and grades the results the way a maintainer would: with partial credit for what works, caps for missing essentials, and zero for changes that should never be merged. On 100 such tasks, seven frontier models span 35 points, the top two are statistically tied but complementary, and no task is solved by every model. We release 24 tasks with their environments, graders and results, and hold out 76, so that the community can inspect the method and reproduce the measurement.

Acknowledgements We thank the ExpertCrowd talent, the engineers of Toptal’s ExpertCrowd network who designed, built and verified every task in this benchmark; EC-SWE exists because of their expertise and care. We also thank the HUD team for their contribution to this work, including the evaluation platform, environment templates and tooling on which every task was built and every run was executed. Finally, we thank the maintainers of the open-source projects whose code the tasks build on.

References I. Badertdinov, A. Golubev, M. Nekrashevich, et al. SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents. In NeurIPS Datasets and Benchmarks Track, 2025. M. Chen, J. Tworek, H. Jun, et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021. E. Chu, R. Agarwal, A. Thangamuthu, et al. FrontierSWE. Proximal blog, https://www.frontierswe.com/blog, 2026. DeepSWE leaderboard. DeepSWE v1.1 results for seven frontier models, as reported via Mercor APEX, 2026. Accessed 4 October 2026. X. Deng, J. Da, E. Pan, et al. SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks? arXiv:2509.16941, 2025. ExpertCrowd. ec-swe: public task subset of ExpertCrowd-SWE. Hugging Face dataset, https://huggingface.co/datasets/ ExpertCrowd/ec-swe, 2026a. ExpertCrowd. ExpertCrowd-SWE benchmark. https://www.toptal.com/expertcrowd/benchmarks/expertcrowd-swebenchmark, 2026b.

15

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

P. Gauthier. Aider polyglot benchmark. https://aider.chat/2024/12/21/polyglot.html, 2024. W. Huang, C. Lee, L. Tng, and S. Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. arXiv:2607.07946, 2026. HUD. hud-python: SDK for building and evaluating agent environments (v0.6.21). https://github.com/hud-evals/hudpython, 2026. N. Jain, K. Han, A. Gu, et al. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In ICLR, 2025. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In ICLR, 2024. A. Kottamasu, C. Mahapatra, S. Lee, et al. APEX-SWE. arXiv:2601.08806, 2026. S. Liang, S. Garg, and R. Zilouchian Moghaddam. The SWE-Bench illusion: When state-of-the-art LLMs remember instead of reason. arXiv:2506.12286, 2025. L. Madaan, A. K. Singh, R. Schaeffer, et al. Quantifying variance in evaluation benchmarks. arXiv:2406.10229, 2024. M. A. Merrill, A. G. Shaw, N. Carlini, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv:2601.11868, 2026. OpenAI. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/, 2024. OpenAI. Why we no longer evaluate SWE-bench Verified. https://openai.com/index/why-we-no-longer-evaluate-swebench-verified/, 2026a. 23 February 2026. OpenAI. Separating signal from noise in coding evaluations. https://openai.com/index/separating-signal-from-noisecoding-evaluations/, 2026b. 8 July 2026. SWE-agent contributors. mini-swe-agent: A minimal, model-agnostic SWE-agent harness. https://github.com/SWEagent/mini-swe-agent, 2025. X. Wang, B. Li, Y. Song, et al. OpenHands: An open platform for AI software developers as generalist agents. In ICLR, 2025. C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying LLM-based software engineering agents. arXiv:2407.01489, 2024. J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In NeurIPS, 2024. J. Yang, C. E. Jimenez, A. L. Zhang, et al. SWE-bench Multimodal: Do AI systems generalize to visual software domains? In ICLR, 2025a. J. Yang, K. Lieret, C. E. Jimenez, et al. SWE-smith: Scaling data for software engineering agents. arXiv:2504.21798, 2025b. D. Zan, Z. Huang, W. Liu, et al. Multi-SWE-bench: A multilingual benchmark for issue resolving. arXiv:2504.02605, 2025. L. Zhang, S. He, C. Zhang, et al. SWE-bench goes live! arXiv:2505.23419, 2025. L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS Datasets and Benchmarks Track, 2023. M. Zhuge, C. Zhao, D. Ashley, et al. Agent-as-a-judge: Evaluate agents with agents. arXiv:2410.10934, 2024.

16

ExpertCrowd-SWE

Toptal ExpertCrowd · Technical report

Appendix A Prompt template Every task prompt consists of an environment preamble followed by the author’s ticket. The preamble below is reproduced from one task, with task-specific details elided ([. . . ]). Authors adapt the environment sentences to their repository; the opening and closing instructions follow a common template. You are working in a coding repository located at /workspace/repo. Please make your changes directly in /workspace/repo, not in a copy or a separate worktree. Environment: this container has about 2 vCPUs and a hard memory limit, and a fully parallel build or test run can exhaust it. Build and test package by package, e.g. [...]. Treat the sandbox as offline: the toolchain runs against a pre-populated module cache, so builds and tests work but nothing can be fetched. [...how to start any backing service the repository's own tests need...] Use the tools provided to complete the following task. Hidden tests grade your work when you finish; do not modify existing tests. # Ticket <author-written ticket>

17

Related documents

• PAUSE: Search-Free Steering of Coding Agents Before Their First Source-Code Edit#1414475
• PAUSE: Search-Free Steering of Coding Agents Before Their First Source-Code Edit#1406508
• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?#820620
• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?#477068
• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?#471307