SWE-Time: Benchmarking Temporal Repository Understanding

SWE-Time: Benchmarking Temporal Repository Understanding Shijun Xiao1,3,† , Xinrong Chen2,† , Hao Liu3 , Wensheng Zhang4 , Yongqiang Tang3,∗ 1 Nankai University, Tianjin, China

2 Baidu, Inc., Beijing, China

3 Institute of Automation, Chinese Academy of Sciences, Beijing, China 4 Guangzhou University, Guangzhou, China † Equal contribution. ∗ Corresponding author: yongqiang.tang@ia.ac.cn

Abstract

arXiv:2611.12083v1 [cs.SE] 9 Nov 2026

Reason Backward

Repository-level code-understanding benchmarks have progressively expanded the spatial scope of reasoning, from local code to entire repositories, while largely treating each repository as a static snapshot. However, how models reason about repository evolution over time remains largely unexplored. We introduce SWETime, a benchmark for temporal repository understanding that makes repository evolution itself the reasoning target. SWE-Time evaluates two complementary temporal directions: retrospective understanding of repository evolution through historical regression analysis and prospective understanding of intended changes from pre-change repository states. To enable realistic yet reliable evaluation in these settings, we introduce the RealWorld-to-Verifiable Construction Framework, which transforms realworld developer queries and historical evidence into verifiable evaluation tasks. We evaluate 15 coding agents on 1,661 cases from 65 GitHub repositories. Within Claude Code, Claude Opus 4.8 achieves the highest SWE-Time Avg. of 61.9, leaving substantial headroom. Controlled analyses highlight the importance of historical code evidence and developer intent, while consistent reasoning across repository states remains challenging. These results highlight repository evolution as a distinct and challenging dimension of code understanding.

1

When did this regression first appear? repo/

src/ tests/

Temporal scope

repo/

Δt

src/ tests/

S t−1

Reason Forward What will implementing this request affect? repo/

Δ t+1

src/ tests/

St

S t+1

Repository-level How do components work together?

Cross-file level How do these files interact?

File-level

How does this file work?

Function-level What does this function return?

api.py

utils.py

f()

api.py

utils.py

f()

f()

f()

Spatial scope

Figure 1: Existing code-understanding benchmarks focus on a fixed repository snapshot, whereas SWE-Time evaluates how code evolves through retrospective and prospective reasoning. Spatial scope increases from functions to files, cross-file interactions, and repositories. St−1 , St , and St+1 denote repository states, connected by changes ∆t and ∆t+1 . Dashed future states are hidden from the agent.

2024). Despite their different objectives, these tasks share a common requirement: code understanding (CU), the ability to reason about program behavior and dependencies (Yang et al., 2024; Li et al., 2025; Wang et al., 2026; Lu et al., 2021; Yan et al., 2024; Nguyen et al., 2025). Dedicated CU benchmarks assess this shared capability directly at the repository level. For example, SWE-QAPro and RepoProbe use natural-language questions about repository code (Cai et al., 2026; Yang et al., 2026), whereas RepoReason and RepoReasoner evaluate assertion verification, output prediction, and call-chain prediction (Li et al., 2026; Wang et al., 2026). These benchmarks expand the spatial scope of CU evaluation from individual functions and files

Introduction “Why do we remember the past but not the future?” —Stephen Hawking, A Brief History of Time

Coding has become one of the most important applications of large language models, motivating numerous benchmarks for code generation (Chen et al., 2021; Hendrycks et al., 2021; Liu et al., 2023; Jain et al., 2025; Zhuo et al., 2025), software repair (Jimenez et al., 2024), and repository-level reasoning (Liu et al., 2024; Ding et al., 2023; Li et al., 1

to entire repositories. However, they generally assess code within a fixed repository snapshot. In practice, each snapshot is part of an evolving repository history, as shown in Figure 1. Current behavior reflects past changes, while the effects of a new change depend on the current repository state (Ren et al., 2004). For example, when a feature that previously worked starts failing, developers need to identify which change introduced the regression (Zeller, 1999). Before implementing a request, developers need to anticipate its effects on repository components and behavior, even though the modified code does not yet exist (Sillito et al., 2006; Ko et al., 2007). These problems motivate evaluating both retrospective reasoning about the origins of current behavior and prospective reasoning about the consequences of future changes. To assess these two forms of reasoning, we introduce SWE-Time, a benchmark for temporal repository understanding. Its two tasks instantiate these reasoning directions. In Task Backward, agents trace a reported regression through repository history to identify the first-bad commit. In Task Forward, they reason from developer intent and the pre-change repository state to predict how the requested change would affect repository behavior. They assess whether agents can relate code changes to repository behavior in both directions. Realistic, reproducible evaluation requires authentic developer queries and verifiable reference answers (Just et al., 2014; Widyasari et al., 2020; Gyimesi et al., 2019; Tomassi et al., 2019). Existing CU benchmarks fall into two categories: natural-language QA benchmarks and programmatic benchmarks. QA benchmarks (Cai et al., 2026; Yang et al., 2026) offer developer-oriented questions but often score open-ended answers with LLM judges (Zheng et al., 2023). Programmatic benchmarks (Li et al., 2026; Wang et al., 2026), by contrast, support deterministic scoring but typically do not start from natural-language developer requests. We introduce the RealWorld-to-Verifiable Construction Framework, using independently verifiable historical evidence to identify authentic development events suitable for reliable evaluation. The resulting tasks combine authentic developer queries with deterministic answer verification. Applying this procedure yields 1,661 cases from 65 GitHub repositories. We evaluate 15 coding agents on the resulting benchmark, using Claude Code for the main model comparison and Codex for supplementary references. Claude Opus 4.8

achieves the highest SWE-Time Avg. within Claude Code at 61.9. To examine the sources of difficulty, we conduct controlled experiments on historical code evidence and developer intent, and assess correctness across paired repository states using Pair Exact. These analyses show that both historical evidence and developer intent contribute to performance, while maintaining correct predictions across repository states remains difficult. Our contributions are as follows: • We formulate temporal repository understanding as retrospective and prospective reasoning, and introduce SWE-Time to evaluate both through repository-grounded tasks. • We introduce the RealWorld-to-Verifiable Construction Framework, using independently verifiable historical evidence to identify authentic development events suitable for reliable evaluation. • We evaluate 15 coding agents and use controlled experiments to characterize the roles of historical evidence, developer intent, and repository state in their performance.

2

Related Work

Spatial Repository Understanding. CRUXEval evaluates function-level reasoning through input and output prediction (Gu et al., 2024), while LongCodeU covers structure, data flow, and dependencies in long code contexts (Li et al., 2025). At the repository level, SWE-QA and CoReQA transform information from development activities into questions (Peng et al., 2026; Chen et al., 2025); SWEQA-Pro synthesizes questions from issues and code (Cai et al., 2026); and the CU track of SWECompass generates code-analysis tasks (Xu et al., 2026). RepoProbe draws on GitHub Discussions, with some queries rewritten by an LLM, and uses an LLM judge to score fact-level checklists (Yang et al., 2026). These tasks offer developer-oriented inputs, but query provenance and scoring determinism differ across benchmarks. Programmatic benchmarks instead use fixed targets: RepoReason evaluates assertion verification (Li et al., 2026), and RepoReasoner evaluates output and call-chain prediction (Wang et al., 2026). Their scoring does not require an LLM judge, but their inputs are not authentic natural-language developer requests. These code-understanding tasks primarily assess a given code context. SWE-Time 2

Benchmark

Primary Task

Repo-Level?

Real Developer Code Verifiable? Temporal? Query? Understanding?

CRUXEval LongCodeU SWE-QA CoReQA SWE-QA-Pro SWE-Compass CU RepoProbe RepoReason RepoReasoner LLMBisect AgentSZZ LinkAnchor SWE-EVO

Function-Level Code Reasoning Long-Code Understanding Repository QA Repository QA Repository QA Code Understanding Repository QA Assertion Verification Output / Call-Chain Prediction Bug-Introducing Commit Localization Bug-Introducing Commit Localization Issue–Commit Linking Long-Horizon Software Evolution

✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✗ ✗ △ △ △ △ △ ✗ ✗ ✗ ✗ ✓ ✓

✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓

SWE-Time

Temporal Repository Understanding

✓

✓

✓

✓

✓

Table 1: Comparison with representative code reasoning, repository understanding, and temporal softwareengineering work. ✓/ ✗ indicate whether a work satisfies each property; △ indicates that task queries include LLM synthesis or rewriting of real developer artifacts. The Code Understanding column identifies dedicated comprehension tasks. Sources are discussed in Section 2.

additionally makes changes between repository states part of the evaluation target.

commit-indexed history: H = (C, P, {Sc }c∈C ),

Temporal Software Engineering. SZZ traces bug-fixing changes to historical changes that may have introduced the problem (Sliwerski et al., 2005; Williams and Spacco, 2008; Rosa et al., 2021). LLMBisect and AgentSZZ use LLMs or agents to reason over fixing evidence, code, and commit history to identify bug-inducing commits (Zhang et al., 2026; Lyu et al., 2026). LinkAnchor recovers resolving-commit links from issues and repository history (Akhavan et al., 2026). SWE-EVO uses original release notes and linked developer artifacts to specify long-horizon software evolution tasks, validating resulting code changes with tests (Le et al., 2026). These studies already use software history and involve code and temporal reasoning. SWE-Time differs in its evaluation setting: Backward starts from a sanitized regression report and locates the first-bad boundary without future fixing evidence, while Forward infers workflow consequences from a pre-change issue and repository state. Its outputs directly assess historical localization and change consequences without requiring a repair patch. Table 1 summarizes task, input, scoring, and temporal properties.

3

SWE-Time Benchmark

3.1

Benchmark Overview

Sc = Apply(Spar(c) , δc ),

(1)

where H denotes the repository history, C the commit set, P the Git parent relations, Sc the repository state at commit c, and δc the changes introduced by c. The function par(c) identifies the parent commit, and Apply applies δc to that parent’s state. The state-transition expression applies to single-parent commits, while P also records merge-parent relations in the Git DAG. SWE-Time assesses this ability through retrospective and prospective reasoning, combining authentic developer queries with deterministically verifiable, repository-grounded Gold. Figure 2 summarizes task construction and evaluation. 3.2

Design Philosophy: Repository Evolution as the Evaluation Target

Task Backward. Task Backward is motivated by developers’ need to identify the change that first introduced a reported regression. It evaluates retrospective repository understanding by asking agents to locate the first-bad commit in repository history. The agent receives a sanitized regression report and the repository history available up to cutoff commit τi , and must return the full 40-character SHA of the first-bad commit. We define the Gold commit and its independently verified behavioral boundary as follows: c⋆i = Resolve(efb i , Hτi ),

SWE-Time evaluates temporal repository understanding. We model repository evolution as a

Bi (Spar(c⋆i ) ) = 0, 3

Bi (Sc⋆i ) = 1,

(2)

Find verifiable targets

Validate Gold

Task Backward

a1b2c3d

ValueError

Sₖ₋₁

c* History ≤ cutoff

Bisected to a1b2c3d

Sₜ

Forward cutoff

Δ*

c* Issue I Empty CSV input.

Reject / unresolved

Fixing PR / Δ* + tests/test_csv.py

Issue: empty input

Issue–PR match

Handle empty rows

Sanitized issue

Historical PR if not rows: return []

core.yml

Yes

No

f

{io.yml}

Reject / unresolved

Routing semantics tests

Validation-only probe check(load([]))

Sₖ₋₁

Sₖ

After-cutoff evidence

Frozen evaluator f

Fixing PR / Δ* ~ src/io/loader.py + tests/test_csv.py

Hidden Git history .git/

io.yml, core.yml

f

{io.yml}

input.

src/ io/loader.py .github/workflows/

src/io/parser.py

No

Same Gold?

Admitted Handle empty CSV Pre-change repository Sₜ

Plausible alternative implementations src/io/loader.py

First-bad commit c*

a1b2c3d

cutoff

Unique implementation scope?

io.yml

Repository

with history ≤ cutoff src/ workflows/

Hidden from agent

Pass

CI context E PR synchronize

Gold workflows R* {io.yml} R* = f(Sₜ, Δ*, E)

Admitted

Fail

Human audit

Target branch: main

src/io/loader.py

Bisected to

Workflow answerability

Human review

Workflows in Sₜ

Empty input crashes.

Raises ValueError

Pass Fail

Sanitized report

Sₖ

c*

Returns []

Consistent attribution

Backward

Task Forward

Independent behavior replay Symptom-derived probe check(load([]))

Historical attribution

Empty input now crashes. →

Agent input

Commit correctness

Regression report

load([])

Isolate evidence

Yes

CI context E PR synchronize Target branch: main

Gold workflows R* {io.yml}

Figure 2: RealWorld-to-Verifiable Construction Framework. Task Backward identifies the first-bad commit from a regression report and repository history up to the cutoff. Task Forward predicts activated GitHub Actions workflows from an issue and pre-change repository under a fixed CI context, with the fixing patch hidden. To instantiate these tasks, our framework establishes targets from real developer reports and issue–PR pairs. Backward validates first-bad commits through historical checks and independent behavioral replay; Forward validates workflow targets through an independently tested routing evaluator and human review of plausible implementations. Candidates with divergent Gold or unresolved judgments are excluded. Historical information boundaries separate agent-visible inputs from hidden construction evidence, with packaging checks, network isolation, and human leakage review.

where i indexes cases, Hτi denotes the repository history visible at cutoff commit τi , and efb i denotes developer-provided first-bad or completed bisect evidence. The function Resolve maps this evidence to the Gold commit c⋆i within that history. The states Spar(c⋆i ) and Sc⋆i are immediately before and after that commit. The independently constructed behavioral check Bi returns 0 when the regression symptom is absent and 1 when it is present. Agents may inspect and execute visible historical states and construct their own symptom checks from the report. Developer-provided first-bad evidence and independent Gold-verification checks are used only during construction and remain hidden from the agent. Each case receives a score of 1 if the predicted commit SHA exactly matches the Gold SHA, and 0 otherwise.

fications from developer intent and reason about their workflow consequences. Because reasonable implementations of the same request may yield different workflow sets, we screen candidates for workflow-level ambiguity during construction, as described in Section 3.3. The agent prediction and Gold construction are defined as follows: bi = hθ (Ii , St , Ei ), R i Ri⋆ = f (Sti , ∆⋆i , Ei ),

(3)

bi denotes the predicted workflow set, hθ where R the agent’s prediction process parameterized by θ, Ii the developer issue, and Sti the pre-change repository state at commit ti . The context Ei fixes the event type, activity type, and target branch. The Gold set Ri⋆ is computed by the deterministic workflow-routing function f using the historical fixing PR’s changed-path set ∆⋆i , which is hidden from the agent. The routing function evaluates workflow activation from the configurations in the pre-change state:

Task Forward. Before implementing a requested change, developers need to anticipate its consequences for the repository. Task Forward evaluates this prospective understanding through GitHub Actions workflow prediction. Given a developer issue, the pre-change repository state, and a fixed pullrequest event and branch context, the agent predicts which GitHub Actions workflows an intended but unimplemented change would activate. The historical fixing patch and changed paths remain hidden, requiring agents to infer plausible modi-

Ri⋆ = {w ∈ Wi | Φ(w, ∆⋆i , Ei ) = 1}, Φ(w, ∆, E) = ϕevent ∧ ϕbranch ∧ ϕpath ,

(4)

where Wi = W(Sti ) denotes the workflow configurations in the pre-change state, w an individual workflow, and Φ its activation predicate 4

30%

4.6%

6.2%

11.0%

23.1%

25%

52.8%

65 repos

Developer Events

15.0%

1,661 cases 20.0% 16.6%

16.9%

C / C++ (15) Python (13) Go (11) TypeScript / JS (10) Java (3)

Rust (3) C# (2) Dart (2) Ruby (2) Others (4)

(a) Programming languages

2011–2016 2017–2020 2021–2024 2025 2026

77 183 249 275 877

Samples Smoothed

Samples Smoothed

Median = 363

Median = 4

Median = 28

30%

15% 10% 5% 0%

20%

10%

0% 0

10

100

1k

10k

100k

History window (# commits)

(b) Developer event period

40%

20%

Fraction of cases

15.4%

Programming Languages

Fraction of cases

4.6% 4.6%

Samples Smoothed

Fraction of cases

3.1% 3.1% 3.1%

(c) Backward reasoning space

30%

20%

10%

0% 1

10

100

Changed paths

1k

0

50

100

Workflows in St

(d) Forward reasoning scope

Figure 3: Characteristics of SWE-Time. We show (a) the repository-level programming-language distribution, (b) the temporal distribution of real developer events, (c) the historical search space of Task Backward, and (d) the change and repository-context scope of Task Forward. These statistics summarize the diversity and reasoning scale of SWE-Time without relying on model-dependent measurements.

for a changed-path set ∆ and routing context E. In the second line, the right-hand terms abbreviate ϕevent (w, E), ϕbranch (w, E), and ϕpath (w, ∆), which evaluate event type and action, target-branch filters, and changed-path include/exclude patterns with their matching precedence, respectively; ∧ denotes logical conjunction. The conjunction summarizes the conditions for an eligible event entry; a workflow is selected if at least one eligible entry satisfies all three predicates. Substituting the historical scope ∆⋆i and fixed context Ei yields the Gold set Ri⋆ . Each case receives a score of 1 if the complete predicted workflow set exactly matches the Gold set, and 0 otherwise. 3.3

For Task Forward, we pair issues with reliably linked, merged fixing PRs, recover St from each PR’s base commit, and compute candidate workflow Gold from the historical changed paths ∆∗ under fixed context E using the routing function defined in § 3.2. 3.3.2

Historical evidence supplies candidate targets; admission additionally requires independent validation and screening for plausible alternative answers. For Task Backward, developer localization establishes the first-bad target, while repeated symptomderived tests or probes on the Gold commit and its parent independently verify the reported good-tobad boundary. All admitted cases pass historicalconsistency checks; conflicting, retracted, or unresolved attributions are excluded. These checks support historical localization and behavioral validity without assuming global monotonicity, as detailed in Appendix A. For Task Forward, routing tests with independently specified expected outputs validate event triggers, branch and path filters, and pattern precedence. Correct routing alone does not resolve implementation ambiguity. Experienced developers review scope sufficiency, consistency between the issue and historical PR, and whether reasonable alternative implementations change workflow Gold. Candidates may be admitted with multiple reasonable implementations when the reviewed alternatives yield the same workflow Gold. Candidates with insufficient scope constraints or divergent Gold are rejected; disputed decisions undergo human adjudication, and unresolved cases are excluded. Of 1,899 candidates, 1,030 are accepted,

Construction Framework: RealWorld-to-Verifiable

Real software development activities do not inherently constitute verifiable evaluation tasks (Kalliamvakou et al., 2014). Developer requirements are often open-ended, and their historical implementations may not be the only reasonable solutions. We introduce a RealWorldto-Verifiable Construction Framework that uses independently verifiable historical evidence to identify authentic development events suitable for reliable evaluation. The framework integrates target construction, Gold validation with ambiguity screening, and temporal isolation. 3.3.1

Validating the Gold

Constructing Verifiable Targets

We select events with traceable historical evidence and targets verifiable through repository behavior or native rules. For Task Backward, we use regression reports with explicit first-bad localization evidence, taking the reported symptoms as queries and the attributed commits as candidate Gold. 5

Task Backward Model

Direct Local

Cross

Staged Local

Task Forward Direct

Overall

Cross

Local

Staged

SWE-Time Avg. Overall

Cross

Local

Cross

78.5 61.0 23.8 69.2 54.7 57.6 77.3 73.3 76.2 17.4 75.6 73.3 27.3

56.3 46.5 20.0 52.6 40.5 39.1 61.4 54.4 57.7 19.1 54.9 54.0 25.6

58.0 43.8 15.5 47.3 38.5 43.9 54.3 53.3 60.8 16.0 57.8 51.0 22.7

62.7 49.0 19.4 54.0 43.0 46.2 61.4 58.3 63.9 18.4 61.5 57.0 25.0

61.9 48.5 27.9 54.5 44.0 45.9 59.7 56.0 60.9 30.9 60.0 55.5 24.3

79.7 77.3

59.1 54.4

58.3 58.0

63.4 61.7

62.0 58.7

Claude Code Claude Opus 4.8 Claude Sonnet 4.6 Gemini 3.1 Pro Gemini 3.8 Flash Gemini 3.6 Flash DeepSeek V4 Flash GLM 5.3 GLM 5.2 Qwen 3.8 Max Qwen 3.6 Plus Kimi K3 Kimi K2.6 Hunyuan 3

77.7 63.1 50.8 64.6 59.2 55.4 69.2 63.8 69.2 56.9 73.1 66.9 31.5

63.0 47.4 37.0 58.4 48.0 48.0 57.8 53.2 59.0 45.1 62.4 54.3 25.4

61.0 47.7 39.0 56.4 43.6 48.8 59.3 57.0 58.1 44.8 56.4 55.8 23.8

45.5 36.5 20.5 41.7 31.4 31.4 47.4 42.3 46.8 28.8 44.2 41.0 14.1

61.2 48.0 36.3 55.0 45.0 45.6 58.0 53.7 57.8 43.4 58.5 54.0 23.5

81.1 68.9 37.8 74.3 58.1 58.1 78.4 74.3 77.0 37.8 75.7 74.3 36.5

Codex GPT 5.5 GPT 5.4

73.8 69.2

65.3 59.5

61.0 58.7

43.6 36.5

60.5 55.6

77.0 74.3

Table 2: Performance (%) on Tasks Backward and Forward. The former traces an observed behavior back to its historical cause, while the latter reasons from a change intent to its repository-level consequences. Claude Code is the main evaluation harness; Codex provides supplementary references for harness diversity. We report temporal complexity (Direct / Staged) using task-specific criteria: historical-localization difficulty in Backward and direct routing versus additional change-scope or impact-propagation inference in Forward. We also report spatial scope (Local: local scope; Cross: across files, components, or workflows). Overall aggregates each task, and SWE-Time Avg. averages the two task-level scores.

740 rejected, and 129 excluded as uncertain, as reported in Appendix B. 3.3.3

repository components. Temporal Complexity (Direct/Staged) distinguishes direct target recovery from intermediate historical reasoning in Backward or modification-scope and consequence inference in Forward, as defined in Appendix C. Labels are assigned after admission and frozen before model evaluation. Figure 3 summarizes repository languages, developer-event periods, and task scales.

Temporal Isolation

Both tasks use the complete history to construct Gold, but restrict agent access to information available at the target historical time. Backward exposes repository history up to the report-derived cutoff and a sanitized regression report, with Goldverification probes hidden. Forward exposes only the pre-change working tree and sanitized issue, without Git history or fixing-PR evidence. These boundaries are enforced through input sanitization, automated packaging checks of historicalview integrity and future-evidence exclusion, and runtime network restrictions. Every case also undergoes human review for indirect answer leakage. Detailed procedures are provided in Appendix C. 3.3.4

4

Experiments

We organize our experiments around four research questions. §4.2 establishes the capabilities and limitations of current coding agents on SWE-Time. §4.3 examines dependence on historical code evidence and sensitivity to repositorystate changes. §4.4 examines how developer intent and gold change-scope information contribute to performance. §4.5 relates performance to history-window, changed-path, and workflowcontext sizes.

Benchmark Characteristics

SWE-Time contains 1,661 cases from 65 GitHub repositories: 631 cases from 49 repositories in Backward and 1,030 from 16 in Forward. We characterize the benchmark along two dimensions. Spatial Scope (Local/Cross) distinguishes reasoning within a local component from reasoning across

4.1

Experimental Setup

We evaluate 15 coding agents: 13 under Claude Code (Anthropic, 2026) for the main model comparison, and GPT 5.5 and GPT 5.4 under 6

Codex (OpenAI, 2026) as supplementary references to broaden harness coverage. Within each harness, models share the tool interface, task instructions, and inference-budget policy. Model rankings are reported within harness groups. All evaluations enforce the historical information boundaries in Section 3.3; execution details appear in Appendix C. RQ1 baselines and RQ2–RQ3 use fixed 300-case diagnostic subsets per task. Claude Opus 4.8, the highest-scoring Claude Code agent by SWE-Time Avg., serves as the reference. Controlled conditions reuse the same cases; Overall is case-weighted on each subset. RQ4 analyzes this reference agent’s full-task predictions. 4.2

Task

Baseline / Agent

Overall (%)

Backward

Recent-1 BM25 (commit message) BM25 (diff / paths) History Retrieval + LLM Reranking Claude Opus 4.8 (General Agent)

0.7 2.3 9.0 14.0 62.0

Forward

Repository Prior Claude Opus 4.8 (General Agent)

31.3 63.3

Table 3: Baseline and shortcut analysis.

main substantially below the general agent. On Backward, LLM reranking reaches 14.0%, improving on BM25’s 9.0% but remaining below the agent’s 62.0%. On Forward, Repository Prior reaches 31.3%, compared with 63.3%. These results indicate that retrieval and repository-level priors alone are insufficient for the two tasks. The repository-level robustness analysis reported in Appendix D finds case-weighted and repositorymacro scores within 2.4 percentage points across five Forward configurations, with the ordering of the four Claude Code models unchanged after removing any single repository.

RQ1: Overall Capability

Table 2 reports coding-agent performance on the full SWE-Time benchmark. Claude Opus 4.8 leads the Claude Code group with a SWE-Time Avg. of 61.9, scoring 61.2 on Backward and 62.7 on Forward. Qwen 3.8 Max leads Forward within this harness at 63.9. These results show substantial headroom on SWE-Time. For every agent, Direct cases outperform Staged cases in both tasks, indicating greater difficulty with intermediate historical localization or prospective scope and consequence inference. Local–Cross differences are less consistent. We compare recency, retrieval, and repositoryprior baselines with Claude Opus 4.8 on the same diagnostic cases, as reported in Table 3. For Backward, Recent-1 returns the latest commit in the visible candidate history. The two BM25 baselines (Robertson and Zaragoza, 2009) rank visible commits using either their commit messages or changed paths and unified diffs, returning the topranked commit for the sanitized regression report. Repository Prior evaluates f (St , {p}, E) for every Git-tracked path p and predicts the most frequent complete workflow set, without the issue or ∆∗ . Under the same historical information boundary, our task-specific Backward baseline, History Retrieval + LLM Reranking, retrieves visible candidate commits and uses Claude Opus 4.8 to rerank them against the regression report. Fix-conditioned localization methods such as AgentSZZ and LLMBisect (Lyu et al., 2026; Zhang et al., 2026) require fixing evidence excluded by our historical cutoff, preventing direct evaluation under their original input assumptions. As shown in Table 3, lightweight baselines re-

4.3

RQ2: Temporal Repository Evolution

We examine whether performance depends on repository evolution through two controlled interventions summarized in Table 4. In Backward, we remove historical code states and diffs while preserving the regression report, commit sequence, and metadata. In Forward, we hold developer intent and change scope fixed while placing the same change in a different historical repository state. Task Backward Condition D-L

D-C

S-L

S-C

O

Meta. only 47.8 24.7 23.1 18.1 28.0 Full hist. 82.6+34.8 66.7+42.0 57.7+34.6 41.7+23.6 62.0+34.0 Task Forward Original Paired 63.3

54.3

Inv. Err. ↓

Invalid ↓

Pair Exact

12.8

6.0

48.7

Table 4: Temporal repository evolution. Backward removes historical code while retaining metadata; Forward contrasts historical states with issue, change scope, and routing context fixed. Task scores and paired-state metrics are percentages. Blue subscripts report gains over Metadata-Only.

Overall falls from 62.0 to 28.0, a 34.0-point drop, with declines in all four subsets. Historical code 7

4.5

evidence thus contributes substantially to first-bad localization beyond commit metadata.

RQ4: Difficulty Analysis

As shown in Figure 5, performance decreases as the reasoning space grows. In Backward, performance drops from 65.2 to 57.0 across history-window bins. Forward shows larger declines: 21.0 percentage points across changed-path bins and 31.4 across workflow-context bins. The largest gap occurs with workflow context, suggesting that prospective consequence reasoning becomes particularly difficult as the relevant repository context expands. Overall, larger historical search spaces in Backward and more complex pre-change repository contexts in Forward are associated with lower performance.

Table 4(b) shows that Exact drops from 63.3% to 54.3%, with 48.7% Pair Exact, 12.8% Invariance Error, and 6.0% Invalid. Fewer than half of pairs are solved in both states, showing that prediction changes do not ensure correct adaptation to historical routing transitions.

80

Performance (%)

65.2 70

58.3

57.0

60 50 40

RQ3: Developer-Intent Contribution

L2

L3

[357, Commits [0,73) [73,357) 1.4K)

Using the interventions shown in Figure 4, we examine the contributions of developer-query information and gold change scope while holding repository context fixed. In Backward, symptom ablation reduces Overall from 62.0 to 41.7, a 20.3-point drop, with declines in all four subsets. Regression symptoms thus provide substantial information for first-bad localization.

Changed paths Workflow context

75 50 25 0

L1

4.4

100

History window

62.7

Performance (%)

For Forward, we pair each original state with the nearest valid earlier state in the same first-parent history that changes workflow Gold and reverses at least one shared workflow’s activation. We hold the developer issue, change scope (∆∗ ), and evaluation context (E) fixed, excluding semantically invalid pairs before evaluation. Pair Exact requires correctness in both states; Invariance Error measures unchanged predictions despite different Gold targets among scorable pairs; Invalid denotes pairs containing unscorable outputs.

L4

L1

≥1.4K

Paths [1,2) WFs [3,20)

(a) Task Backward

L2

L3

L4

[2,4) [20,28)

[4,12) [28,36)

≥12 ≥36

(b) Task Forward

Figure 5: Performance across task-complexity factors. We group cases by (a) searchable history window in Task Backward and (b) changed-path scope and workflow-context size in Task Forward. Bars show performance within each bin, and error bars denote 95% Wilson confidence intervals (Wilson, 1927). Tables below the panels report the corresponding bin ranges.

D-L D-C S-L S-C O D-L D-C S-L S-C O

5

No I 50.0 37.5 36.2 34.2 37.3

Conclusion

−Sym. 55.1 43.2 38.5 30.6 41.7 Full I 80.6 77.1 55.2 59.5 63.3

We introduced SWE-Time, extending repository understanding from static snapshots to evolving trajectories. SWE-Time contains 1,661 cases from 65 GitHub repositories, evaluating first-bad localization and prospective prediction of GitHub Actions workflow activation. Our RealWorld-to-Verifiable Construction Framework pairs authentic developer queries with independently validated Gold for deterministic evaluation. Evaluation of 15 coding agents under two harnesses reveals substantial headroom: Claude Opus 4.8 leads the Claude Code group with a SWE-Time Avg. of 61.9. Controlled analyses further show that historical code evidence and developer intent both contribute materially to performance, while maintaining correct workflow predictions across paired repository states remains difficult. We hope SWE-Time provides a foundation for developing coding agents that understand

Full 82.6 66.7 57.7 41.7 62.0 +Δ*

(a) Task Backward

94.4 83.3 83.8 85.6 85.7 (b) Task Forward

Figure 4: Developer-intent contribution. We intervene on the developer query while holding repository context fixed. −Sym. removes the regression symptom, and +∆∗ provides the gold change scope.

In Forward, removing the issue lowers Overall from 63.3 to 37.3; adding gold change scope to Full Issue raises it to 85.7. The 26.0-point Full-vs.-NoIssue gap measures the contribution of developer intent, whereas the additional 22.3-point Oracle gain suggests that change-scope inference is a substantial source of difficulty. This gain is especially large on Staged cases; even with the gold scope, residual errors remain in consequence reasoning. 8

not only repository state, but also repository evolution.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374.

Limitations SWE-Time focuses on two concrete forms of temporal repository understanding: retrospective localization of first-bad commits and prospective prediction of GitHub Actions workflow activation. These tasks capture complementary directions along repository history, but they do not cover the full range of temporal reasoning required in software engineering. In particular, Task Forward evaluates CI routing under a fixed pull-request event context, so generalization to other CI systems, build pipelines, or non-GitHub development environments remains to be studied. The benchmark also relies on repositories with sufficiently recoverable public issue, pull-request, commit, and workflow histories, which may bias the dataset toward projects with richer development records. Because the repositories are public, models may have encountered their code or development records during pretraining. Our experiments cover 15 coding agents under two agent harnesses. Although they span several recent model families, the reported results should not be assumed to characterize future models or alternative agent frameworks.

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. Thomas J. DiCiccio and Bradley Efron. 1996. Bootstrap confidence intervals. Statistical Science, 11(3):189 – 228. Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. B. Efron. 1979. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1):1 – 26. C. A. Field and A. H. Welsh. 2007. Bootstrapping clustered data. Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(3):369–390. GitHub. 2026. Workflow syntax for GitHub Actions. Online documentation. Accessed October 6, 2026. Martin Gruber, Stephan Lukasczyk, Florian Kroiß, and Gordon Fraser. 2021. An empirical study of flaky tests in python. In 14th IEEE Conference on Software Testing, Verification and Validation, ICST 2021, Porto de Galinhas, Brazil, April 12-16, 2021, pages 148– 158. IEEE.

References Arshia Akhavan, Alireza Hoseinpour, Abbas Heydarnoori, Hamid Bagheri, and Mehdi Keshani. 2026. Linkanchor: An autonomous llm-based agent for issue-to-commit link recovery. Proc. ACM Softw. Eng., 3(FSE):4184–4207.

Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. In Fortyfirst International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 16568–16621. PMLR / OpenReview.net.

Anthropic. 2026. Claude Code: Overview. Online documentation. Accessed October 6, 2026. Songcheng Cai, Zhiheng Lyu, Yuansheng Ni, Xiangchao Chen, Baichuan Zhou, Shenzhe Zhu, Yi Lu, Haozhe Wang, Chi Ruan, Benjamin Schneider, Weixu Zhang, Xiang Li, Andy Zheng, Yuyu Zhang, Ping Nie, and Wenhu Chen. 2026. Swe-qa-pro: A representative benchmark and scalable training recipe for repository-level code understanding. In Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, pages 16958–16999. Association for Computational Linguistics.

Péter Gyimesi, Béla Vancsics, Andrea Stocco, Davood Mazinanian, Árpád Beszédes, Rudolf Ferenc, and Ali Mesbah. 2019. Bugsjs: a benchmark of javascript bugs. In 12th IEEE Conference on Software Testing, Validation and Verification, ICST 2019, Xi’an, China, April 22-27, 2019, pages 90–101. IEEE.

Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, Pengfei Gao, Ping Yang, and Shuiguang Deng. 2025. Coreqa: Uncovering potentials of language models in code repository question answering. CoRR, abs/2501.03447.

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with APPS. In Proceedings of the Neural

9

2024. Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, volume ACL 2024 of Findings of ACL, pages 3603–3614. Association for Computational Linguistics.

Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando SolarLezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Jia Li, Yuxin Su, and Michael R. Lyu. 2026. From laboratory to real-world applications: Benchmarking agentic code reasoning at the repository level. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2026, San Diego, California, United States, July 2-7, 2026, pages 8845–8869. Association for Computational Linguistics.

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4j: a database of existing faults to enable controlled testing studies for java programs. In International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014, pages 437–440. ACM.

Tianyang Liu, Canwen Xu, and Julian J. McAuley. 2024. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. Germán, and Daniela E. Damian. 2014. The promises and perils of mining github. In 11th Working Conference on Mining Software Repositories, MSR 2014, Proceedings, May 31 - June 1, 2014, Hyderabad, India, pages 92–101. ACM.

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, and 3 others. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.

Amy J. Ko, Robert DeLine, and Gina Venolia. 2007. Information needs in collocated software development teams. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007, pages 344–353. IEEE Computer Society. Tue Le, Minh V. T. Thai, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. 2026. Swe-evo: Benchmarking coding agents in longhorizon software evolution scenarios. Preprint, arXiv:2512.18470.

Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014, pages 643–653. ACM.

Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Jia Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, and Zhi Jin. 2025. Longcodeu: Benchmarking longcontext language models on long code understanding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 27309–27327. Association for Computational Linguistics.

Yunbo Lyu, Jieke Shi, Hong Jin Kang, Ratnadira Widyasari, Junda He, Yuqing Niu, Chengran Yang, Junkai Chen, Zhou Yang, Julia Lawall, and David Lo. 2026. Agentszz: Teaching the LLM agent to play detective with bug-inducing commits. CoRR, abs/2604.02665. Dung Manh Nguyen, Thang Chau Phan, Nam Le Hai, Tien-Thong Doan, Nam V. Nguyen, Quang Pham, and Nghi D. Q. Bui. 2025. Codemmlu: A multi-task benchmark for assessing code understanding & reasoning capabilities of codellms. In The Thirteenth International Conference on Learning Representations,

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, and Mengfei Yang.

10

ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Lo, and Eng Lieh Ouh. 2020. Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, pages 1556–1560. ACM.

OpenAI. 2026. Codex: Open-source coding agent. Online documentation. Accessed October 6, 2026. Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen, and Xiaodong Gu. 2026. SWE-QA: can language models answer repository-level code questions? In Findings of the Association for Computational Linguistics, ACL 2026, San Diego, California, United States, July 2-7, 2026, pages 8230–8245. Association for Computational Linguistics.

Chadd C. Williams and Jaime Spacco. 2008. SZZ revisited: verifying when changes induce fixes. In Proceedings of the 2008 Workshop on Defects in Large Software Systems, held in conjunction with the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2008), DEFECTS 2008, Seattle, Washington, USA, July 20, 2008, pages 32–36. ACM.

Xiaoxia Ren, Fenil Shah, Frank Tip, Barbara G. Ryder, and Ophelia C. Chesley. 2004. Chianti: a tool for change impact analysis of java programs. In Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2004, October 24-28, 2004, Vancouver, BC, Canada, pages 432–448. ACM.

Edwin B. Wilson. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212. Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Haoyang Huang, Xinping Lei, Yifan Yao, Huaixi Tang, Zhiyi Lai, Kepeng Lei, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Wenqiang Zhu, Wen Xiang, Zongxian Feng, Han Li, Junqi Xiong, Dailin Li, and 19 others. 2026. SWE-compass: Towards unified evaluation of agentic coding abilities for large language models. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research, pages 140878–140902. PMLR.

Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. Giovanni Rosa, Luca Pascarella, Simone Scalabrino, Rosalia Tufano, Gabriele Bavota, Michele Lanza, and Rocco Oliveto. 2021. Evaluating SZZ implementations through a developer-informed oracle. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pages 436–447. IEEE.

Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, and Shuiguang Deng. 2024. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating llms on code understanding and generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 5511–5558. Association for Computational Linguistics.

Jonathan Sillito, Gail C. Murphy, and Kris De Volder. 2006. Questions programmers ask during software evolution tasks. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2006, Portland, Oregon, USA, November 5-11, 2006, pages 23–34. ACM. Jacek Sliwerski, Thomas Zimmermann, and Andreas Zeller. 2005. When do changes induce fixes? ACM SIGSOFT Softw. Eng. Notes, 30(4):1–5.

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

David A. Tomassi, Naji Dmeiri, Yichen Wang, Antara Bhowmick, Yen-Chuan Liu, Premkumar T. Devanbu, Bogdan Vasilescu, and Cindy Rubio-González. 2019. Bugswarm: mining and continuously growing a dataset of reproducible failures and fixes. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pages 339–349. IEEE / ACM.

Yuexi Yang, Alyssa Wu, Ji Luo, Richeng Xuan, Zhichao Hu, Yuhong Liu, and Zhen Qin. 2026. Repoprobe: Benchmarking architecture-aware repository comprehension with checklists. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering, ASE ’26, page 1973–1985, New York, NY, USA. Association for Computing Machinery.

Yanlin Wang, Suiquan Wang, Yanli Wang, Bowen Zhang, Daya Guo, Jiachi Chen, and Zibin Zheng. 2026. Reporeasoner: Evaluating repository-level code reasoning ability of long-context language models. Proc. ACM Softw. Eng., 3(FSE):2790–2812. Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David

Andreas Zeller. 1999. Yesterday, my program worked. today, it does not. why? In Software Engineering -

11

ESEC/FSE’99, 7th European Software Engineering Conference, Held Jointly with the 7th ACM SIGSOFT Symposium on the Foundations of Software Engineering, Toulouse, France, September 1999, Proceedings, volume 1687 of Lecture Notes in Computer Science, pages 253–267. Springer. Zheng Zhang, Haonan Li, Xingyu Li, Hang Zhang, and Zhiyun Qian. 2026. Llmbisect: Breaking barriers in bug bisection with A comparative analysis pipeline. In 33rd Annual Network and Distributed System Security Symposium, NDSS 2026, San Diego, California, USA, February 23-27, 2026. The Internet Society. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Hao-Nan Zhu and Cindy Rubio-González. 2023. On the reproducibility of software defect datasets. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pages 2324–2335. IEEE. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 2 others. 2025. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

12

A

First-Bad Gold Attribution and Validation

Selection stage Accepted by semantic screening Materialized for evaluation Retained after final human review

Backward evaluates whether an agent can trace a reported regression to the commit that first introduced it within the available repository history. Its Gold originates from a developer’s completed historical localization of the reported regression. We subsequently resolve and validate the attributed commit, screen cases for ambiguity and information leakage, and independently check the behavioral transition at the attributed boundary. This appendix describes the construction evidence for the 631 admitted cases from 49 repositories and illustrates why some seemingly relevant reports are excluded. A.1

Cases 666 657 631

Table A.1: Backward candidate selection after semantic screening.

A.2

Historical and Behavioral Validation

Gold validation comprises historical attribution, structural consistency, and independent behavioral confirmation. Historical and structural checks. For each candidate, we verify the identity and location of the attributed commit in the Git history, its relationship to the cutoff, and the consistency of its parent information. Table A.2 reports the results for the admitted set. Candidates with Gold commits that were merge commits, outside the permitted default-branch history, or later than the cutoff were excluded during construction.

First-Bad Gold Attribution

Developer-grounded origin. We discover real GitHub regression reports through high-precision references to historical localization, such as first bad commit, bisected to, regression introduced by, and bisect found. A candidate must contain a completed developer attribution that identifies a specific commit as the first bad revision for the reported symptom. A commit range, a tentative suspect, or an unsubstantiated mention of a SHA is insufficient. Both direct first-bad statements and completed bisect reports are eligible; their differing linguistic formulations do not change the target definition.

Validation check Full, valid 40-character Gold SHA Gold reachable from the historical cutoff Gold is a non-merge commit Parent information consistent across repository checks Cutoff and temporal-consistency validation Final construction validation gate

Resolving the historical target. We associate each attribution with its original issue or comment and resolve abbreviated hashes against the source repository to a unique, full 40-character commit SHA. We recover a historical cutoff at which the regression report is available and verify that the attributed commit belongs to the permitted history. In particular, the target must be reachable from the cutoff and satisfy the benchmark’s default-branch and non-merge selection constraints.

Passed 631 631 631 631 631 631

Table A.2: Recorded structural validation of the 631 admitted Backward cases.

Independent behavioral confirmation. During dataset construction, the 631 admitted cases underwent independent good-to-bad verification in a separate, build-capable execution environment. For each case, the probe was defined from two observable parts of the developer’s regression report: the trigger (reproduction steps, input, or target test) and the reported failure (such as a specific diagnostic, crash, assertion, or failing test). Its pass/fail criterion was fixed for that case before comparing revisions and evaluated through machine-checkable signals, including exit status, test outcomes, crash signals, or diagnostic substrings. A nonzero exit code counted as evidence of the regression only when it corresponded to the reported failure, rather than an unrelated build or environment error (Zhu and Rubio-González, 2023). For the same trigger and symptom-specific pred-

Selection. Semantic screening retained 666 candidates, of which 657 were materialized for evaluation. Final human review retained 631 cases. Table A.1 summarizes these successive selection stages. The resulting cases retain the regression symptom as the developer-facing query but remove the attribution used to establish the Gold. Thus, the agent must recover the historical origin from the report and the repository, rather than read the bisect answer from the input. 13

icate Bi , we classified a revision as bad if the reported failure appeared, and good if the trigger completed without that failure (for example, when the relevant test passed). We applied the same criterion under comparable conditions to the Gold commit c⋆i and its immediate parent par(c⋆i ), verifying the local good-to-bad boundary defined in Equation 2. Each revision was probed repeatedly, and a stable classification required agreement across repeated executions (Luo et al., 2014; Gruber et al., 2021). Inconsistent outcomes were treated as flaky; failures to exercise the trigger or reproduce the specific reported symptom were treated as nonreproducible rather than counted as evidence of a bad revision. These checks used the independent verification environment, not the repository snapshot distributed to agents, which intentionally did not require project build tools. Developer localization establishes the historical first-bad target, while repeated adjacent-state probes independently confirm its behavioral boundary. Historical checks and ambiguity screening corroborate the attribution for the reported symptom.

sanitized report does not retain a revealing SHA, culprit link, bisect outcome, or equivalent answerbearing clue. Sanitization removes such material rather than rewriting the developer’s technical symptom. The historical information boundary and evaluation-package isolation are described separately in Appendix C. Semantic screening uses a five-part rubric covering attribution credibility, authentic regression symptoms, avoidance of answer-derived evidence, absence of answer leakage, and sufficient information. An LLM-assisted pass proposes decisions supported by quotations that are checked against the original source material; Gold SHAs remain grounded in developer attributions. The review process additionally uses an 80-case human calibration sample, explicit human rulings for disputed candidates, and a human review ledger. The final review excluded 26 further cases from the 657 materialized candidates. Admission requires evidence supporting a historical first-bad target for the same regression symptom. The examples below illustrate both accepted attributions and rejected candidates. A.4

A.3 Ambiguity Screening and Human Review

Worked Regression Cases

The cases below are drawn from the construction records. For accepted cases, the developer’s bisect attribution is used to determine the Gold, while the agent-visible query excludes the attribution itself. Rejected cases illustrate where the same construction procedure refuses to turn a historical SHA into a benchmark answer.

A developer’s reference to a commit is not sufficient for admission unless it clearly identifies the origin of the same regression symptom presented to the agent. We therefore screen for three classes of problems. Attribution ambiguity. Candidates are flagged when the issue contains competing culprit commits, a later correction or retraction, an explicitly uncertain assertion, or an attribution that refers to another bug. An unresolved conflict cannot be converted into a unique Gold by choosing the most recent or most frequently mentioned SHA.

A1 • curl-9931

Accept

An explicit bisect across a connection-layer refactor. Input — regression symptom. An HTTP/3/QUIC request to 127.0.0.1:8443 fails with curl: (35) error:0A00010B:SSL routines::wrong version number. The sanitized report retains the connection mode, error, and diagnostic context.

Insufficient developer-facing evidence. After removing answer-bearing material, the remaining query must still describe an observable regression clearly enough to support meaningful historical investigation. Reports consisting only of version metadata, commit lists, or an isolated bisect transcript are not admitted if their actual symptoms or reproduction context are missing.

Historical evidence. A developer explicitly identifies the Gold revision as the first bad commit, according to my bisect. This attribution is removed from the agent-facing report. The searchable history window contains 26 commits.

Historical inconsistency and leakage. We check that the attributed revision is compatible with the historical cutoff and selected branch, and that the 14

Repository transition. The Gold, lib: connection filters (cfilter) addition, is a 49-file connection-layer refactor spanning lib/cfilters.*, connect.c, url.c, and TLS/SSH components. Localization requires relating the connection failure to this change across components.

R1 • curl-17030 Symptom–Attribution Mismatch.

Input — regression symptom. Configuration lines of certain lengths are ignored. Historical evidence. Extraction selected the candidate revision listed below from a bisect discussion of a distinct netrc-parsing problem. The configuration-line regression was attributed to a revision beginning d8618f4d. These attributions concern different symptoms.

Validation and decision. The completed bisect establishes first-bad attribution; historical and structural checks pass, and independent parent/Gold probes confirm the behavioral boundary. Accept: a specific symptom, definite attribution, and valid historical target.

Validation and decision. Reject: the extracted bisect identifies another problem and does not establish a first-bad target for the agent-visible symptom. A valid SHA alone does not resolve this mismatch.

Gold SHA: dafdb20a26d0c890e83dea61a104b75408481ebd. Case ID: curl_curl__9931__85aaefd8. Source: curl/curl #9931.

A2 • flutter-177420

Reject

Candidate SHA: 142ac257b3242459b284020c59f1902b9687a954. Candidate ID: curl_curl__17030__6be98a4c. Source: curl/curl #17030.

Accept

A regression introduced by an engine-roll commit.

R2 • cpython-86867

Input — regression symptom. On an iOS simulator, BackdropFilterLayer allows blur behind a CupertinoSheetRoute to extend beyond its rounded boundary after the animation settles. Sanitization removes the bisect attribution and answer-bearing reference.

Reject

Insufficient symptom after sanitization. Input — regression report. The issue is titled Discrepency between configure.ac and configure. The extracted packet contains migration metadata and bisect information; the concrete ./configure failure appears in a comment absent from the packet.

Historical evidence. Completed developer localization identifies the Gold revision within a searchable history window of 43,902 commits.

Historical evidence. The candidate commit listed below is identifiable, but its attribution overlaps with the limited technical content in the packet.

Repository transition. An automated Flutter Engine roll changes only bin/internal/engine.version, updating the engine consumed by Flutter. The first-bad target is this repository revision, rather than a lower-level engine commit outside its history.

Validation and decision. Removing bisect and commit details leaves an underspecified query; retaining them exposes Gold. Reject: the sanitized packet lacks an independently intelligible regression symptom. Candidate SHA: c6d7e82d19c091af698d4e4b3623648e259843e3. Candidate ID: python_cpython__86867__1bc451ca. Source: python/cpython #86867.

Validation and decision. The completed bisect identifies the repository-level transition, which passes the structural and behavioral checks. Accept: an automated version-roll commit marking the reported regression’s historical boundary.

Together, these cases illustrate the admission boundary: a traceable developer first-bad attribution, a consistent Git revision, a behaviorally relevant transition, and a usable regression report are required for a Backward evaluation instance. Candidates failing attribution consistency or query suf-

Gold SHA: 7bb9352911d0471568a77fc57b94a646fe9571af. Case ID: flutter_flutter__177420__bfb00274. Source: flutter/flutter #177420.

15

ϕevent checks the configured event and activity type against E, including whether types admits synchronize. The branch predicate ϕbranch matches the target branch against branches or branches-ignore. The path predicate ϕpath accepts a paths filter if at least one changed path remains included, and a paths-ignore filter if at least one changed path is not ignored. Absent branch or path filters impose no restriction. Ordered include/exclude patterns preserve their precedence: a later matching negative pattern excludes a branch or path, while a later positive match reincludes it (GitHub, 2026). An event entry activates the workflow only when its event, branch, and path predicates all hold; a workflow is selected if an eligible entry passes. The evaluator also resolves reusable-workflow relationships. The primary evaluation target remains the activated workflow set, rather than the historical PR’s observed execution trace. The frozen evaluator, gold_routing_v4_1. effective_workflows, is checked by a suite with independently hand-specified expected outputs. All 29 workflow-routing tests pass, covering event-action compatibility, target-branch filtering, and path-pattern behavior. This suite independently checks the specified routing semantics. Recomputing each stored Gold from its recorded St , ∆∗ , E reproduces the accepted-case manifest and establishes reproducibility of the recorded labels.

ficiency are excluded rather than assigned speculative Gold labels.

B

Forward Gold Construction and Validation

Task Forward predicts the workflows activated by an intended repository change, given a real developer issue and the repository before its implementation. The historical fixing pull request (PR) supplies a reproducible Gold, but a request may admit multiple reasonable implementations. Deterministic routing and unambiguous task construction are therefore separate requirements. We describe the routing checks, developer review protocol, five worked examples, and independent audits used to assess this distinction. B.1

Gold Construction and Routing Validation

Each candidate originates from a real GitHub issue linked to a merged fixing PR. Within the commitindexed repository history H defined in Section 3.1, we recover the pre-change repository tree St at the fixing PR’s base commit and extract the paths ∆∗ changed by that PR. We omit the case index i in this appendix. Under a fixed routing context E, the activated-workflow Gold is R∗ = f (St , ∆∗ , E),

(B.1)

where R∗ denotes the Gold workflow set, St the pre-change repository state, ∆∗ the historical changed-path set, E the fixed routing context, and f the frozen routing evaluator. Workflow activation follows Equation 4. The benchmark fixes a non-draft pull-request synchronization context, including eligible pull_request and pull_request_target triggers, direct activation, and the historical target branch. The evaluator uses workflow definitions in St , excluding workflow files introduced by the subsequent fix. Gold construction uses neither observed post-change CI executions nor an LLM judge. The agent receives only the sanitized issue I, pre-change state St , and context E, and predicts a workflow set through hθ (I, St , E) as defined in Section 3.2. The historical changed paths, fixing PR, and Gold are hidden from the agent; an explicit file-path prediction is not required. The evaluator implements the activation predicates in Equation 4 from the workflow configurations in St . For each eligible event entry,

B.2

Human Review and Admission Protocol

The admission criterion is workflow-level answerability, rather than uniqueness of a fixing patch. For two developer-validated change scopes under the same historical context, we define routing equivalence as ∆a ∼St ,E ∆b ⇐⇒ Ra = Rb ,

(B.2)

where ∆a and ∆b denote changed-path sets for two reasonable implementations, St the fixed prechange repository state, and E the routing context. The workflow sets Ra = f (St , ∆a , E) and Rb = f (St , ∆b , E) are computed by the same frozen evaluator f ; ∼St ,E means that both scopes produce the same workflow set. The review process assesses whether the issue and pre-change context sufficiently constrain the requested change, whether identified reasonable alternatives yield divergent workflow sets, and whether the historical PR and agent-visible inputs are consistent and free of answer leakage. A candidate can 16

in the repository snapshots. Historical PRs establish actual changes; the alternative scopes shown here were separately reviewed by developers. A diagnostic routing probe tests a rule but need not implement the issue. For rejected examples, outputs are counterfactual evaluations of historical and alternative scopes; these candidates are excluded from the released benchmark.

Box B.1. Forward candidate review instructions. Input. A real developer issue I, pre-change repository tree St , frozen routing context E, and historical PR evidence for provenance and consistency checks. Inspect the agentvisible view separately from evidence reserved for Gold construction. 1. Scope sufficiency. Does the issue, together with St , sufficiently constrain a credible modification direction? Identify the implicated components and supporting evidence. Do not demand a unique patch; assign Reject or Uncertain when the request is too underspecified to establish a defensible target.

Case A — Constrained Modification Scope

2. Workflow-level ambiguity. Propose or inspect reasonable alternative implementations. For each credible changed-path set, evaluate the workflows under the same St and E. Accept code-level variation when the workflow set is unchanged. Reject when a reasonable alternative yields a different target. Record the paths, rationale, and differing outputs; arbitrary paths are not alternative implementations.

A1 • dyninst-2203

Developer request. In dyninst/dyninst, the issue names Failed to transform trace <N> near line 44 of dyninstAPI/src/Relocation/ Transformers/Transformer.C and requests a more informative error message.

3. Historical consistency and leakage. Verify that the fixing PR addresses the issue and that historical evidence supports the case. Check that the agent-visible input does not disclose the future implementation, changed paths, Gold, or other answer-bearing future evidence.

Routing evidence. Historical pr-tests.yaml admits PR actions opened, synchronize, reopened, and ready_for_review on master, with paths **.h, **.C, **.c, **.cmake, **CMakeLists.txt, and tests/*. The other three activated workflows use similarly broad C/C++ patterns.

Decision. Assign Accept, Reject, or Uncertain with supporting evidence. Do not admit unresolved cases. Refer disputed decisions to a third human adjudicator.

be admitted with multiple implementation locations when the reviewed alternatives remain routingequivalent. A reasonable alternative yielding a different workflow set, insufficient evidence, or unresolved uncertainty precludes admission. Box B.1 gives the operational rubric. The screening considered 1,899 Forward candidates: 1,030 were accepted, 740 rejected, and 129 excluded as uncertain. Two senior software engineers independently annotated a separate reliability sample of 200 candidates using the three decision labels. Pre-adjudication agreement was 91.0%, with Cohen’s κ = 0.856 (Cohen, 1960); a third reviewer adjudicated the 18 disagreements. Agreement is calculated from the reviewers’ original labels, before adjudication. B.3

Accept

Reviewed scope

Output

Named Transformer.C statement (historical) Same fix plus a nearby header

4 workflows Same 4

The workflow set is {binary-tests, codegen-cross-build, libabigail, pr-tests}. An unrelated .C file matches the routing filters but does not implement this issue; it is a diagnostic probe. Admission rationale. The report directly identifies the emitting statement, and the reviewed fixes preserve the reproducible workflow Gold. Historical PR #2204. Case ID: dyninst_dyninst__2203__af3e532.

Worked Admission Cases: Inputs, Implementations, and Outputs

A2 • psutil-2639

Accept

Developer request. In giampaolo/psutil, an AIX source-distribution build of psutil v7.1.0 fails because psutil/_psutil_posix.h is missing. The packaging fix must make this header available.

The following five examples make the admission rule concrete. Each specifies the developer-facing issue, relevant excerpts of historical routing rules, changed-path choices, activated workflows, and admission decision. The full configurations remain 17

Routing evidence. build.yml admits push and pull_request without a path filter; bsd.yml matches POSIX source paths; issues.yml admits the fixed PR event. Reviewed scope

Output

psutil/_psutil_posix.h (historical) Same header change plus MANIFEST.in Same header change plus setup.py

3 workflows Same 3

excluding fuzz-smoke; it is not a credible standalone implementation of the framework analyzer. Admission rationale. The reviewed crate locations lie within the allowlist and preserve all 11 workflows; the accepted-case manifest and frozen evaluator agree. Historical PR #2488. Case ID: fallow_rs_fallow__2400__75ca70b.

Same 3 Case C — Divergent Workflow Gold

All reviewed scopes activate {bsd, build, issues}. Admission rationale. The issue identifies a specific missing artifact and packaging objective. The historical fix and reviewed packaging variants are routing-equivalent.

C1 • data-prep-kit-1047

Developer request. “HAP transform crashes when using a GPU” can plausibly require a transform-local fix, shared processing-library fix, or GPU toolchain change.

Historical PR #2640. Case ID: giampaolo_psutil__2639__fb0792e.

Routing evidence. test-universal-hap.yml admits PRs to dev or releases/**, with .make.*, transforms/universal/hap/**, data-processing-lib/**, and !**.md. test-processing-lib.yml also matches .make.* and data-processing-lib/**, without a transform-local requirement.

Case B — Equivalent Workflow Gold B1 • fallow-2400

Accept

Developer request. “Support SvelteKit v3” in fallow-rs/fallow admits core-plugin, extraction-framework, or configuration implementations. Routing evidence. On PRs to main, fuzz-smoke.yml admits crates/extract/**, crates/core/**, crates/config/**, and crates/types/**, among other paths.

Reviewed scope

Output

core/src/plugins/ sveltekit.rs plus integration tests/fixtures (historical) extract/src/frameworks/ sveltekit.rs plus a core test config/src/frameworks.rs plus the core plugin

11 workflows

Output

3 transforms/universal/hap/ dpk_hap/transform.py workflows (historical) data-processing-lib/** 33 workflows .make.versions and 57 .make.defaults workflows

Scopes. Paths below are relative to crates/. Reviewed scope

Reject

The historical set is {test-misc, test-universal-hap, test-universal-hap-kfp}. The shared scope adds test-processing-lib and sibling tests; toolchain changes activate further KFP and packaging workflows.

Same 11 Same 11

Admission rationale. The same request admits reasonable scopes with different workflow sets (3, 33, and 57). The candidate is excluded from the released benchmark.

The common set is {allocs, bench, bloat, ci, commitlint, coupling, coverage, cross-arch, dependabot-auto-merge, ecosystem, fuzz-smoke}. A probe restricted to crates/cli and crates/output activates ten workflows,

Historical PR #1056. Case ID: data_prep_kit_data_prep_kit__1047__3245825.

18

C2 • data-prep-kit-1015

satisfying the full review criteria. The released Forward benchmark therefore retains 1,030 cases, while the original pre-replacement audit outcomes remain reported.

Reject

Developer request. “Improve performance of the Readability transform” follows a slowdown on a large HuggingFace parquet workload. A fix could optimize the local transform or the shared abstract_transform base class.

Subset

Routing evidence. test-language-readability.yml matches transforms/language/readability/** and data-processing-lib/**. test-processing-lib.yml and roughly 30 related test-* workflows also match the shared-library paths. Reviewed scope

Output

transforms/language/ readability/** (historical) data-processing-lib/.../ abstract_transform.py

2 workflows 33 workflows

No Divergent Inconclusive divergence

Direct–Local Direct–Cross Staged–Local Staged–Cross

25 25 25 25

25 24 24 24

0 0 1 0

0 1 0 1

Total

100

97

1

2

Table B.1: Independent audit of previously accepted Task Forward cases.

B.5

Alternative Implementation Audit and Scoring Sensitivity

In the same 100-case audit, developers proposed and reviewed alternative implementation scopes grounded in each issue and its historical repository. 34 cases had at least one developer-validated alternative scope suitable for routing comparison. For 33 cases, the alternative preserved the historical Gold; one produced a different workflow set, yielding 97.1% alternative-scope Gold agreement. The worked cases in Section B.3 illustrate stable and divergent routing. Its two rejected Case C examples are construction-stage candidates, separate from the single divergent case in this independent audit. We also measure scoring sensitivity to reasonable alternative targets. On the original 100-case pre-replacement audit sample, the reference agent obtains 63.0% Overall against historical Gold and 64.0% Overall when validated alternative Gold sets are additionally accepted, a gain of 1.0 percentage point. This analysis holds predictions fixed and varies the accepted Gold targets. The balanced audit describes the examined sample and its validated alternatives; the benchmark’s primary historicalGold scoring protocol remains unchanged. Together, routing validation establishes reproducible labels, human review excludes identified workflow ambiguities, and independent audits detect missed cases and guide corrections before release.

The historical set is {test-language-readability, test-misc}. The shared-library fix additionally activates test-processing-lib and other shared-library-dependent tests. Admission rationale. The request does not determine local versus shared optimization, and the reasonable scopes yield different outputs (2 versus 33). The candidate is excluded from the released benchmark. Historical PR #1026. Case ID: data_prep_kit_data_prep_kit__1015__54161c7.

B.4

Audited

Independent Audit of Accepted Cases

We independently examined 100 previously accepted cases, sampling 25 from each Direct–Local, Direct–Cross, Staged–Local, and Staged–Cross stratum. This audit was separate from the original admission decisions. Reviewers reassessed scope sufficiency and whether reasonable alternative implementations changed the Gold. The audit found 97 cases with no identified Gold divergence, one divergent case, and two inconclusive cases. Among the 98 conclusive decisions, 97 showed no identified divergence (99.0%). Table B.1 reports the stratum-level results. The divergent case and both inconclusive cases were removed and replaced by three newly admitted cases

C

Historical Information Isolation and Reproducibility

SWE-Time constructs verifiable targets from later repository evidence while evaluating agents at an earlier information boundary. Construction and 19

agent execution use separate environments. This appendix describes historical views, leakage controls, evaluation checks, diagnostic labels, and task instructions; Appendices A and B detail Gold attribution and routing validation, respectively. C.1

workflow definitions remain construction-only evidence. The canonical context represents a nondraft pull-request synchronize event, with the target branch resolved at St and provided consistently to both the agent and the routing evaluator. Gold is computed from workflow definitions already present in St .

Task-Specific Historical Views

Backward: cutoff-restricted Git history. Each instance contains a sanitized regression report and Git history ending at a case-specific cutoff derived from the report’s historical context. The pipeline resolves the developer-attributed first-bad SHA, checks ancestry and first-parent branch consistency, and materializes an isolated repository retaining original commit identities and permitted historical states. Under Full History, agents may inspect permitted revisions and diffs, build or test code, and create diagnostic probes. Developer bisect attributions and construction-time good/bad probes remain hidden.

C.2

Packaging and Leakage Checks

Each Backward package has a dedicated evaluation ref. Before admission, the audit checks cutoff availability, Gold membership in the permitted history, inaccessibility of the immediate post-cutoff firstparent probe, and visible-tree consistency with its recorded digest. It also checks for unintended refs and shallow-history artifacts. Packages from the same repository are materialized serially to avoid cross-instance contamination. Forward packages archive the pre-change tree into the execution image without a Git object database. Snapshot integrity is checked against the historical tree used to compute Gold. The canonical changed-path list and routing target remain hidden for construction and grading. Paths naturally mentioned in the issue remain visible even when they overlap with the fix. Both tasks supplement automatic checks with case-level review for indirect answer leakage. The runner uses network/DNS restrictions and proxy overrides to block GitHub and other external sources of future issue, commit, or PR evidence while permitting required evaluation infrastructure connections. Network-isolation checks are recorded with each trial.

Historical metadata and evidence removal. The pre-cutoff candidate history includes history_metadata.jsonl, recording each commit’s SHA, parents, timestamp, author, and message. Both Full History and the Metadata-Only condition in RQ2 expose these fields. Full History additionally exposes historical tree and blob objects for code inspection and diff computation; Metadata-Only removes historical code states and diffs while preserving commit sequence and metadata. In RQ1, BM25 (commit message) uses pre-cutoff messages, whereas BM25 (diff / paths) uses changed paths and patch content, which the Metadata-Only agent cannot access. Reports are sanitized by deletion rather than paraphrasing: culprit SHAs, bisect results, commit or pull-request links, and equivalent answerbearing spans are removed; reproduction steps, error messages, and substantive symptoms remain. Auxiliary repository materials, including release notes and documents identifying the culprit, are also screened in the agent-visible view. Historical code and changes remain available for investigation.

C.3

Agent Execution and Scoring

The benchmark is executed in per-case containerized tasks using Claude Code v2.1.220 and Codex v0.112.0. Both harnesses have a 3,600-second trial timeout; Codex uses reasoning_effort= "high" and override_timeout_sec=3600. Automatic trial retries are disabled (max_retries= 0); coverage backfills use separate invocations of the evaluation runner. Comparisons within the same harness and evaluation condition use common task instructions, tool-access policies, and inference-budget policies. Backward permits repository inspection, Git operations, local execution, and agent-authored tests or probes, while the construction-time regression oracle remains hidden. Expiration without a correct submission is unsuccessful. The Backward grader and answer-

Forward: pre-change working tree. Each instance exposes the developer issue, the repository state St reconstructed from the fixing PR’s base commit, and a fixed evaluation context E. The tree is extracted from the historical repository mirror without a .git directory. The actual fixing PR, its patch, the changed-path set ∆∗ , and subsequent 20

bearing task metadata are kept outside the agent workspace; grading artifacts are installed only after the agent run. Forward requires predictions of activated workflows, without executing prospective fixes or triggering actual CI runs. Scoring is deterministic and independent of the agent’s explanation. Backward accepts a complete 40-character commit SHA and compares it with the stored first-bad Gold. Forward compares predicted workflow-path sets against the frozen routing Gold. Missing or extra workflow identifiers prevent an Exact match. A well-formed empty workflow set is a scorable prediction and is evaluated by the same set comparison. The Forward evaluator resolves event/action eligibility, target-branch filters, and changed-path include/exclude patterns. The event context, evaluator semantics, case selection, and scoring rules are fixed independently of model predictions. C.4

states. Invalid is the percentage of retained pairs containing at least one unscorable output. Goldconstruction artifacts, agent-visible inputs, and hidden scoring data are stored and accessed separately. C.5

Diagnostic Label Definitions

Temporal Low and Temporal High correspond to Direct and Staged, respectively. These labels are assigned independently of model performance and frozen before evaluation. Their task-specific criteria describe the reasoning required to recover the target. Backward. Local/Cross compares the component implicated by the regression symptom with the component implicated by the Gold change: Local denotes the same component, and Cross denotes different components. Direct denotes a clear historical boundary; Staged denotes cases with credible earlier fix-like changes that must be distinguished when localizing the regression origin. This distinction concerns historical localization difficulty; Gold admission separately requires a resolved developer attribution and behavioral validation.

Preflight Validation and Result Aggregation

Before model evaluation, task packages and scorers undergo complementary checks: an oracle submission containing the stored Gold must receive full credit, while malformed submissions and valid submissions that differ from Gold must receive no credit. These tests validate agreement among the packaged task, hidden target, output parser, and scorer. Independent historical and behavioral Gold validation is described in Appendix A; handspecified workflow-routing tests validate evaluator semantics in Appendix B. Each trial is associated with its canonical case identifier. Task and protocol manifests associate the input state, frozen evaluation context, and Gold with the corresponding result. Reported Overall scores are case-weighted Exact over the relevant task set; timeouts, missing submissions, and unscorable outputs receive zero and remain in the denominator. The SWE-Time average gives Backward and Forward equal weight. Controlled experiments reuse fixed diagnostic subsets, whereas full-task analyses use the complete admitted sets. In paired-state analyses, semantically invalid candidate pairs are excluded before evaluation. Pair Exact requires correct predictions in both states and is computed over all retained pairs, with any unscorable output counted as a failure. Invariance Error is the proportion of pairs with unchanged predictions despite different Gold targets, computed only among pairs with scorable outputs in both

Forward. Local denotes a historical changedpath set confined to one top-level repository component; Cross spans at least two such components. Temporal labels are assigned through offline inspection of the pre-change snapshot. Direct corresponds to explicit routing, where the request and repository evidence support direct application of workflow rules. Staged covers scope grounding or impact escalation, requiring further inference of the modification scope or its effects before workflow activation can be determined. C.6

Task Instruction Examples

The following excerpts show the task inputs, evaluation context, and answer format. Long report logs are abbreviated, and the Forward example presents the workflow-level instructions and response fields. Prompt C.1 • Backward

Excerpt

Task. Locate the commit that introduced this regression. Report. Assignment to uint series drops datatype. Assigning a uint value to a slice of a uint series results in a dataframe of type int. With Pandas 0.19.2 the series retains uint64; with version 0.21.0 it becomes int64.

21

snapshot.

What you have. /testbed/repo is a Git clone of pandas-dev/pandas truncated just after the regression was reported. The commit that introduced the regression is an ancestor of HEAD; the later fixing commit is absent. Reading history, gitlog, gitdiff, gitshow, and writing probe scripts support investigation.

Rules. Do not use the network or fetch the upstream fix. Solve the task from the provided repository. C.7

RQ1 baselines and RQ2–RQ3 use a fixed 300-case diagnostic subset for each task. Controlled conditions within a task reuse the same cases. Full Report and Full Issue retain the base instructions; the query interventions below edit the instruction text while reusing the corresponding repository image and task specification.

What to produce. Write the full SHA of the first-bad commit into /testbed/answer.txt: exactly one line of 40 lowercase hexadecimal characters, with no explanation or surrounding punctuation. Abbreviated SHAs are rejected, and the commit must exist in the provided repository. Rules. Do not use the network, rewrite history, add remotes, or fetch objects. Solve the task from the provided repository. Prompt C.2 • Forward

Controlled Input Interventions

Symptom-Ablated. For Backward, symptomdefining spans and title phrasing are deleted from the sanitized report. The remaining report content is retained verbatim, and the available historical search space is unchanged.

Excerpt

No Issue. For Forward, the issue-introduction line and the complete report block are removed. Repository access, routing context, task instructions, and the response format remain the same.

Task. Recover the symbolic verification state this change produces. Report. In giampaolo/psutil, an AIX source-distribution build of psutil v7.1.0 fails because psutil/_psutil_posix.h is missing.

Oracle Scope. For Forward, an Actual change scope block containing the normalized, sorted historical changed-path set ∆∗ is inserted before the repository-access instructions. The issue, pre-change snapshot, and routing context are unchanged. This diagnostic condition supplies change-scope information in addition to the base inputs. The Metadata-Only intervention removes historical code evidence as described in Section C.1. Paired-state contrasts retain the issue, historical change scope, and routing context while replacing the repository state, as described in RQ2.

What you have. /testbed/repo contains the pre-change working tree, without Git history. The PR that actually fixed the issue is absent and cannot be fetched. Evaluation context. Assume the prospective fix has updated an existing non-draft PR: action synchronize, target branch master. Consider pull_request and pull_request_target; apply native types, branches, branches-ignore, paths, and paths-ignore. Consider direct workflow activation, excluding push, workflow_dispatch, schedule, and workflow_run.

D

Repository-Level Robustness

We assess sensitivity to repository weighting and single-repository removal using full-task predictions. Table D.1 reports Task Forward results for four models under Claude Code and GPT 5.4 under Codex as a reference; model comparisons remain within the Claude Code group. Caseweighted Overall averages case-level correctness, whereas Repo-Macro first computes performance within each repository and then weights repositories equally. Leave-one-repository-out (LOO) evaluation removes each of the 16 repositories in

What to produce. Write JSON to /testbed/answer.json, with the case identifier and predicted workflow paths: {"case_id": "<case-id>", "activated_workflows": [ "<workflow-path>" ]}

Workflow paths must use their exact locations under .github/workflows/ in the provided

22

turn and recomputes case-weighted Overall on the remaining cases. Model

CaseRepoweighted Macro

LOO Range

Max. Shift

Qwen 3.8 Max Claude Opus 4.8 GLM 5.2 DeepSeek V4 Flash

63.9 62.7 58.3 46.2

61.5 64.6 58.1 47.8

59.5–71.2 58.1–69.9 53.8–65.0 41.5–51.5

7.3 7.2 6.7 5.3

GPT 5.4

61.7

61.6

56.8–68.7

7.1

Table D.1: Repository-level robustness on Task Forward. Scores and LOO ranges report Overall (%). LOO Range gives the minimum and maximum scores after single-repository removal; Max. Shift gives the largest absolute change from the full-task score in percentage points. GPT 5.4 is a Codex reference; the other models use Claude Code.

The small differences between case-weighted and repository-macro scores indicate that the substantial performance headroom persists under equal repository weighting. Across the four Claude Code models, unchanged ordering in all 16 LOO evaluations further shows that their relative performance is robust to removing any single repository. Absolute scores nevertheless remain sensitive to repository composition, and the Opus–Qwen ordering reverses under repository-macro aggregation. Repository-composition uncertainty. For Claude Opus 4.8, repository-clustered bootstrap (Field and Welsh, 2007; Efron, 1979) gives 95% confidence intervals (CIs) for case-weighted Overall of [55.4, 66.9] for Backward and [49.4, 71.6] for Forward. We use 10,000 resamples and random seed 20261011. Within each task, we draw repositories uniformly with replacement, drawing the original number of repositories (49 for Backward and 16 for Forward). Each selected repository contributes all its case outcomes, with repeated contributions when sampled more than once. We recompute case-weighted Overall from the pooled correct and total counts and take the 2.5th and 97.5th percentiles as the interval endpoints (DiCiccio and Efron, 1996). These intervals quantify uncertainty under repository-level resampling; the wider Forward interval indicates greater sensitivity to repository composition.

23

Related documents

• An Empirical Investigation of Pre-Trained Deep Learning Model Reuse in the Scientific Process#792717
• An Empirical Investigation of Pre-Trained Deep Learning Model Reuse in the Scientific Process#788717
• Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries#1202688
• Idea Traces: Threading Accretive Content across Scientific Papers#1489791
• Data-Driven Simulations to Assess the Impact of Study Imperfections in Time-to-Event Analyses#345329