arXiv CSopen access

What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents

What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents

arXiv:2607.06184v1 [cs.SE] 7 Jul 2026

Rui Shu, Chun Yong Chong, Xin Zhou, Yun Peng, Zihan Wu Xu Han, Zeyang Zhuang, Guowen Yuan, Yuan Wang

Abstract—Coding agents are ranked almost entirely by resolve rate: whether their final patch passes the target tests. Yet two agents can reach the same outcome through very different processes, and a single pass/fail label says nothing about why a run failed or why an accepted run spent extra steps, time, or tokens. This process evidence lives in the trajectory, which records a run’s searches, reads, edits, tool calls, validation, and reversions. However, raw traces are heterogeneous and hard to compare across runs. We present T RACE P ROBE, a trajectory-diagnostic framework that recovers what resolve rate hides. T RACE P ROBE normalizes each raw run into a canonical nine-type action taxonomy with deterministic effect labels, then applies two rule-based modules: I NSIGHT names single-trajectory anti-patterns adapted from established debugging practice (e.g., search loops, verification skips), while C ONVERGE aligns pairs of runs and classifies where their behavior diverges under controlled references. Applying T RACE P ROBE to 2,500 trajectories from five production settings on SWE-Bench Verified, we find that (i) file choice is too coarse to separate success from failure, whereas function selection and completion behavior localize it; (ii) I NSIGHT anti-patterns act mainly as corpus-level difficulty clues, with search loops the most stable; and (iii) even resolved runs differ in how quickly they reach relevant code and how much failed work they incur. Trajectory structure thus adds auditable diagnostic context to outcomes by localizing inspection targets, suggesting failure hypotheses, and prioritizing runs for review. Index Terms—code agents, trajectory analysis, anti-patterns, software engineering, LLM agents

I. I NTRODUCTION LLM-powered coding agents are now widely used in developer environments [1]–[3]. They inspect repositories, edit files, invoke tools, run tests, and iterate on feedback. As these systems move into production workflows, practitioners need to compare not only whether an agent solved a task, but also how it behaved while attempting the task. As the key metric, resolve rate records whether the final patch passes target tests. This outcome is useful for ranking systems and guiding release decisions, but it hides the process differences that matter to agent developers. This limitation appears even when two runs reach the same outcome. Figure 1 shows an example where Claude Code (Opus 4.6) and OpenCode (GLM-5) both resolve the same SWE-Bench task. The pass/fail label treats both runs as successes, but their processes differ. Claude Code reaches a targeted fix with no failed actions, while OpenCode takes a longer path with repeated failures and recovery work. This example illustrates why process evidence is required in addition to Preprint. Under review.

Claude Code

OpenCode

10 steps

49 steps

failed recovery

failed recovery ordering inefficiency off-anchor read failed recovery

Phases:

understand

implement

validate

debug

report

Fig. 1: In this example, Claude Code (Opus 4.6) and OpenCode (GLM-5) both RESOLVE the task (SWE-Bench pytest-7982), but their trajectories differ substantially. Claude Code reaches a targeted fix in 10 steps with no failed actions, while OpenCode takes 49 steps and includes repeated failed/recovery spans. Colored dots show workflow phases, and dashed gray lines show actions aligned by our framework. Red brackets provide informal visual annotations for this example, and our analysis later defines the patterns and divergence signals used for measurement. pass/fail outcomes. Prior work has approached this need from two directions. One direction questions whether pass/fail outcomes are reliable, showing that generated patches can exploit weak tests, recall benchmark artifacts, or be affected by infrastructure and run-to-run noise [4]–[8]. The other direction evaluates or characterizes trajectories directly. AgentBoard scores progress against predefined sub-goals [9], TrajEval grounds failure analysis against reference patches [10], process-centric analysis profiles behavior [11], and Majgaonkar et al. [12] summarize success and failure trajectories with aggregate statistics. These methods show that agent trajectories contain useful evidence. However, they still leave a gap for process diagnosis. Practitioners need deterministic diagnostics that name concrete trajectory patterns, compare runs under

controlled references, and state the oracle assumptions behind each signal. Closing this gap is not trivial because raw trajectories are not directly comparable. A single run interleaves multiple heterogeneous events with no shared schema, including reasoning, tool calls, edits, shell output, and test logs. The relevant signals are also sparse, such as an early read of the wrong file, a skipped validation step, or repeated searches for the same evidence. Aggregate counts lose this local context, while manual inspection does not scale across large trajectory corpora. Process diagnosis therefore requires a representation that preserves local evidence and still supports corpuslevel comparison. On top of this representation, the analysis should support comparison across agent settings, not only case-by-case debugging. It should give practitioners concrete failure clues, point to avoidable effort in resolved runs, and compare agent behavior under controlled references. To be auditable, these signals need deterministic rules, explicit oracle assumptions, and named structural patterns rather than LLM judgments or aggregate statistics alone. This work asks three diagnostic questions: what observable clues are associated with failure, where successful runs spend avoidable effort, and what comparisons among different agents’ runs reveal about behavioral divergences. We present a framework named T RACE P ROBE to answer these questions through a shared representation for within-trajectory diagnosis and between-trajectory comparison. T RACE P ROBE normalizes each raw trajectory into canonical actions from a nine-type taxonomy: file read, file write, search, command, sub-agent spawn, plan, navigate, fetch, and reason. It then assigns each action a deterministic effect label from observable state transitions. Two rule-based modules build on this substrate. The I NSIGHT module profiles one trajectory for recurring failure modes such as search loops and verification skips, while the C ONVERGE module aligns two trajectories and classifies divergences such as off-anchor exploration and rapid rewrite. The structural detectors and most effect labels are oraclefree, while off-anchor labels, milestones, and benchmarkinformed divergence classifications use gold-patch anchors when available. We scope T RACE P ROBE as a diagnostic framework, not a replacement for resolve rate, a per-run root-cause oracle, or a learned policy for fixing agents. It provides auditable signals for comparing process distributions across agent settings, especially when the study design holds the task, scaffold, or model fixed. Within this scope, we apply T RACE P ROBE to 2,500 trajectories from SWE-Bench Verified tasks across five production settings spanning three scaffolds (Claude Code, Codex, OpenCode) and three model backbones (Opus 4.6, GPT-5.4, GLM-5). This design supports both same scaffold model contrasts and same model scaffold contrasts. Our study surfaces three results. First, comparing failed and successful runs on the same task shows where failure evidence should be localized. File choice alone is too coarse, because two successful runs can touch the same files, but clearer signals appear in function selection and completion behavior. Second,

the I NSIGHT module mainly provides corpus-level difficulty clues. Search loops are the most stable clue, while other signals are split-sensitive. A smaller completion evidence signal shows only a modest failed-run gap. Third, the C ONVERGE module milestones and classifications show that successful runs can differ in how quickly they reach relevant code, how much failed work they perform, and how their process changes with the scaffold or model. The main contributions of this paper are threefold: 1) We introduce a deterministic canonical action representation for coding agent trajectories that separates oracle-free structural diagnostics from anchor-grounded milestones and divergence evidence. 2) We define a detector and alignment methodology that turns established SE and debugging anti-pattern concepts into exact trajectory predicates and produces reference-scoped divergence evidence with explicit oracle assumptions. 3) We conduct a multi-setting study showing that trajectories reveal differences between agent settings that resolve rate hides, including where runs diverge, when they reach relevant code, and how much failed work they perform. All data and code are released at https://doi.org/10.5281/ zenodo.20789918 for future study. II. R ELATED W ORK A. Code Agent Evaluation SWE-Bench [13] uses resolve rate as the default metric for coding agents, but a final pass/fail outcome is only a coarse signal. Prior work shows two reasons for caution. First, a generated patch can satisfy permissive tests while missing developer intent [4], [5]. Second, reported scores can be distorted by benchmark memory, infrastructure noise, benchmark saturation, or run-to-run variance [6]–[8], [14]. Recent benchmarks respond by making outcome evaluation broader and more robust. Some benchmarks collect live or contamination-resistant tasks [15], [16], while others target enterprise repairs, multilingual repositories, unified coding agent skills, or security sensitive fixes [17]–[21]. These efforts make final outcomes more informative. Our work asks a complementary question: what process produced that outcome? Unlike AgentBoard [9], which scores progress against predefined sub-goal sequences, T RACE P ROBE normalizes observed agent events and compares trajectories directly without requiring a hidden ideal path. Agent scaffolding systems such as SWEagent, OpenHands, and Agentless [22]–[24] improve codingagent performance, but their evaluations still focus mainly on final outcomes. T RACE P ROBE adds a complementary process view (i.e., a common representation) for asking how agents searched, edited, validated, recovered, and diverged. B. Agent Trajectory Analysis Beyond final-outcome benchmarks, a growing body of work evaluates the trajectories that agents produce. TrajEval [10], TRAJECT-Bench [25], and process-level configuration evaluation [26] support fine-grained trajectory assessment using

reference patches, synthetic reference trajectories, or environment configuration tasks. Process-centric analysis [11] and empirical trajectory characterization [12], [27]–[29] also show that trajectory structure matters. These studies show that trajectories are useful, but they usually summarize them as scores, graphs, profiles, or correlations. Our work instead pairs oracle-free detectors with run-to-run alignment. It maps each raw trace to a shared action format, then applies named detector rules with fixed thresholds. As a result, each measurement is tied to an explicit rule and can be compared across agent settings. T RACE P ROBE targets a different use case than trajectory scoring or LLM judge localization [30], i.e., corpus level comparison of how agent settings behave. For that purpose, C ONVERGE adapts event sequence matching and visual comparison ideas [31] to coding agent traces: it aligns two runs, localizes unmatched or reordered action spans, and reports reference-scoped diagnostics with fixed thresholds. Trajectory analysis is also used for failure attribution, inspection, monitoring, and agent improvement. Failure attribution systems [32]–[37] localize why failed trajectories went wrong, while MAST and silent failure detectors focus on multi-agent failures [38], [39]. Our scope is single-agent SWE coding tasks. T RACE P ROBE compares successful and failed runs and links each diagnostic signal to trace evidence. Adaptive monitors such as TRACE [40] trade repeatability for model-based flexibility, while our work uses fixed rules for repeatable measurement. Because these systems answer different questions, direct accuracy comparisons would not measure the same construct. We therefore use baselines defined on the same trajectories: success/failure outcomes alone and raw telemetry such as steps, tokens, and tool counts. Inspection and visualization systems help users examine traces directly [41]– [44]. Other work uses trajectories to improve agents during or after execution, for example through reward models, course correction, trajectory reuse, harness evolution, or richer trace representations [45]–[52]. T RACE P ROBE’s core contribution is a reusable detector catalog for real SWE trajectories, which helps practitioners apply the same diagnostic rules across agent settings and compare process differences with trace level evidence. C. Anti-Pattern Detection in Software Engineering Software engineering research has long used named patterns to describe code smells [53], design anti-patterns [54], organizational debt [55], socio-technical anti-patterns in ML-enabled software [56], and CI/CD configuration issues [57]. Many of those settings scan static artifacts, but agent trajectories are dynamic event sequences whose shape depends on scaffold, prompt, and tool interface. Our work brings this anti-pattern tradition to trajectory analysis by using named signals for recurring agent behavior. It maps raw events to canonical actions with deterministic effect labels, then defines structural detectors over that representation. For example, analysis paralysis and thrashing [54] motivate search-loop and re-readchurn detectors. Prior trajectory characterization and failure taxonomy studies identify important behavior patterns [58].

T RACE P ROBE turns such patterns into a detector catalog that can be tested by outcome group, reused across agent settings, and used to explain what changed in an agent’s process. III. T RACE P ROBE : O FFLINE T RAJECTORY A NALYSIS A. Architecture Overview At a high level, Figure 2 shows how T RACE P ROBE separates representation construction from diagnostic analysis. Before describing the modules, we define six terms used throughout the methodology. A reference is another trajectory chosen for comparison. An anchor set is the set of taskrelevant files supplied by a benchmark patch when available. A canonical action is a typed event record, and an effect label records its observable outcome. A divergence span is a contiguous block of actions that remains unmatched or reordered after C ONVERGE aligns two runs. A milestone is the first step that reaches a task-relevant artifact, and an I NSIGHT detector flags a named pattern in one trajectory. T RACE P ROBE first converts raw trajectories into normalized steps, typed canonical actions, and deterministic effect labels. I NSIGHT then analyzes one run without a reference or anchor set. C ONVERGE compares a run with a reference trajectory and labels its divergence spans. Milestones use the anchor set to record when a run first reaches task-relevant artifacts. B. Shared Representation To compare trajectories across scaffolds, T RACE P ROBE first builds a shared deterministic representation. The representation starts with normalized steps that preserve event context in a common schema. Each step is then converted into a canonical action, which records the action type and comparable target. On top of this action sequence, T RACE P ROBE derives workflow phases to summarize coarse progress and effect labels to record observable action outcomes. This process starts with scaffold adapters. They translate scaffold-specific traces into a common step schema that preserves role, typed tool-call records, resource use, timing, and error status. This layer absorbs format differences across agents. For example, Claude Code may split one API response across multiple trajectory entries, while OpenCode stores comparable information inline. T RACE P ROBE then maps each normalized step to a canonical action with an action type and target. The action type uses the nine-label set in Table I to distinguish the kind of work performed by the step. The target stores the comparable object for the action, such as a path, command, query, scope, or plan summary. File targets use repo-relative paths, allowing C ONVERGE to align the same artifact across different local work directories. T RACE P ROBE also derives a deterministic workflow phase from observed step fields. The classifier assigns the phase from the step’s dominant event, separating planning, implementation, validation, debugging, repository inspection, and reporting activity. These phases support phase-level profiles and provide the phase set used by semantic tags. To decide whether a read, search, or milestone touched taskrelevant code, T RACE P ROBE needs a task-relevance set. When

INPUTS

Raw trajectories

SHARED REPRESENTATION

Normalize

Insight · single-trajectory

scaffold adapters → steps

structural + semantic detectors + per-run trajectory profiles

Claude Code · Codex · OpenCode heterogeneous JSON traces

Canonical actions type · target · effect

trace / data flow

DIAGNOSIS MODULES

auxiliary LLM tags phase + action tags semantic detectors

Converge · cross-trajectory

gold-patch changed files → enclosing functions

Divergence

LCS alignment to a reference divergence patterns file-sel · edit-stab · completion

Effect labels Anchor set · oracle

Anti-pattern per-run failure signals

Semantic labeler

anchor (oracle) grounding auxiliary semantic (LLM)

OUTPUTS

layer verdicts reference-scoped

observable outcomes

Typed action sequence

Milestones · oracle-grounded

shared, frozen input to all analyses

M1 read · M2 write · M3 all M4 validate · M5 justified

Progress timing milestone reach times

Fig. 2: T RACE P ROBE architecture. Raw trajectories are normalized into a shared canonical representation, then analyzed by I NSIGHT for single-run patterns and by C ONVERGE for comparison against another run chosen as the reference trajectory. TABLE I: Step-level label schema in T RACE P ROBE. Deterministic canonical actions, effect labels, and workflow phases support the main analyses, while optional LLM semantic tags are used only by detectors marked semantic. Layer

Label Set

Role in Analysis

Deterministic layers Canonical Action

FILE READ, FILE WRITE, SEARCH, COMMAND, PLAN, NAVIGATE, FETCH, AGENT SPAWN, REASON

Effect Label

SURVIVED, FAILED, REVERTED, JUSTIFIED, RECORDED, OFF-ANCHOR, REASONING

Workflow Phase

UNDERSTAND, PLAN, IMPLEMENT, DEBUG, VALIDATE, REPORT

Deterministic event sequence used by I NSIGHT, C ONVERGE, and process-profile summaries. Observable action outcome used in detector predicates and process-cost summaries. Coarse workflow stage used for phase-level profiles. Semantic tags use the same phase set.

Semantic layer: LLM-based fine-grained action tags, grouped by their phase Understand Plan Implement Debug Validate Report

spec intake, scope clarification, constraint extraction, tool discovery, delegated research, external research, file discovery, code reading, context synthesis plan management, task breakdown, sequencing, dependency mapping, risk planning implement api schema, implement runtime logic, implement tests, implement generated artifacts, implement config, implement docs, implement refactor, implement migration debug reproduction, debug root cause, debug hypothesis test, debug fix selection validation run, validation review, validation lint, validation build, validation unit tests, validation integration tests, validation e2e tests, validation regression, validation performance progress update, final reporting, change summary, blockers risks, next steps handoff

a benchmark supplies a ground-truth patch, as in SWE-Bench Verified, the modified files define the trajectory’s anchor set, or gold-patch file set. T RACE P ROBE uses these anchors for offanchor labels, milestone extraction, and trajectory comparisons that refer to task-relevant artifacts. An off-anchor label marks a successful read or search outside the anchor set. For nonbenchmark traces, T RACE P ROBE can instead derive task relevance from surviving writes and test/import references. The detector catalog uses only trajectory events, not the SWEBench issue description. As a result, same-task comparisons are driven by what agents did rather than by prompt text that is identical across agents. After action typing, T RACE P ROBE assigns each action a deterministic effect label using only observed trajectory data, not an LLM judge. A write is survived when it persists to the final state, and reverted when a later write to the same file overwrites it. An action is justified when it reads or searches a task-relevant file, or runs a validation command. A non-

Fine-grained tags used only by detectors marked semantic.

workspace meta-action, such as a plan update, navigation, fetch, or sub-agent spawn, is recorded when it succeeds. When reporting effect percentages, T RACE P ROBE excludes reasoning-only steps because they do not change files or return command status. These labels describe what happened during the run, not whether the action was correct. File-level anchors can still be too coarse when two runs touch the same file but edit different code regions. For function-level analysis, T RACE P ROBE deterministically maps each patch hunk to the most-specific enclosing def or class, using the hunk header and an indentation-aware scan without an LLM. When the resolver cannot identify an enclosing function, the hunk remains a valid file-level anchor. The semantic layer is separate from this deterministic core. For detectors marked semantic, the step labeler prompts an LLM to choose one phase/action pair from the closed taxonomy in Table I. The parser validates each response and derives the phase from a valid action when needed. Because these tags are produced by an LLM, they may change if the labeling

model, prompt, or provider behavior changes. We therefore use them only for detectors marked semantic and for exploratory analyses, not for the main deterministic claims. C. Anti-Patterns: Single-Trajectory Diagnosis In I NSIGHT, an anti-pattern detector is a deterministic predicate over one trajectory. If the predicate is satisfied, I NSIGHT reports a named pattern together with the local steps that support it. To keep this evidence auditable, the reported catalog is intentionally selective rather than exhaustive. A detector is included only when its triggering condition can be stated exactly, its inputs are observable tool-event features, and its numeric threshold is frozen before evaluation. Table II gives the exact predicates, thresholds, and inputs that support claims in this work. The first block contains the single-trajectory detectors used by I NSIGHT. For traceability, the same table also lists the frozen cross-trajectory patterns and milestone definitions introduced in the next subsections. Structural detectors support the main single-trajectory claims, while semantic detectors remain exploratory. Ambiguous patterns such as rewrites move to cross-trajectory comparison, where a reference can disambiguate them. D. Divergence Patterns: Cross-Trajectory Diagnosis Some process patterns are ambiguous without another run for comparison. C ONVERGE handles these cases by asking a reference-scoped question: how does one agent deviate from a reference process? The study design selects the reference trajectory, for example a resolved run for the same task or a contrast that holds either the scaffold or the model fixed. The compared trajectory is the run being diagnosed. C ONVERGE aligns the two trajectories and classifies the unmatched or reordered regions as named forms of avoidable effort, instability, or drift. C ONVERGE begins with a monotonic longest-commonsubsequence (LCS) alignment between a reference R and compared trajectory C. Two actions can match only when their action types, targets, and effect labels are compatible. Compatibility is action-specific. File actions compare repo-relative paths, while non-file actions use normalized search scope, command intent, or task intent. Effect labels are matched conservatively: successful workspace effects can match each other, failure-like and non-workspace effects require the same label, and off-anchor actions are treated permissively so uncertain exploration does not inflate divergence. These rules preserve process differences such as wrong-file work, reversed writes, and failed actions where the reference succeeded, while avoiding over-counting equivalent local choices. From the matched pairs and unmatched actions, C ONVERGE derives coverage, cost, and ordering metrics that summarize omitted reference behavior, added work, and out-of-order matches. This alignment is deterministic and runs in O(nm). To check whether classifications depend on the unit-cost LCS choice, we also compare against a weighted Needleman– Wunsch variant. After alignment, C ONVERGE groups unmatched or reordered actions into contiguous divergence

spans. Each span is scored by a few observable metrics, such as off-anchor ratio, reversed or rapid rewrites, and deadend or out-of-order counts. Each metric has frozen good and warn thresholds. A value past the warn threshold is high-tier evidence, and a value in the good-to-warn band is medium-tier. C ONVERGE summarizes evidence from divergence spans in three diagnostic layers: file selection, edit stability, and completion behavior. The file-selection layer asks whether the compared run explored or changed a different part of the repository from the reference. The edit-stability layer captures unstable rewrites and repeated failed actions, including error-spiral counts from the structural detector catalog. The completion-behavior layer captures unresolved dead ends and actions that reach the same targets in an inefficient order. Within each layer, C ONVERGE combines pattern counts with evidence tiers and reports a strong, moderate, or weak classification. These labels describe agreement with the reference in that layer, not absolute run quality. A strong classification means there is no medium- or high-tier divergence evidence. A moderate classification means medium-tier evidence, and a weak classification means high-tier evidence or multiple medium-tier patterns. All classification thresholds are fixed before the reported analyses, and every classification remains relative to the chosen reference. To assess reference dependence, we repeat the analysis with a per-task most-efficient reference, defined as the resolved candidate reference with the fewest trajectory steps. E. Task-Relevant Progress Milestones Anti-patterns and divergence patterns describe what a trajectory contains and how it differs from another run. Milestones add a temporal view by recording when the agent first makes task-relevant progress. A milestone is the earliest step at which a specified event occurs with respect to the benchmark’s anchor set. T RACE P ROBE extracts five milestones from each trajectory: M1 first read of an anchor file, M2 first write to an anchor file, M3 all anchor files written, M4 first passing validation command, M5 first justified action. Each milestone yields a (reached, step-to-first-reach) pair, and trajectories that never reach the event are right-censored and reported separately. Because milestones require the anchor set, they are oracle-grounded. They still operate on one trajectory at a time, so the empirical study uses them as a temporal process summary alongside anti-patterns and divergence patterns. IV. E MPIRICAL S ETUP A. Corpora and Collection Protocol We collect the trajectory corpus from all 500 SWE-Bench Verified [13], [59] tasks. Each task is run under five production settings. We define a setting as a scaffold plus a model: Claude Code + Opus 4.6, Codex + GPT-5.4, and OpenCode with Opus 4.6, GPT-5.4, and GLM-5. This yields 2,500 trajectories in total. This design supports two contrast families. The three OpenCode variants form same-scaffold contrasts where the model changes. Claude Code vs. OpenCode on Opus and Codex vs. OpenCode on GPT-5.4 form same-model

TABLE II: Measurement catalog used in the reported analyses. Each row gives a named detector, its frozen predicate, the required inputs, and how the signal is used in the results. Thresholds are fixed before the stability, sensitivity, and transfer checks. Detector

Frozen Predicate

Input and Condition

Use in Results

Canonical action types and command classifier; no oracle.

Primary failure-associated clue; stable under post-freeze checks.

Structural single-trajectory detectors (deterministic; no semantic labeler) Search Loop

At least 10 consecutive SEARCH or FILE READ actions with no FILE WRITE and no validation COMMAND between them.

Re-read Churn

Same canonical file path is read at least 3 times within a 10-action window, with no intervening write to that file.

Tool Oscillation

For one file, at least 2 READ–WRITE–READ cycles where the middle write is deterministically labeled failed or reverted.

No Formal Tail Validation Unsupported Completion Claim Shell-over-Tool Structured Plan Absence Redundant Search

Canonical FILE READ/FILE WRITE targets; no oracle. Canonical targets plus effect labels; no oracle.

Secondary failure-associated clue; split-sensitive.

Low-prevalence process clue; not task-fixed. Late-run behavior control; No recognized validation command in the overlap of the post-last-source-write Command classifier; no oracle; not treated as failure region and final 5 actions. descriptive late-run signal. evidence. Within-task Final reasoning text matches a word-boundary fixed/done/resolved regex and Final REASON text plus completion-evidence signal; no successful validation command appears after the last source write. command outcomes; no oracle. does not judge statement truth. Scaffold behavior control; A shell command’s first word is cat/head/tail/less/more, grep-family, rg/ag, or Requires observable shell and not treated as aggregate find while structured read/search tools are exposed. tool surface; no oracle. failure evidence. At least 5 FILE WRITE actions occur with no preceding structured-plan or Canonical FILE WRITE actions Plan-phase failure-associated TODO tool call (tool-gated; weaker variant). plus plan-tool surface; no oracle. clue. Cross-cutting Same exact-normalized SEARCH query recurs at least 2 times within a Canonical SEARCH query text; failure-associated clue; 10-action window. no oracle. distinct from search loop.

Semantic single-trajectory detectors (labeler-based; require phase/action labels) At least 3 transitions between the same two semantic phases within a 6-action Requires semantic phase labels Phase Oscillation window. (optional labeler); no oracle. Requires semantic phase/action Semantic Fruitless At least 5 code-read actions where at least 4 read files never appear in any labels (optional labeler); no Exploration later implement step. oracle.

Semantic process clue; failure-associated. Understand-phase failure-associated clue.

Cross-trajectory divergence patterns (deterministic; reference-aligned) Off-anchor Exploration Scope Drift Rapid Rewrite

Compared READ/SEARCH targets have off-anchor ratio at least 0.85 against the reference read set or, when available, the gold changed-file set.

Cross-trajectory alignment plus reference or anchor set.

Compared writes target files outside the reference write set or, in benchmark-informed mode, outside the gold changed-file set. A write to file f is followed by another write to f within 3 steps that overwrites or reverses it, and the reference does not also rewrite.

Cross-trajectory write targets plus reference/anchor set. Cross-trajectory write sequence and reference alignment.

File-selection divergence; anchor-grounded when gold files are used. File-selection classification evidence. Edit-stability classification evidence.

Oracle-grounded milestones Milestones

M1 first anchor read, M2 first anchor write, M3 all anchors written, M4 first passing validation, M5 first justified action.

contrasts where the scaffold changes. To measure run-to-run variability, we repeat a 50-task stratified random subsample three independent times under each setting, producing 750 additional trajectories. The sample is grouped by how many of the five settings resolve each task: 20 easy, 20 medium, and 10 hard tasks. This repeated-run check estimates within-task variance for each process-profile metric and keeps the main results from depending on single-trace attribution. For crossbenchmark transfer, we also use all 266 SWE-Bench Pro [17] Python tasks, collecting all 1,330 task-setting runs. We use SWE-Bench Pro to check whether the main associations and scaffold patterns carry to a harder same-family benchmark. All settings receive the same issue-only prompt and use CLI-default inference parameters. Each task runs once in an isolated working directory at the task’s base commit. Failed, timed-out, and zero-patch runs are retained rather than filtered. We use GPT-5.4 as the semantic step labeler over the closed taxonomy in Table I. B. Baselines and Controls Because our focus is process diagnosis rather than task solving, we compare against baselines that use the same runs

Anchor set plus canonical actions; unreached milestones are right-censored.

Oracle-grounded pace profile.

and the same outcome labels. The outcome-only baseline is resolve rate alone. The raw-telemetry baseline uses steps, tokens, durations, failed tool calls, and edit counts. We also use same-outcome and same-task controls, including resolvedvs-resolved pairs and frozen-threshold reuse. Together, these baselines separate trajectory-specific signal from outcome differences, generic run cost, and task difficulty. V. R ESULTS The empirical study asks what trajectory structure adds beyond success/failure outcomes. We organize the evaluation from single-run signals to controlled cross-run comparisons. RQ1 asks which anti-patterns act as failure clues, how same-task comparisons localize divergence, and whether the strongest detector signal transfers after thresholds are frozen. RQ2 asks whether deterministic effect labels explain process differences beyond raw telemetry such as steps and failed tool calls. RQ3 asks whether agent settings with similar resolve rates differ in process profiles and task-relevant progress timing. RQ4 asks whether reference-scoped cross-run classifications are stable under reference changes and whether samemodel controls reveal scaffold-driven process differences.

TABLE III: Detector prevalence by outcome and SWEBench Verified difficulty. Resolve/Fail report trajectory-level prevalence over all five settings. Easy/Medium/Hard follow the SWE-Bench Verified human time-to-fix buckets (<15m, 15–60m, and >1h, with Hard merging the 1–4h and >4h buckets) [59]. Detector

Resolve

Fail

Easy

Medium

Hard

Structural (deterministic) Search Loop Re-read Churn Tool Oscillation No Formal Tail Validation Unsupported Completion Claim Shell-over-Tool Structured Plan Absence Redundant Search

41.1 34.0 5.0 63.2 7.6 15.0 9.1 9.6

56.1 44.7 8.8 58.2 9.5 14.9 13.3 11.4

34.2 25.5 3.4 60.1 8.0 12.0 7.0 7.7

49.5 43.0 6.7 62.9 8.2 16.6 11.0 11.4

67.6 51.6 13.8 63.1 8.0 18.7 19.6 12.9

Semantic (labeler-based) Phase Oscillation Semantic Fruitless Exploration

23.4 38.4

29.6 47.2

19.7 32.3

27.0 44.5

37.8 56.0

A. RQ1: Controlled Failure Evidence We use detector prevalence as a first pass over the catalog. Because each task appears under multiple settings, the aggregate comparisons in Table III are descriptive rather than causal. This screen identifies search loop as the clearest failure clue because it is common and associated with failed runs. Reread churn provides a secondary but less stable clue, while tool oscillation is weaker. The difficulty columns qualify this interpretation. Many detectors become more common as tasks get harder, so the resolved–failed gap can reflect residual task difficulty rather than a failure-specific process. Other rows reinforce the same caution. Shell-over-tool mainly reflects scaffold convention, and no formal tail validation occurs more often in resolved runs. The catalog therefore provides deterministic pattern definitions and step-level evidence for inspection, rather than a list of failure predictors. The same limitation appears in a simpler raw-telemetry baseline. Figure 3 shows that failed runs usually shift toward more steps within each setting, but the resolved and failed distributions still overlap. Harder SWE-Bench Verified tasks also have longer runs and a higher failure share. Step count can therefore spot expensive runs at the population level, but it does not explain why a particular run failed. Same-task divergence. Detector prevalence and step counts can show that runs differ, but they do not show where the difference occurs. We therefore align each failed run against a resolved run of the same task. For each diagnostic layer, we report the percentage of alignments where C ONVERGE rates agreement with the reference as moderate or weak. File selection measures whether the compared run searched, read, or wrote in a different file scope from the reference. Edit stability measures unstable rewrites and repeated failed actions. Completion behavior measures unresolved dead ends and inefficient action ordering near the end of the run, while function selection applies the same comparison at the enclosing-function level when anchors are available. Across the failed/resolved contrasts in Table IV, file selection is usually the largest or tied-largest of the three layers, and completion behavior is also substantial. The useful signal is not simply that a failed run did more work, but where its work

Fig. 3: Step-count distributions by outcome. Split violins use a log scale, dashed lines mark quartiles, and labels report resolved/failed counts (R/F). Panel (a) groups runs by setting, while panel (b) groups runs by SWE-Bench Verified task difficulty. departed from a successful run on the same task. This points inspection first toward file scope, then toward completion behavior and edit stability. Function-level localization. The same-task result motivates a finer target view. File-level divergence is broad because agents can inspect different files before reaching a valid solution. We therefore resolve each benchmark patch hunk to its enclosing function and ask whether the failed run reached the same code region as a resolved run for the same task. Figure 4 shows when the first function-level divergence appears in the failed run. Across contrasts, the median lead time is around 20 steps before the failed run ends. This means function-level divergence is available early enough to guide inspection, but it does not prove why the run failed. The distinction is useful because failures do not all have the same shape. Some runs work in the wrong file scope, some reach the right file but miss the relevant function, and some reach the right function but still make the wrong edit. A file-level view collapses these cases, while function-level analysis separates scope recovery from patch reasoning or validation failure. Cross-benchmark transfer. Finally, we test whether the frozen detector catalog carries beyond SWE-Bench Verified. We apply it unchanged to all Python tasks in SWE-Bench Pro. In Table V, the prevalence column shows how often a detector is observed. The outcome link column is directional: values below 1 mean the detector appears more in failed runs, and values above 1 mean it appears more in resolved runs. Search loop is the most stable transfer signal. It appears at different

TABLE IV: Same-task divergence by diagnostic layer. Cells report the percentage of comparisons where the failed run departs from the resolved reference. N is the number of task-level comparisons in each row. Function Selection is a finer refinement over comparisons with resolvable function anchors, so its denominator is at most N. Contrast

Held fixed

N

File Selection

Edit Stability

Completion Behavior

Function Selection

Claude Code-Opus vs. OpenCode-Opus Codex-GPT vs. OpenCode-GPT OpenCode-Opus vs. OpenCode-GPT OpenCode-Opus vs. OpenCode-GLM OpenCode-GPT vs. OpenCode-GLM

Model Model Scaffold Scaffold Scaffold

52 101 92 73 101

38.5 95.0 71.7 39.7 79.2

15.4 37.6 42.4 38.4 43.6

38.5 57.4 42.4 39.7 51.5

52.4 42.6 17.0 20.0 19.2

Fig. 5: Effect-label distribution by agent setting. Each stacked bar shows the share of non-reasoning canonical actions assigned to each effect label. Off-anchor uses the gold-patch anchor set, and the other labels are oracle-free. Fig. 4: Lead time from the earliest same-task function-level divergence to failure. Each violin shows one failed/resolved contrast. Thick black bars mark the interquartile range, labels mark medians, and the red dashed line marks the pooled median. TABLE V: Cross-benchmark transfer check of the frozen structural detectors. The % columns show how often each detector is observed. Outcome link values below 1 mean the detector is more common in failed runs, while values above 1 mean it is more common in resolved runs. Detector Search Loop Re-read Churn Tool Oscillation No Formal Tail Validation Unsupported Completion Claim Shell-over-Tool Structured Plan Absence Redundant Search

Verified Pro Verified % Outcome Link Pro % Outcome Link 45.2 37.0 6.0 61.8 8.1 15.0 10.2 10.1

0.55 0.64 0.55 1.23 0.78 1.01 0.65 0.83

65.0 30.3 4.8 46.8 5.4 33.6 30.0 13.5

0.64 0.84 1.34 0.85 1.11 0.88 0.84 1.19

rates across the two benchmarks, but its outcome-link value stays below 1 in both, meaning it remains more common in failed runs. Re-read churn keeps the same direction but moves closer to 1 on Pro, so its association with failure weakens. Several other detectors move toward 1 or change direction, which means their prevalence is more benchmark-specific. The useful insight is that detector definitions can transfer, but detector thresholds should be audited on the target benchmark. Search loop remains a portable warning sign, while the shifted prevalence rates show why practitioners should not reuse thresholds blindly across benchmarks.

B. RQ2: Effect Labels as Diagnostics This RQ asks whether deterministic effect labels reveal process differences that raw telemetry cannot show. Raw counts report how many steps, tool calls, or failures occurred. Effect labels add a different view: they record whether actions produced surviving changes, failed, were reverted, were justified by task-relevant evidence, or moved outside the anchor set. We therefore use Figure 5 to compare how agent settings distribute their work across these outcomes. Effect labels give each setting a compact process profile. From the figure (per-setting values in Table VI), Codex has the largest survived-action share, OpenCode/GPT-5.4 has the largest failed-action share, and OpenCode/Opus 4.6 has the lowest off-anchor share. These differences show that settings can spend their actions in different ways even when raw counts or final outcomes look similar. The profile yields three practical insights. The survived-action share shows how much work becomes durable code change, the failed and reverted shares show how much effort is lost, and the offanchor share shows how much exploration falls outside the benchmark patch scope. These signals make effect labels useful for comparing settings and selecting runs for inspection. They should not be read as standalone explanations for why a specific run passed or failed. C. RQ3: Setting-Level Process Differences RQ3 asks whether agent settings with similar resolve rates still behave differently. Table VI summarizes each setting with process metrics and oracle-grounded milestone reach. Most process rows are oracle-free action or effect metrics.

TABLE VI: Process profiles and milestone reach by agent setting. Rows from Steps/task through Harmful ratio report per-task medians unless marked as means. Resolve rate and milestone rows report the percentage of tasks reaching each outcome or anchor-grounded event. Off-anchor and milestone rows use the gold-patch anchor set. Claude Code

Codex

OpenCode GLM

OpenCode GPT

OpenCode Opus

77.4 27 25.0 96.7 27.6 3.3 36.9 5.1 17.6 0.07

78.4 24 41.0 91.9 36.2 8.0 32.0 4.0 14.3 0.10

71.6 33 33.0 100.0 26.1 8.3 37.7 2.2 18.1 0.12

64.2 14 22.0 100.0 16.7 17.6 34.2 4.7 23.0 0.20

71.4 17 16.5 100.0 23.1 4.4 41.7 9.5 10.2 0.08

Oracle-grounded milestone reach (% of 500 tasks) M1 first read 99.8 98.2 99.4 M2 first write 95.0 95.8 93.6 M3 all written 85.0 86.2 83.6 M4 validation 2.6 2.4 4.4 M5 justified 99.8 99.8 99.8

99.2 91.8 82.2 1.4 100.0

97.8 94.2 84.6 2.8 100.0

Metric Resolve rate % Steps/task Tool calls Tool success % Survived % Failed % Justified % Recorded % (mean) Off-anchor % Harmful ratio

The off-anchor row and milestone rows use the gold-patch anchor set, so they are interpreted only for benchmark tasks with anchors. Pairwise comparisons are paired by task, and attribution claims are limited to same-scaffold OpenCode or same-model contrasts. The process rows show why resolve rate alone is incomplete. Codex has the highest resolve rate and the largest survived-action share, but it also uses many tool calls. OpenCode/GPT-5.4 has the fewest steps, yet it has the highest failed-action share and harmful ratio, which is the share of a run’s non-reasoning actions that failed or were reverted. OpenCode/Opus 4.6 has the lowest off-anchor share and the highest justified-action share. These differences show that settings with similar resolve rates can still differ in durable edits, failed work, and task-scope focus. For practitioners, this profile helps identify what changed after a model or scaffold update, even when the final resolve rate changes little. Milestones add timing information that the aggregate process rows cannot show (Figure 6). In raw steps, OpenCode/Opus reaches the first anchor write earliest, while Claude Code and Codex reach it later. After normalizing by run length, OpenCode/GLM moves earlier because it reaches task-relevant files early within longer runs. This contrast separates quick progress from short runs. An agent can take more steps overall while still reaching relevant code early in its own trajectory. Milestones therefore measure progress timing, not patch correctness. Taken together, the process profiles and milestones give practitioners two complementary checks. The profile rows show what kind of work a setting performs, while milestones show when the setting reaches task-relevant code. This makes RQ3 useful for comparing model or scaffold updates even when resolve rate changes little.

Fig. 6: First anchor write timing (M2) by setting. Points mark medians and bars mark IQR. The left panel reports raw steps, and the right panel normalizes by run length. Other milestones are omitted because M3 usually coincides with M2 on the single-file tasks that dominate the benchmark, M1 and M5 saturate by step 2, and M4 is reached by under 3% of runs. TABLE VII: Reference-scoped divergence by diagnostic dimension. Cells report the percentage of same-model scaffold comparisons where the compared run has moderate or weak agreement with the fixed reference. Reference change reports how often the agreement category changes after replacing the fixed Codex reference with the per-task most-efficient resolved run. Small reference-change values, especially for file selection, show that these comparisons are largely insensitive to which resolved reference is chosen. The reference-change check covers 435 shared tasks. Diagnostic dimension File selection Edit stability Completion behavior

GPT-5.4 divergence %

Reference change %

Opus 4.6 divergence %

93.0 31.4 47.0

0.69 22.1 8.7

18.6 13.2 13.6

D. RQ4: Reference-Scoped Comparisons RQ4 checks how C ONVERGE behaves when the comparison reference changes and when the model is held fixed while the scaffold changes. The GPT-5.4 contrast compares Codex-GPT with OpenCode-GPT, and the Opus 4.6 contrast compares Claude Code-Opus with OpenCode-Opus. These same-model contrasts let us ask whether scaffold-driven process differences are visible beyond resolve rate. The middle column of Table VII is the reference-change check. C ONVERGE aligns a compared run against a resolved reference run for the same task and assigns strong, moderate, or weak agreement for each diagnostic dimension. Because a task may have multiple resolved runs, we recompute the comparison with the fixed Codex reference and with the pertask most-efficient resolved reference. Reference change is the percentage of comparisons where that agreement category changes. File selection changes in only 0.69% of comparisons, so file-scope evidence is almost insensitive to reference choice. Edit stability changes in 22.1% because recovery and rewrite

behavior can vary even among resolved runs. Completion behavior falls between them at 8.7%. Thus, reference-scoped comparison is robust for file selection, while edit stability should be read with more reference sensitivity. The outer columns show that scaffold effects depend on the model being held fixed. Under GPT-5.4, the two scaffolds diverge strongly, especially in file selection. Under Opus 4.6, the same kind of scaffold contrast produces much less divergence in every diagnostic dimension. This means that a scaffold comparison is not a constant property of the scaffold pair. Holding the model fixed exposes process differences that endpoint resolve rates can hide. At the same time, the result is not an absolute quality ranking of the scaffolds; it describes how a compared run departs from a specified reference. Cross-benchmark scaffold control. SWE-Bench Pro provides a further same-model scaffold check. Claude Code and OpenCode share Opus 4.6 and have similar resolve rates on this corpus, yet their detector patterns differ. Across the 266 shared Pro tasks, structured plan absence is higher for Claude Code (49.6% vs 0.4%) and re-read churn is higher for OpenCode (43.6% vs 16.9%). The difference is therefore not simply that one scaffold records more of every detector pattern. This result reinforces the RQ1 caution that a prevalence shift can be a scaffold signature unless it is checked against outcome, same-task, or same-model evidence. The Pro scaffold control therefore complements the detector-transfer result. The same frozen detector signals can reveal both outcome association and process shifts, but the interpretation depends on the contrast being analyzed. VI. D ISCUSSION AND T HREATS TO VALIDITY Discussion. This study shows why process diagnosis should be reported alongside outcome evaluation. Two runs can share the same success/failure outcome while following different processes, so a T RACE P ROBE report should be read as inspection evidence rather than automatic remediation advice. To expose these differences, I NSIGHT identifies recurring anti-patterns that help prioritize runs and task families for review. For crossrun comparison, C ONVERGE localizes divergence against a specified reference, while effect labels and milestones describe what work was performed and when task-relevant code was reached. Taken together, these views provide the strongest evidence through same-task localization at file and function granularity, the stable search-loop clue, and the deterministic representation that makes comparisons repeatable. At the same time, the controls narrow the claim. Several aggregate signals weaken under task-fixed analysis, and the transfer check preserves only part of the detector pattern. For practical use, process profiles should accompany resolve rate. Practitioners should prefer controlled contrasts over aggregate prevalence shifts, and detector thresholds should be audited on the target benchmark before conclusions are reused. Under this scope, T RACE P ROBE supports inspection and comparison, but it is not a replacement for resolve-rate evaluation, a per-run causal oracle, or a remediation method.

Threats to validity. The main calibration threat is benchmarkspecific detector tuning. Several detector thresholds were chosen on SWE-Bench Verified. The Verified split and rawthreshold sensitivity check assess stability within that corpus after thresholds are fixed, while the SWE-Bench Pro check asks whether the strongest direction carries to a disjoint benchmark. These checks support stability after thresholds are fixed and bounded transfer, not independent threshold discovery. However, they do not show that the same thresholds will produce the same prevalence rates or conclusions on every benchmark. T RACE P ROBE is general because new trajectories can be normalized, labeled, and compared through the same pipeline, but benchmark-specific claims still require benchmark-specific threshold audits. Beyond calibration, sampling and construct limits also shape the claims. Each maincorpus task/setting pair is run once, outcome labels inherit SWE-Bench test-suite limits, and C ONVERGE agreement assessments measure reference-relative divergence rather than absolute quality or human-validated labels. Both benchmarks are Python issue to PR repair, so results may not transfer to other languages or task types, and we do not yet evaluate whether the diagnostics change practitioner outcomes. Execution measurements add another limitation because they can depend on the provider and scaffold. Duration, token counts, and monetary cost can change with CLI versions, provider-side batching, tokenizer accounting, rate limits, tool permissions, and budget-stop reporting. For that reason, we interpret them as observed process-cost measurements in this collection, not as intrinsic model speed or efficiency. VII. C ONCLUSION Success/failure outcomes are necessary for evaluating coding agents, but they do not explain the process that produced an accepted or failed run. This work addresses that gap by treating coding agent trajectories as structured process evidence. T RA CE P ROBE normalizes raw traces into canonical actions and deterministic effect labels, uses I NSIGHT to identify recurring single-trajectory anti-patterns, and uses C ONVERGE to compare runs against explicit references. In our SWE-Bench Verified study and SWE-Bench Pro transfer check, T RACE P ROBE identifies population-level failure clues, localizes where failed and resolved runs diverge, and shows that resolved runs can still differ in progress timing, failed work, and observed process cost. These findings support inspection and comparison, while the controls keep the claims short of explaining exactly why a particular run failed or automatically fixing it. Future directions include pairing these deterministic diagnostics with human-validated root-cause analysis and intervention studies. Those studies can test whether changes to prompts, retrieval, memory, tool policy, or validation strategy improve agent behavior while preserving controlled comparisons and keeping deterministic evidence separate from human judgment. R EFERENCES [1] Anthropic, “Claude Code,” https://claude.com/product/claude-code, 2025, agentic coding assistant by Anthropic, accessed 2026.

[2] OpenAI, “Codex,” https://openai.com/codex/, 2025, cloud-based software engineering agent by OpenAI, accessed 2026. [3] OpenCode Contributors, “OpenCode: An open-source AI coding agent,” https://opencode.ai/, 2025, open-source coding agent supporting multiple model backends, accessed 2026. [4] B. Yu, Y. Zhu, P. He, and D. Kang, “Utboost: Rigorous evaluation of coding agents on swe-bench,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Association for Computational Linguistics, 2025, pp. 3762–3774. [Online]. Available: https://doi.org/10.18653/v1/2025.acl-long.189 [5] Y. Wang, M. Pradel, and Z. Liu, “Are ”solved issues” in swe-bench really solved correctly? an empirical study,” CoRR, vol. abs/2503.15223, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2503.15223 [6] S. Liang, S. Garg, and R. Z. Moghaddam, “The swe-bench illusion: When state-of-the-art llms remember instead of reason,” CoRR, vol. abs/2506.12286, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2506.12286 [7] Anthropic, “Quantifying infrastructure noise in agentic coding evals,” https://www.anthropic.com/engineering/infrastructure-noise, 2026, anthropic Engineering Blog. [8] OpenAI, “Why SWE-bench Verified no longer measures frontier coding capabilities,” https://openai.com/index/ why-we-no-longer-evaluate-swe-bench-verified/, 2025, openAI Blog. [9] C. Ma, J. Zhang, Z. Zhu, C. Yang, Y. Yang, Y. Jin, Z. Lan, L. Kong, and J. He, “Agentboard: An analytical evaluation board of multi-turn LLM agents,” in Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, Eds., 2024. [Online]. Available: http://papers.nips.cc/paper files/paper/2024/ hash/877b40688e330a0e2a3fc24084208dfa-Abstract-Datasets and Benchmarks Track.html [10] M. Kim, D. Wang, S. Cui, F. Farmahinifarahani, S. Garg, B. Ray, T. Y. Zhuo, R. Mukherjee, and V. Kumar, “TRAJEVAL: decomposing code agent trajectories for fine-grained diagnosis,” CoRR, vol. abs/2603.24631, 2026. [Online]. Available: https://doi.org/10.48550/ arXiv.2603.24631 [11] S. Liu, Y. Chen, R. Krishna, S. Sinha, J. Ganhotra, and R. Jabbarvand, “Process-centric analysis of agentic software systems,” CoRR, vol. abs/2512.02393, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2512.02393 [12] O. Majgaonkar, Z. Fei, X. Li, F. Sarro, and H. Ye, “Understanding code agent behaviour: An empirical study of success and failure trajectories,” CoRR, vol. abs/2511.00197, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2511.00197 [13] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https: //openreview.net/forum?id=VTF8yNQM66 [14] B. H. Bjarnason, A. Silva, and M. Monperrus, “On randomness in agentic evals,” CoRR, vol. abs/2602.07150, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2602.07150 [15] L. Zhang, S. He, C. Zhang, Y. Kang, B. Li, C. Xie, J. Wang, M. Wang, Y. Huang, S. Fu, E. Nallipogu, Q. Lin, Y. Dang, S. Rajmohan, and D. Zhang, “Swe-bench goes live!” in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diago, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, D. Belgrave, C. Zhang, L. N. Montoya, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, N. Chen, I. V. M. Ruı́z, and A. Loaiza-Bonilla, Eds., 2025. [Online]. Available: http://papers.nips.cc/paper files/paper/2025/ hash/d83c4a745789690f82e86d0ef752ae7c-Abstract-Datasets and Benchmarks Track.html [16] I. Badertdinov, A. Golubev, M. Nekrashevich, A. Shevtsov, S. Karasik, A. Andriushchenko, M. Trofimova, D. Litvintseva, and B. Yangel, “Swerebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,” CoRR, vol. abs/2505.20411, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2505.20411 [17] X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, C. B. C. Zhang, N. Jacobson, B. Liu, and

B. Kenstler, “Swe-bench pro: Can AI agents solve long-horizon software engineering tasks?” CoRR, vol. abs/2509.16941, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2509.16941 [18] D. Zan, Z. Huang, W. Liu, H. Chen, L. Zhang, S. Xin, L. Chen, Q. Liu, X. Zhong, A. Li, S. Liu, Y. Xiao, L. Chen, Y. Zhang, J. Su, T. Liu, R. Long, K. Shen, and L. Xiang, “Multi-swe-bench: A multilingual benchmark for issue resolving,” CoRR, vol. abs/2504.02605, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2504.02605 [19] M. S. Rashid, C. Bock, Y. Zhuang, A. Buchholz, T. B. Esler, S. Valentin, L. Franceschi, M. Wistuba, P. T. Sivaprasad, W. J. Kim, A. Deoras, G. Zappella, and L. Callot, “Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents,” CoRR, vol. abs/2504.08703, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2504.08703 [20] J. Xu, K. Deng, W. Li, S. Yu, H. Tang, H. Huang, Z. Lai, Z. Zhan, Y. Wu, C. Zhang, K. Lei, Y. Yao, X. Lei, W. Zhu, Z. Feng, H. Li, J. Xiong, D. Li, Z. Gao, K. Wu, W. Xiang, Z. Zhan, Y. Zhang, W. Gong, Z. Gao, G. Wang, Y. Xue, M. Li, M. Xie, X. Zhang, J. Wang, W. Zhuang, Z. Lin, H. Wang, Z. Zhang, Y. Zhang, H. Zhang, B. Chen, and J. Liu, “Swe-compass: Towards unified evaluation of agentic coding abilities for large language models,” CoRR, vol. abs/2511.05459, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2511.05459 [21] J. Chen, H. Huang, Y. Lyu, J. An, J. Shi, C. Yang, T. Zhang, H. Tian, Y. Li, Z. Li, X. Zhou, X. Hu, and D. Lo, “Secureagentbench: Benchmarking secure code generation under realistic vulnerability scenarios,” CoRR, vol. abs/2509.22097, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2509.22097 [22] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” in Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, Eds., 2024. [Online]. Available: http://papers.nips.cc/paper files/paper/2024/hash/ 5a7c947568c1b1328ccc5230172e1e7c-Abstract-Conference.html [23] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, and et al., “Openhands: An open platform for AI software developers as generalist agents,” in The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [Online]. Available: https://openreview.net/forum?id=OJd3ayDDoF [24] C. S. Xia, Y. Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying llm-based software engineering agents,” CoRR, vol. abs/2407.01489, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2407.01489 [25] P. He, Z. Dai, B. He, H. Liu, X. Tang, H. Lu, J. Li, J. Ding, S. Mukherjee, S. Wang, Y. Xing, J. Tang, and B. Dumoulin, “Traject-bench:a trajectory-aware benchmark for evaluating agentic tool use,” CoRR, vol. abs/2510.04550, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2510.04550 [26] J. Kuang, Y. Li, X. Zhang, Y. Li, D. Yin, X. Sun, Y. Shen, and P. S. Yu, “Process-level trajectory evaluation for environment configuration in software engineering agents,” CoRR, vol. abs/2510.25694, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2510.25694 [27] T. Mehtiyev and W. Assunção, “Beyond resolution rates: Behavioral drivers of coding agent success and failure,” CoRR, vol. abs/2604.02547, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2604.02547 [28] I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought-action-result trajectories,” in 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 2025, pp. 2846–2857. [Online]. Available: https://doi.org/10.1109/ASE63991. 2025.00234 [29] I. Ceka, S. Pujar, S. Ramji, L. Buratti, G. E. Kaiser, and B. Ray, “Understanding software engineering agents through the lens of traceability: An empirical study,” CoRR, vol. abs/2506.08311, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2506.08311 [30] D. Deshpande, V. Gangal, H. Mehta, J. Krishnan, A. Kannappan, and R. Qian, “TRAIL: trace reasoning and agentic issue localization,” CoRR, vol. abs/2505.08638, 2025. [Online]. Available: https://doi.org/ 10.48550/arXiv.2505.08638 [31] S. Guo, Z. Jin, Q. Chen, D. Gotz, H. Zha, and N. Cao, “Interpretable anomaly detection in event sequences via sequence matching and visual comparison,” IEEE Trans. Vis. Comput. Graph., vol. 28, no. 12, pp. 4531–4545, 2022. [Online]. Available: https: //doi.org/10.1109/TVCG.2021.3093585

[32] S. Barke, A. Goyal, A. Khare, A. Singh, S. Nath, and C. Bansal, “Agentrx: Diagnosing AI agent failures from execution trajectories,” CoRR, vol. abs/2602.02475, 2026. [Online]. Available: https://doi.org/ 10.48550/arXiv.2602.02475 [33] M. Wang, X. Xie, and Y. Huo, “Trajaudit: Automated failure diagnosis for agentic coding systems,” CoRR, vol. abs/2605.26563, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2605.26563 [34] T. Ou, W. Guo, A. Gandhi, G. Neubig, and X. Yue, “Agentdiagnose: An open toolkit for diagnosing LLM agent trajectories,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025 - System Demonstrations, Suzhou, China, November 4-9, 2025, I. Habernal, P. Schulam, and J. Tiedemann, Eds. Association for Computational Linguistics, 2025, pp. 207–215. [Online]. Available: https://doi.org/10.18653/v1/2025.emnlp-demos.15 [35] S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y. Chen, and Q. Wu, “Which agent causes task failures and when? on automated failure attribution of LLM multiagent systems,” in Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, ser. Proceedings of Machine Learning Research, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, Eds., vol. 267. PMLR / OpenReview.net, 2025. [Online]. Available: https://proceedings.mlr.press/v267/zhang25cq.html [36] G. Zhang, J. Wang, J. Chen, W. Zhou, K. Wang, and S. Yan, “Agentracer: Who is inducing failure in the LLM agentic systems?” CoRR, vol. abs/2509.03312, 2025. [Online]. Available: https://doi.org/ 10.48550/arXiv.2509.03312 [37] K. Zhu, Z. Liu, B. Li, M. Tian, Y. Yang, J. Zhang, P. Han, Q. Xie, F. Cui, W. Zhang, X. Ma, X. Yu, G. Ramesh, J. Wu, Z. Liu, P. Lu, J. Zou, and J. You, “Where LLM agents fail and how they can learn from failures,” CoRR, vol. abs/2509.25370, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2509.25370 [38] M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. G. Parameswaran, D. Klein, K. Ramchandran, M. A. Zaharia, J. E. Gonzalez, and I. Stoica, “Why do multi-agent LLM systems fail?” in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diago, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025, D. Belgrave, C. Zhang, L. N. Montoya, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, N. Chen, I. V. M. Ruı́z, and A. Loaiza-Bonilla, Eds., 2025. [Online]. Available: http://papers.nips.cc/paper files/paper/ 2025/hash/b1041e52d3be19f0a9bc491657488e4a-Abstract-Datasets and Benchmarks Track.html [39] D. Pathak, H. Kumar, A. Roy, F. George, M. Verma, and P. Moogi, “Detecting silent failures in multi-agentic AI trajectories,” CoRR, vol. abs/2511.04032, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2511.04032 [40] V. Mittapalli, S. J. Dani, S. S. Pilli, S. Ansu, M. Teymoorianfard, F. Dernoncourt, H. Chen, Y. Wang, R. A. Rossi, and N. K. Ahmed, “Trace: Trajectory reasoning through adaptive cross-step evidence aggregation for llm agents,” 2026. [Online]. Available: https://arxiv.org/abs/2606.07054 [41] M. Desmond, J. Y. Lee, I. Ibrahim, J. M. Johnson, A. Sil, J. MacNair, and R. Puri, “Agent trajectory explorer: Visualizing and providing feedback on agent trajectories,” in Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2025, Philadelphia, PA, USA, February 25 - March 4, 2025, T. Walsh, J. Shah, and Z. Kolter, Eds. AAAI Press, 2025, pp. 29 634–29 636. [Online]. Available: https://doi.org/10.1609/aaai.v39i28.35350 [42] T. Bula, S. Pujar, L. Buratti, M. A. Bornea, and A. Sil, “Seaview: Software engineering agent visual interface for enhanced workflow,” CoRR, vol. abs/2504.08696, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2504.08696 [43] J. Wang, Y. Chen, M. Pan, C. M. Yeh, and M. Das, “Illuminating LLM coding agents: Visual analytics for deeper understanding and enhancement,” CoRR, vol. abs/2508.12555, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2508.12555 [44] H. Li, Y. Yao, L. Zhu, R. Feng, H. Ye, J. Wang, Y. He, P. Zou, L. Zhang, X. Lei, H. Huang, K. Deng, M. Sun, Z. Zhang, H. Ye, and J. Liu, “Codetracer: Towards traceable agent states,” CoRR, vol. abs/2604.11641, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2604.11641 [45] Z. Xi, C. Liao, G. Li, Z. Zhang, W. Chen, B. Wang, S. Jin, Y. Zhou, J. Guan, W. Wu, T. Ji, T. Gui, Q. Zhang, and X. Huang, “Agentprm:

Process reward models for LLM agents via step-wise promise and progress,” in Proceedings of the ACM Web Conference 2026, WWW 2026, Dubai, United Arab Emirates, originally scheduled for April 1317, 2026, rescheduled for June 29 - July 3, 2026, H. Hacid, Y. Maarek, F. Bonchi, I. Guy, and E. Yilmaz, Eds. ACM, 2026, pp. 4184–4195. [Online]. Available: https://doi.org/10.1145/3774904.3792551 [46] H. Wang, C. T. Leong, J. Wang, J. Wang, and W. Li, “SPA-RL: reinforcing LLM agents via stepwise progress attribution,” CoRR, vol. abs/2505.20732, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2505.20732 [47] S. Gandhi, J. Tsay, J. Ganhotra, K. Kate, and Y. Rizk, “When agents go astray: Course-correcting SWE agents with prms,” CoRR, vol. abs/2509.02360, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2509.02360 [48] Y. Xu, D. Lu, Z. Shen, J. Wang, Z. Wang, Y. Mao, C. Xiong, and T. Yu, “Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials,” in The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [Online]. Available: https://openreview. net/forum?id=EEgYUccwsV [49] J. Ni, Y. Liu, X. Liu, Y. Sun, M. Zhou, P. Cheng, D. Wang, E. Zhao, X. Jiang, and G. Jiang, “Trace2skill: Distill trajectory-local lessons into transferable agent skills,” CoRR, vol. abs/2603.25158, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2603.25158 [50] G. Fang, V. Isahagian, K. R. Jayaram, R. Kumar, V. Muthusamy, P. Oum, and G. Thomas, “Trajectory-informed memory generation for self-improving agent systems,” CoRR, vol. abs/2603.10600, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2603.10600 [51] A. Cuadron, A. Desai, L. G. Schroeder, X. Wang, W. Ma, D. Li, Y. Wang, I. Stoica, G. Neubig, and J. E. Gonzalez, “Shepherd: Patternguided trajectory selection for coding agents on SWE-bench,” 2026. [Online]. Available: https://openreview.net/forum?id=ZBOFr4ryBk [52] J. Lin, S. Liu, C. Pan, L. Lin, S. Dou, X. Huang, H. Yan, Z. Han, and T. Gui, “Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses,” CoRR, vol. abs/2604.25850, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2604.25850 [53] M. Fowler, Refactoring - Improving the Design of Existing Code, ser. Addison Wesley object technology series. Addison-Wesley, 1999. [Online]. Available: http://martinfowler.com/books/refactoring.html [54] W. J. Brown, R. C. Malveau, H. W. McCormick III, and T. J. Mowbray, “Refactoring software, architectures, and projects in crisis,” Google Scholar Google Scholar Digital Library Digital Library, 1998. [55] D. A. Tamburri, P. Kruchten, P. Lago, and H. van Vliet, “Social debt in software engineering: insights from industry,” J. Internet Serv. Appl., vol. 6, no. 1, pp. 10:1–10:17, 2015. [Online]. Available: https://doi.org/10.1186/s13174-015-0024-6 [56] A. Mailach and N. Siegmund, “Socio-technical anti-patterns in building ml-enabled software: Insights from leaders on the forefront,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 690–702. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023. 00067 [57] F. Zampetti, C. Vassallo, S. Panichella, G. Canfora, H. C. Gall, and M. D. Penta, “An empirical characterization of bad practices in continuous integration,” Empir. Softw. Eng., vol. 25, no. 2, pp. 1095–1135, 2020. [Online]. Available: https://doi.org/10.1007/s10664-019-09785-8 [58] S. Liu, F. Liu, L. Li, X. Tan, Y. Zhu, X. Lian, and L. Zhang, “An empirical study on failures in automated issue solving,” CoRR, vol. abs/2509.13941, 2025. [Online]. Available: https://doi.org/10.48550/ arXiv.2509.13941 [59] OpenAI, “Introducing SWE-bench Verified,” https://openai.com/index/ introducing-swe-bench-verified/, 2024, human-verified 500-task subset of SWE-bench; curated by OpenAI in collaboration with the SWE-bench authors.