Online Monitoring and Corrective Steering of Programming Agents

Online Monitoring and Corrective Steering of Programming Agents Shuyang Liu† , Saman Dehghan† , Ji Young Kim† , Jatin Ganhotra∗ , Martin Hirzel∗ , and Reyhaneh Jabbarvand† † University of Illinois Urbana-Champaign, USA, {sl225, samand2, jyk14, reyhaneh}@illinois.edu

arXiv:2608.06701v1 [cs.SE] 7 Aug 2026

∗ IBM, USA, {jatinganhotra, hirzel}@us.ibm.com

Abstract—Fixing GitHub issues in large-scale projects is a long-horizon task, especially when a fix requires changes across multiple locations or the issue description lacks the information needed to localize and repair it. As a result, agents traverse long trajectories that are prone to inefficiency and error: they drift away from their intended plan, repeat failed actions, or terminate without a working patch. This paper proposes L IVE P LAN to monitor, detect, and correct such behavioral inefficiencies and drifts in real time. L IVE P LAN decouples judging from advising: a deterministic, rule-based monitor examines general signals over the trajectory to detect issues without invoking an LLM, and only when an issue is detected does it consult an advisor LLM for a high-level, next-step correction. This design avoids the misleading re-planning and costly interventions of prior approaches. We implement L IVE P LAN on top of SWE-agent and evaluate it using five LLMs (three as executor agents and two as advisors) across SWE-bench Verified and SWE-bench Pro. Compared to vanilla SWE-agent, L IVE P LAN notably improves issue resolution rates, achieving consistent gains of up to 15.2% (average: 9.9%), while incurring only an additional cost of $0.08 per instance. The additional solutions concentrate on medium and hard instances. L IVE P LAN consistently outperforms alternative approaches in resolution rate, with minimal regression on already successful runs and new successes on problems that no baseline solves.

I. I NTRODUCTION LLM-based agents solve complex software-engineering tasks by following a plan: the sequence of issue-resolution phases prescribed in their system prompt, e.g., localize, reproduce, patch, validate [37]. Yet an agent may drift away from this plan during execution [18], e.g., skipping validation, or exhibit inefficiencies [8, 19] such as repeated actions and tool call failures, which cause failures or inflate costs even when the agent succeeds [17]. Existing attempts to correct such drift suffer from three main limitations: (1) They fold judging (is the trajectory in trouble) and advising (what should the agent do to fix the trouble) into a single mechanism: an advisor LLM serves as the judge to evaluate the trajectory and, based on that evaluation, recommends corrective actions for the remainder of the trajectory [31]. Judging and advising are inherently conflicting tasks: an advisor prompted to diagnose and provide corrective advice is incentivized to find a problem, a tendency related to the documented sycophantic bias of LLMs toward framing their prompts [30]. The advisor may surface a problem that does not exist and impose misleading advice that derails a run that was, in fact, on track. Overcoming this limitation requires separating judging and advising. Figure 1 shows an example where SAGE [11] re-plans a trajectory for an issue originally resolved by SWE-agent [37]

SAGE, ansible-185d41031 in SWE-bench Pro run with Gemini-2.5-Flash

New Plan

1

Customized High-Level Plan 1. Identify all locations in the default stdout callback (and related callback methods) where host labels are constructed, especially for delegated tasks. 2. Implement a robust `CallbackBase.host_label(result)` in `lib/ansible/plugins/callback/__init_ _.py` that mirrors existing formatting and handles missing `_result` or delegation metadata safely. 3. Replace each duplicated host-label construction with calls to `CallbackBase.host_label(result)` while preserving any surrounding formatting (coloring, prefixes). 4. Run targeted checks or minimal validation to ensure delegated and non-delegated outputs are consistent with previous behavior.

Thought: The user wants to implement the `host_label` static method in `CallbackBase` Action: view lib/ansible/plugins/callback/__init__.py Observation: The path lib/ansible/plugins/callback/__init__.py is not an absolute path...

2

Thought: I made a mistake in the path; it should be absolute. I will correct it and retry viewing the file. view /app/lib/ansible/plugins/callback/__init__.py --view_range -100 -1

Observation: Invalid `view_range`: [-100, -1]. Its first element `-100`

… (Repeat Failed view) 10

Thought: I'm still not finding `CallbackBase`. I will try to be more strategic and use `rg` to find the exact line numbers of `class CallbackBase`

… (Change strategy) 43

Submit

Fig. 1: An example of SAGE [11] incorrect advice (a new global plan), causing SWE-agent to fail on a SWE-bench Pro [7] issue it had originally resolved in a vanilla run. with Gemini-2.5-Flash. SAGE uses GPT-5.2-Codex as an advisor to analyze an existing trajectory—blind to the resolution status—and generates a new plan for re-execution to resolve possible trajectory problems. The advisor’s plan incorrectly directs the agent to implement CallbackBase in lib/ansible/plugins/callback/__init__.py, which does not exist in the repository. This causes the agent to struggle through its first 10 steps, trying to locate a file that does not exist, ultimately terminating without patching. (2) Using an LLM-as-a-judge requires determining how often to trigger it for analysis. Some techniques wait until the end of the trajectory to perform a post-hoc analysis and rerun the agent with advice (new plan) based on the analysis [8, 11, 27]. However, although the advisor can provide a global plan to guide the agent from the beginning, the benefit comes with caveats: if the plan is incorrect, similar to Fig. 1, the agent may not be able to understand and recover, given it is instructed to follow the plan. This is also costly, as it feeds the entire trajectory to the advisor LLM and reruns the agent. Other techniques periodically ask the advisor to evaluate a sliding window of the trajectory [9, 10, 25]. However, if the frequency is too high [9, 25], the approach remains costly; if it is too low [10], the steering feedback arrives too late, making it hard for the agent to recover from the failure mode. Overcoming this limitation entails triggering trajectory analysis only when necessary. Figure 2 shows the execution of SWE-agent with DeepSeekV3 on Openlibrary-ba3abfb6a from SWE-bench Pro [7].

Locate the actual validate_record implementation... then inspect the validation checks

Steps 1 - 5 Navigate the directory structure, read relevant files Step 5th

Steps 6 - 9 Continue exploring, search for and read functions

2

Trajectory Representation

vanilla Observation /Advice

Execute action

no

Any signal detected?

Configurable Signal Database

Keep action, keep advice

no

Drop action, keep advice

no Still cooling down?

Generate a custom advice

yes

yes

Block action?

Use pre-defined advice

(based on: issue description, last guidance, recent trajectory, pre-defined advice)

Advisor

Understood! Let's inspect the related functions to understand their current logic view __init__.py --view_range 760 950 Steps 12 - 15 Read broader context Make an initial submission (no edit)

Update the core import validation logic to accept override validation

Incrementally update

yes

Step 10th 11

3

Found it! The ‘load’ function is defined in ‘__init__.py”. Let’s inspect this file and implement the changes. view __init__.py

Intervention Processor

10 Open the relevant portion, and load … to inspect the validation checks

Propose action

Think

Monitor

1

Executor Agent

openlibrary-ba3abfb6a, SWE-bench Pro, Deepseek-V3

Step 15th 16

Legend: Data

submit

Fig. 2: An advisor periodically checking the trajectory fails to resolve an issue due to late feedback.

Rule-based

LLM-based

Gate

Fig. 3: Overview of L IVE P LAN workflow. Pro [7] datasets, using five diverse general and reasoning LLMs (three as executors and two as advisors). The comprehensive evaluation confirms the effectiveness of L IVE P LAN in consistently improving the resolution rate of SWE-agent across all models and benchmarks, with consistent gains of up to 15.2% (average: 9.9%). L IVE P LAN not only consistently resolves more GitHub issues than the baselines but also guides the agent more effectively toward solving medium and hard issues. Besides improving the overall resolution rate, L IVE P LAN yields trajectories that are more compliant with the predefined instruction plan in SWE-agent [18], i.e., explore, localize, repair, and validate [33]. Our contributions are: • Technique. We propose L IVE P LAN , an online monitoring and intervention technique that decouples judging from advising for agent trajectory steering. L IVE P LAN pairs a deterministic, rule-based monitor to detect whether and where a trajectory is in trouble without invoking an LLM, and thus without the risk of hallucinating a non-existent problem (§II-A). The LLM advisor is consulted only when the monitor fires and only for a high-level, next-step correction (§II-B). • Empirical Evaluation. We perform a large-scale evaluation to assess the effectiveness of L IVE P LAN and other techniques on two widely used GitHub issue datasets. The study results in 7752 trajectories, 4668 of which were generated by intervention approaches, providing a valuable dataset for future research. We conducted a manual analysis of successful and unsuccessful interventions by L IVE P LAN and other baselines, discussing major root causes for intervention failures and guidelines for future research.

A periodic advisor, similar to SWE-PRM [10], evaluates the trajectory every five steps. During the first intervention after step 5, the executor is advised to continue exploring the repository to locate the buggy file. At step 10, it views the exact bug location. However, before taking any repair action in the next step, the advisor incorrectly concludes that the buggy location has not been found and that the agent should keep exploring, causing the agent to continue exploring without patching for the next five steps. After step 15, the advisor offers a patching suggestion, but the guidance arrives too late: the executor has already decided to terminate the run and resubmits, leaving the issue unresolved. (3) While LLM-as-a-judge offers generalizability, i.e., the LLM draws on its internal knowledge to flag problems that are not necessarily encoded, it comes at the cost of reliability: LLMs are known to be unreliable evaluators that both miss real problems and flag spurious ones [8]. Overcoming this limitation entails a deterministic judge whose approach can flag a broad class of behavioral drifts. We propose L IVE P LAN for online monitoring of agent trajectories to detect behavioral drifts and intervene with corrective steering. L IVE P LAN separates judgment from advising: it relies on a deterministic, rule-based monitor that examines a set of general signals over the trajectory to determine behavioral drifts (§II-A). These signals are general rather than tied to a specific benchmark or repository, so they flag a broad class of behavioral drift. Upon detection of a behavioral drift, L IVE P LAN generates predefined or custom high-level, corrective, next-step advice to steer the trajectory (§II-B). By separating judgment from advising and planning, judging with a deterministic monitor that does not hallucinate a nonexistent problem, and advising only when that monitor fires, L IVE P LAN addresses the limitations of prior work. We implement L IVE P LAN on top of SWE-agent [37] and compare its performance against four baselines and ablated versions on the SWE-bench Verified [12] and SWE-bench

II. L IVE P LAN A PPROACH L IVE P LAN consists of three components (Figure 3): (1) Monitor incrementally constructs process-centric representations of the trajectory, evaluates them against a configurable set of general signals to determine behavioral drifts, and invokes the Advisor upon drift detection (§II-A); (2) Advisor

2

TABLE I. Blocking and non-blocking behavioral drifts detected by L IVE P LAN. Type

Category

Signal

Behavioral Drift

Description

Plan Violation

Skipping phase(s) in L ANGUTORY.

Premature Patching Skip Patching Skip Validation

Patch before Localization. Finalize without producing a patch. Submit before validating the patch.

Oscillation

Repeated back-edges in G RAPHECTORY

Thought Oscillation

Repeated reasoning pattern without meaningful progress. Repeated execution loops, such as selfloops or cyclic action sequences.

Blocking

Action Oscillation

Non-blocking

Long Stagnation

Long same-phase sequence in L ANGUTORY

Prolonged Navigation Prolonged Reproduction Prolonged Patching Prolonged Validation

Spends many steps in navigation. Prolonged bug reproduction. Performs many consecutive edits. Spends excessive effort validating.

Repeated Action

A back edge in G RAPHECTORY

Action Revisit

Repeatedly performs the same action, such as revisiting previously viewed locations.

Algorithm 1 L IVE P LAN Runtime Monitoring and Intervention

returns a predefined advice corresponding to the drift, or prompts an LLM to generate a custom advice based on issue description, trajectory context, Monitor status, and latest advice, if available (§II-B); depending on the latest action and Monitor observation, (3) Intervention Processor interacts with the executor agent for corrective steering (§II-C).

Require: Issue description I, advisor cooldown threshold θc , max consecutive blocking interventions θi , long stagnation threshold θp 1: T ← ∅ ▷ committed trajectory 2: H ← B UILD I NITIAL H ISTORY(I) ▷ executor-visible history 3: G ← ∅; L ← ∅ ▷ G RAPHECTORY and L ANGUTORY monitor states 4: lastT rigger ← 0; lastAdvisor ← 0; latestAdvice ← ∅ 5: blockCount ← 0 ▷ consecutive blocking interventions 6: while task not finished do 7: step ← P ROPOSE S TEP(H) 8: (G′ , L′ ) ← E XTEND M ONITOR S TATE(G, L, step) 9: rules ← C HECK RULES(G′ , L′ , θp ) 10: if rules = ∅ then 11: (T, H) ← E XECUTE A ND R ECORD(step, T, H) 12: (G, L) ← (G′ , L′ ) 13: blockCount ← 0 14: else 15: preDef inedAdvice ← M ONITOR A DVICE(rules) 16: advice ← preDef inedAdvice 17: if |T | − lastAdvisor ≥ θc then ▷ advisor cooldown elapsed 18: recentSteps ← S TEPS S INCE(T, lastT rigger) 19: customAdvice ← C ALL A DVISOR 20: advice ← customAdvice 21: lastAdvisor ← |T | 22: end if 23: if C ONTAIN B LOCKING RULE(rules) and blockCount < θi then 24: H ← A PPENDA DVICE(H, advice) ▷ do not execute action 25: blockCount ← blockCount + 1 ▷ rollback: discard G′ , L′ 26: else 27: (T, H) ← E XECUTE A ND R ECORD(step, T, H) 28: (G, L) ← (G′ , L′ ) 29: H ← A PPENDA DVICE(H, advice) 30: blockCount ← 0 31: end if 32: latestAdvice ← advice 33: lastT rigger ← |T | 34: end if 35: end while 36: function C ALL A DVISOR 37: Input: I, recentSteps, latestAdvice, preDef inedAdvice 38: Return: one LLM-generated next-step advice 39: end function

A. Monitor Algorithm 1 shows the workflow of Monitor along with other components. At each step, after the executor reasons about the next action but before executing it (Line 7), Monitor tentatively updates the trajectory representations and checks them for behavioral drift (Lines 8–9). If none is observed, execution proceeds (Lines 10–13). Otherwise, Monitor consults the Advisor for corrective steering advice (Lines 14–22). A common strategy is to feed an LLM the entire trajectory, or a slice of it, to review and surface issues. Using an LLM as a judge is prone to hallucination and error [8, 30]. More critically, when a single LLM performs both judging and advising, its steering suggestion is subject to its own biased assessment [6, 14, 20, 30]. To mitigate this, recent work proposes rubric-based judging, directing the LLM to score trajectories against predefined signals [29]. While this constrains the judge and reduces hallucination and bias, it does not eliminate the deeper problem: evaluation still depends on the LLM applying the rubric correctly and, crucially, deterministically. Layering a non-deterministic judge onto an already non-deterministic agent thus compounds rather than contains this instability. Also, a single-prompt rubric is often insufficient: reliable judging may require a tool-equipped sub-agent to actively inspect the environment, substantially increasing cost [3]. Monitor uses two process-centric trajectory representations, G RAPHECTORY and L ANGUTORY [17], to generically encode behavioral drifts and detect them via deterministic algorithms. G RAPHECTORY converts the trajectory into an enriched graph, with nodes as distinct agent actions and edges as chronological execution order; L ANGUTORY abstracts low-level actions into a sequence of problem-solving phases. The original G RAPHECTORY treats only actions as nodes, with thought included as a node property. L IVE P LAN augments G RAPHEC -

TORY with thought nodes and edges to also monitor ineffective thinking that manifests as behavioral drift. It also splits the G RAPHECTORY’s localization phase into navigation (locating the potential bug) and reproduction (confirming it), extending both G RAPHECTORY nodes and L ANGUTORY alphabets with these new phases. This lets Monitor and Advisor detect behavioral drifts and issue corrective steering more specifically.

3

view

Reproduction (R)

Validation (V)

…

1

Navigation (N)

Patch (P)

view

/testbed

view

python

4

test_parser.py

/testbed/html.py

2

17 19 21

format str_replace

3

str_replace

view

/testbed/html.py × <str1, str2>

view

<str3, str4>

grep

5

reproduce.py

3

str_replace /testbed/io/ascii/core.py

4

/testbed/io/ascii/core.py –-view_range [128,300]

submit

/testbed/manager.py × <str5, str6>

5

submit

General

reproduce.py

2

python input

4

23

/testbed/manager.py

1

create /testbed/ascii ×

3

parser.py ×

view

python

view /testbed/io

2 22 20 18

grep

/testbed

1

/testbed/test_html.py

view 24

6 …

(a) Repeated Action

(b) Oscillation

Langutory: NNNNNN

Langutory: NRRP

(c) Long Stagnation

(d) Plan Violation

Fig. 4: Examples of behavioral drifts captured by G RAPHECTORY and L ANGUTORY. We define behavioral drift as a condition in a trajectory that deviates from common expectations. Under this definition, deviation from the specified plan, repetitive thoughts or actions that fail to advance the agent, and stagnation in any problemsolving phase are behavioral drifts. While signals defined this way are general, they may be incomplete. To account for that, L IVE P LAN maintains a configurable signal database that users can augment. Furthermore, as we discuss in §II-B, the LLM Advisor can still use its general knowledge to guide corrective steering. Some behavioral drifts are alarming, e.g., skipping critical problem-solving phases, such as patch validation before submission, which may result in an incorrect patch. Therefore, the execution of actions that manifest such drifts should be blocked. Others represent inefficiencies that can be improved but are not necessarily harmful. For example, prolonged navigation through the repository to localize a bug could be due to ineffective reasoning/tooling, or a legitimate need to gather sufficient context to repair a multi-hunk bug. Therefore, execution of the corresponding actions can be allowed, accompanied by corrective steering advice. Table I lists ten blocking and non-blocking behavioral drifts that Monitor can detect. They can be detected in G RAPHEC TORY and L ANGUTORY through generic signals: a back-edge in G RAPHECTORY, regardless of the nature of the action, indicates Repeated Action (Figure 4a); repeated self-loops or multi-node cycles in G RAPHECTORY indicate Thought or Action Oscillation (Figure 4b); a sequence of the same phase in L ANGUTORY, whose length exceeds the stagnation threshold θp , indicates Long Stagnation (Figure 4c); and a missing phase symbol in L ANGUTORY indicates skipping a plan phase, hence Plan Violation (Figure 4d).

Long Stagnation. Phase-specific messages are used for Navigation, Reproduction, Patching, and Validation. • Prolonged Navigation. You have the following options to explore next, which increase your chance to solve the problem: (1) keep exploring the code base as you do, but focus on the relevant code only; (2) create and run a reproduction test to better localize the bug; (3) if you already have enough information, edit the source code to implement the fix. Oscillation. You are repeating action {action} in the last {X} trajectory steps. The observation has likely not changed. Concretely reason if this action can help you resolve the issue. If not, think about a better action in the next step so that you can resolve the problem.

Fig. 5: Examples of predefined advice in L IVE P LAN. When invoked, the LLM-based Advisor receives four inputs: the issue description, the committed trajectory slice since the last advice (to minimize context), the latest advice if available, and the predefined advice for the detected drift. The predefined advice serves as a high-level hint to help it focus, while the recent trajectory remains the primary evidence for grounding the recommendation. Advisor outputs a next-step recommendation, rather than a long-horizon plan, keeping the intervention focused, avoiding over-constraining the executor, and preventing context clutter and degradation [16, 34, 35]. Since even minimal context can increase cost and latency, L IVE P LAN implements a cooling mechanism: Advisor invokes the LLM only if at least θc trajectory steps have elapsed since the last LLM-generated advice (Line 17).

C. Intervention Processor The Intervention Processor component takes the advice from Advisor and, depending on the drift type, performs one of two actions: (1) for a blocking drift, it rejects the culprit action before execution, discards the tentative G RAPHECTORY and L ANGUTORY updates, and appends the advice to the executor history for corrective steering, up to θi consecutive times to avoid stalling execution (Lines 23–25); (2) for a non-blocking drift, or once the blocking-intervention limit is reached, it lets the executor proceed and appends the advice to steer subsequent decisions (Lines 26–31).

B. Advisor L IVE P LAN either provides predefined advice or invokes the LLM-based Advisor to generate custom advice (Lines 14– 22). Predefined advice offers only high-level, problem-agnostic guidance; Figure 5 shows the predefined messages for Prolonged Navigation and Action Oscillation drifts. Although relevant to the detected drift, it may be too general to help the executor recover, so Advisor is also equipped with an LLM to generate advice specific to the problem.

4

TABLE III. Effectiveness on SWE-bench Verified.

TABLE II. Effectiveness on SWE-bench Pro-Python. Model

Method

Advisor

Executor

GPT-5.2 GPT-5.2 GPT-5.2

DeepSeek (V3)

GPT-5.2 GPT-5.2 GPT-5.2 MiniMax-M3 MiniMax-M3 MiniMax-M3

Success Rate (%)

Average Cost

21.76 18.79 (-2.97) 25.00 (+3.24) 28.79 (+7.03) 34.09 (+12.33)

Model Advisor

Executor

0.05 0.23 0.11 0.12 0.15

GPT-5.2 GPT-5.2

DeepSeek (V3) Gemini (2.5-flash)

Vanilla 13.17 SAGE 18.18 (+5.01) Gemini Predefined Advice 18.94 (+5.77) (2.5-flash) Periodic Advisor 26.14 (+12.97) L IVE P LAN 28.41 (+15.24)

0.78 1.16 0.98 0.84 1.04

MiniMax-M3

MiniMax (M2.5)

Vanilla 52.5 SAGE 50.38 (-2.12) Predefined Advice 54.17 (+1.67) Periodic Advisor 54.54 (+2.04) L IVE P LAN 57.95 (+5.45)

0.24 0.34 0.23 0.32 0.24

MiniMax (M2.5)

Vanilla SAGE Predefined Advice Periodic Advisor L IVE P LAN

Method

Success Rate (%)

Average Cost

Vanilla L IVE P LAN

38.20 49.40 (+11.20)

0.04 0.08

Vanilla L IVE P LAN

37.80 48.40 (+10.60)

0.46 0.58

Vanilla L IVE P LAN

74.20 79.20 (+5.00)

0.30 0.25

for all experiments. In the ablated Periodic Advisor variant, Advisor is triggered every five steps, consistent with the SWE-PRM setting for fair comparison. Similarly, we set the cooling period θc to five steps. For long stagnation detection, we set the threshold θp to seven consecutive steps in the same phase, computed from Vanilla SWE-agent runs prior to any intervention, yielding average maximum consecutive phase lengths of 5.64 for resolved instances and 7.61 for unresolved instances across all models. This threshold thus separates normal progress from likely stagnation without being tuned to any intervention method. For blocking behavioral drifts, L IVE P LAN allows at most five consecutive interventions (θi = 5); beyond this, the agent prevents repeated blocking from stalling execution indefinitely. We run SAGE using their provided prompts and pipeline without modification.

III. E XPERIMENTAL E VALUATION L IVE P LAN is pluggable to any open-source ReActbased [38] agent. We build L IVE P LAN on top of SWE-agent as it offers a stable, well-tested implementation. Benchmarks. We evaluate on two repository-level issue resolution benchmarks: SWE-bench Verified [12] and SWE-bench Pro1 [7], a more challenging benchmark designed to be resistant to contamination. Baselines. We compare L IVE P LAN against (1) SWE-agent using its original, predefined plan with the executor LLM, and (2) SAGE [11], which replaces this original plan with one generated by the advisor LLM given a previous trajectory. We also evaluate two ablated variants of L IVE P LAN: (3) Periodic Advisor, which removes Monitor so Advisor periodically evaluates and intervenes on the trajectory, similar to SWEPRM [10] and Wink [25]2 , and (4) Predefined Advice, which removes Advisor and monitors the trajectory using only predefined message interventions, similar to [17]3 . LLMs. We use three representative LLMs as executors: DeepSeek-V3 [15], an open-source general-purpose model; Gemini-2.5-Flash [5], a lightweight reasoning model; and MiniMax-M2.5 [23], a model optimized for coding and agentic workflows. Each executor is paired with a stronger but affordable advisor: GPT-5.2-Codex [26] (medium reasoning effort) pairs with the general-purpose executors DeepSeekV3 and Gemini-2.5-Flash, while MiniMax-M3 [24] pairs with the coding-specialized MiniMax-M2.5. A stronger advisor is a prerequisite for meaningful intervention: an advisor no more capable than the executor offers no guidance beyond what the executor could already reach through its own reasoning. We apply the same executor-advisor pairing across all intervention approaches, so that differences in resolution rate reflect the intervention mechanism rather than advisor capability. Hyperparameters. Following the default configurations of SWE-agent and SWE-bench Pro [7, 33], we use a perinstance cost limit of $2 and a model temperature of 0

A. RQ1: Effectiveness in Improving Resolution Rate We run Vanilla SWE-agent and L IVE P LAN with all advisor and executor LLMs on both benchmarks. To manage experimental costs, we run the ablated versions and SAGE on SWEbench Pro only across all three executor models. We prioritize SWE-bench Pro as it has emerged as the dominant benchmark for long-horizon, agentic software engineering, with SWEbench Verified increasingly retired as a primary measure of progress. Tables II-III summarize the results. 1) Comparison Against Vanilla Runs: L IVE P LAN consistently achieves a higher resolution rate than Vanilla SWE-agent across both benchmarks, demonstrating the effectiveness of real-time monitoring and corrective steering. Gains are greater on SWE-bench Pro, with improvements of +12.33 and +15.24 percentage points for DeepSeekV3 and Gemini-2.5-Flash, respectively, versus +11.20 and +10.60 points on SWE-bench Verified. Even for MiniMaxM2.5, which is a strong model, L IVE P LAN still improves the success rates by +5.45 points on SWE-bench Pro and +5.00 points on SWE-bench Verified. These gains come with only modest cost increases for DeepSeek-V3 and Gemini-2.5-Flash, while simultaneously reducing costs for MiniMax-M2.5. Figure 6 compares the overlap of resolved instances between L IVE P LAN and Vanilla, grouped by difficulty. Black dots indicate the method(s) that resolve the corresponding instances to a bar, and gray dots mark those that do not. Colors indicate instance difficulty. For SWE-bench Verified, we use the difficulty labels provided by the benchmark based on the estimated developer effort to resolve the issue. For SWE-bench Pro, following prior work [3], we estimate difficulty using the number of files modified in the reference patch: easy (1 file),

1

2 These To account for the cost, we use SWE-bench Pro-Python. techniques have no runnable artifacts available, so this ablation serves as the closest reproducible analog. 3 Artifacts for [17] exist but do not implement all L IVE P LAN monitoring rules and signals; we use our ablated version instead to isolate the impact of the intervention technique.

5

MiniMax-M2.5 Gemini-2.5-Flash

DeepSeek-V3

SWE-bench Pro

SWE-bench Pro

SWE-bench Verified

Easy Medium Hard Very Hard

SWE-bench Verified

Deepseek-V3

both and the superiority of L IVE P LAN over similar alternatives: SWE-PRM (every five steps) and Wink (every step) periodically analyze the trajectory and intervene, similar to the Periodic Advisor variant. Such techniques are guaranteed to trigger the Advisor, whereas L IVE P LAN does so when needed. Table IV shows that the Periodic Advisor triggers Advisor in almost all trajectories4 , while L IVE P LAN triggers it as needed. The average number of interventions in Periodic Advisor is also notably higher than L IVE P LAN. As we confirm through manual analysis, the lower performance despite the higher and denser triggering rate is due to the Advisor providing misleading advice. Using DeepSeek-V3 and Gemini-2.5-Flash, the average cost of L IVE P LAN per instance is higher than that of Periodic Advisor, despite triggering Advisor less often. The additional cost reflects longer successful trajectories rather than more expensive interventions (details in §III-B). Compared to the Predefined Advice variant, which is similar to the proposed technique in [17], L IVE P LAN consistently outperforms it with a considerable margin. This is because this variant only provides predefined, high-level messages, which may be insufficient to help the agent recover from behavioral drifts. Since Predefined Advice variant detects a broader set of issues than [17], it is an upper bound on that technique’s performance, i.e., L IVE P LAN’s margin over that technique would be even larger. Figure 7 illustrates the overlap of resolved instances by all approaches. Across most model-benchmark pairs, L IVE P LAN contributes the largest set of uniquely resolved instances, e.g., 73 for Gemini-2.5-Flash and 32 for MiniMax-M2.5 on SWEbench Verified. The distribution of resolved instances suggests that different intervention strategies exhibit complementary strengths across task subsets. Overall, L IVE P LAN offers the strongest balance between effectiveness and cost.

TABLE IV. Comparison of trigger frequency of the Advisor by different techniques. Method

Average Intervention

Trigger Rate (%)

DeepSeek (V3)

Periodic Advisor L IVE P LAN

3.25 2.33

99.6 93.9

Gemini (2.5-flash)

Periodic Advisor L IVE P LAN

7.37 2.99

97.7 93.9

MiniMax (M2.5)

Periodic Advisor L IVE P LAN

7.38 1.47

99.6 83.3

Model

GPT-5.2

MiniMax (M3)

MiniMax-M2.5

Fig. 7: Overlapping resolved instances among techniques.

Fig. 6: Resolution overlap between L IVE P LAN and Vanilla.

Advisor

Gemini-2.5-Flash

Executor

medium (2–3 files), hard (4–10 files), and very hard (> 10 files). Across all model-benchmark pairs, L IVE P LAN consistently resolves a substantial number of instances beyond those already solved by the vanilla. More importantly, these gains are concentrated on harder tasks: on SWE-bench Verified, more than half of the instances uniquely resolved by L IVE P LAN belong to the hard category; on SWE-bench Pro, nearly all additional resolutions fall into the medium and hard categories. These results suggest that online monitoring and steering are particularly beneficial for more challenging and less contamination-prone software engineering tasks. 2) Comparison Against SAGE: L IVE P LAN consistently outperforms SAGE at even lower cost. SAGE’s notable cost overhead comes from running Vanilla SWE-agent once, feeding its entire trajectory to Advisor for re-planning, and reexecuting the agent with the new plan. SAGE even underperforms the Vanilla run under DeepSeek-V3 and MiniMax-M2.5. Although SAGE produces highly specific guidance, such as exact line numbers or concrete code snippets to insert, this detail is often factually incorrect (Figure 1). Executor LLMs, being generally less capable, often cannot critically evaluate or recover from such erroneous instructions and instead follow the hallucinated guidance faithfully, causing task failures. 3) Comparison Against Ablated Versions and Similar Approaches: Removing Monitor or Advisor degrades the performance of L IVE P LAN, demonstrating the effectiveness of

B. RQ2: Detailed Cost Analysis of L IVE P LAN L IVE P LAN’s gains come at minimal cost and overhead. Table V reports that across all settings, the advisor costs only $0.01–$0.06 per instance on average. Rule-based monitoring incurs nearly zero runtime overhead, i.e., a few milliseconds. The Advisor latency is modest relative to the 4

The trajectories in which the Advisor is not triggered are shorter than the five-step threshold due to unsuccessful early termination.

6

75 47

0.78 — — 0.98 0.06 0.0

— 9.6

MiniMax Vanilla 47 MiniMax-M3 (M2.5) L IVE P LAN 47

0.24 — — 0.24 0.01 0.0

— 29.1

21 24

0.04 — — 0.06 0.02 0.0

— 5.8

52 38

0.46 — — 0.55 0.04 0.0

— 8.1

MiniMax Vanilla 70 MiniMax-M3 (M2.5) L IVE P LAN 58

0.30 — — 0.23 0.02 0.0

— 39.5

Vanilla

DeepSeek-V3 MiniMax-M2.5 R U Gemini-2.5-Flash R U R U 0.55 0.55 1.00 0.67 0.63 0.63 0.98 0.73 0.79 0.75 0.99 0.83 0.81 0.77 0.95 0.84

0.54 0.54 0.99 0.66 0.61 0.61 0.97 0.71 0.78 0.73 0.98 0.81 0.84 0.79 0.96 0.85

0.74 0.74 0.95 0.80 0.71 0.71 0.97 0.78 0.89 0.89 0.98 0.92 0.89 0.89 0.97 0.91

0.63 0.63 0.97 0.72 0.65 0.65 0.97 0.73 0.83 0.83 0.96 0.86 0.88 0.88 0.96 0.90

0.85 0.85 0.80 0.83 0.83 0.83 0.80 0.82 0.97 0.95 0.80 0.90 0.97 0.97 0.80 0.91

0.81 0.81 0.81 0.81 0.80 0.80 0.80 0.80 0.94 0.93 0.82 0.89 0.97 0.96 0.80 0.90

1.0 0.9 0.8 0.7 0.6 0.5

Fig. 9: Average plan compliance score for Resolved (R) and Unresolved (U) instances of Vanilla and L IVE P LAN.

100 80

L IVE P LAN significantly reduces blocking signals. Many of the high-risk behavioral drifts, namely Plan Violation and Action/Thought Oscillation, that are prevalent in the final Vanilla trajectories and are notably observed during L IVE P LAN runs do not appear in final L IVE P LAN trajectories. 55.63% and 40.27% of blocking behavioral drifts in Vanilla trajectories under SWE-bench Pro and SWE-bench Verified drops to 34.37% and 13.67% in L IVE P LAN, respectively. As our manual analysis confirms (§III-D), the remaining blocking drifts are due to the executor either being unable to follow the advice or choosing not to, repeating the drift after exceeding the consecutive-intervention threshold θi . Nonblocking signals may remain in completed L IVE P LAN trajectories. This is because they can signal either an inefficiency or a legitimate, prolonged attempt necessary to solve hard problems. These results show that L IVE P LAN improves execution quality by minimizing high-risk behavioral drifts while preserving the flexibility needed for long-horizon tasks. 2) Plan Compliance Analysis: L IVE P LAN’s final trajectories, specifically resolved SWE-bench Pro instances by DeepSeek-V3 and Gemini-2.5-Flash, exhibited Plan Violation, albeit to a lesser extent than Vanilla, motivating deeper plancompliance analysis. Following prior work [18], we measure trajectory adherence to SWE-agent’s repair plan, i.e., Navigate, Reproduce, Patch, and Validate, via four metrics: Plan Phase Compliance (PPC) penalizes skipped phases; Plan Order Compliance (POC) penalizes out-of-order execution; and Plan Phase Fidelity (PPF) penalizes behaviors outside the plan. The overall Plan Compliance (PC) is the geometric mean of PPC, POC, and PPF. PC=1 only when a trajectory executes all and only the specified phases in the correct order. Figure 9 reports average compliance scores for resolved and unresolved instances of Vanilla and L IVE P LAN. Consistent with prior work [18], resolved trajectories achieve higher PC than unresolved ones, indicating that successful repairs mostly followed the intended workflow. L IVE P LAN improves PC mainly through higher PPC and POC, suggesting online intervention helps agents complete the required phases in the intended order. PPF is occasionally lower under L IVE -

60 40 20

lan

0

eP

la

MiniMax-M2.5 0.4 0.0 7.6 0.8 51.1 48.9 6.1 5.3 0.4 1.1 18.2 21.2 42.4 43.9 6.2 1.4 5.0 0.8 45.6 42.4 24.2 21.6 0.2 0.6 51.6 48.0 58.8 46.8

Va nil

lan eP Liv

la

Gemini-2.5-Flash 35.7 32.7 62.0 13.5 27.1 33.1 8.6 5.6 21.1 20.7 10.9 20.7 53.4 60.5 6.2 4.6 54.2 21.0 35.2 31.8 13.6 12.8 8.6 7.8 13.0 14.6 34.8 33.0

Va nil

Va nil

lan

DeepSeek-V3 86.7 59.5 86.7 17.4 21.6 25.8 0.8 1.5 4.5 11.7 1.9 11.4 15.9 26.1 25.4 14.6 50.4 2.0 15.8 22.8 7.8 7.6 1.6 0.6 7.2 14.6 10.2 12.8

la

[B] Plan Violation [B] Oscillation Navigation [N] Long Reproduction Stagnation Patching Validation [N] Repeated Action [B] Plan Violation [B] Oscillation Navigation [N] Long Reproduction Stagnation Patching Validation [N] Repeated Action

eP

GPT-5.2

Liv

DeepSeek Vanilla (V3) L IVE P LAN Gemini Vanilla (2.5-flash) L IVE P LAN

GPT-5.2

PPC POC PPF PC PPC POC PPF PC PPC POC PPF PC PPC POC PPF PC

Score

— 10.9

GPT-5.2

SWE-bench Pro

0.04 — — 0.12 0.04 0.0

LivePlan

16 30

Liv

Pro Verified

Pro

Exec. Adv. Mon. Adv.

DeepSeek Vanilla (V3) L IVE P LAN Gemini Vanilla (2.5-flash) L IVE P LAN

GPT-5.2

Verified

Method Steps Executor

Vanilla

Dataset Advisor

Cost ($) Latency (s)

SWE-bench Verified

Setting

LivePlan

TABLE V. Cost and latency analysis. Exec: Executor; Adv: Advisor; Mon: Monitor. “—” indicates not applicable.

Fig. 8: Behavioral drifts in final trajectories. runtime of long-horizon repository-level repair tasks. Beyond cost, L IVE P LAN often reduces execution effort by decreasing inefficient exploration. For Gemini-2.5-Flash and MiniMaxM2.5, interventions generally shorten trajectories and, in some cases, even reduce the overall cost. In contrast, trajectories become longer for DeepSeek-V3 because L IVE P LAN discourages common shortcuts such as skipping validation [18], encouraging a more complete and reliable repair process. Regardless, the resulting cost overhead remains small. C. RQ3: Process-Centric Analysis of Trajectories In addition to outcome-centric analysis, we conduct three types of automated process-centric analyses to better understand the notable improvement of L IVE P LAN over Vanilla. 1) Prevalence of Triggering Signals: We measure the prevalence of behavioral drifts in completed trajectories of Vanilla and L IVE P LAN (Figure 8), and the percentage of instances in which L IVE P LAN observes each drift during execution (Table VI). Drifts observed by L IVE P LAN may not remain in the final trajectory, since L IVE P LAN steers the agent away from them. On average, 86.16% and 78.33% of Vanilla trajectories in SWE-bench Pro and SWE-bench Verified show behavioral drifts. They are more prevalent in SWEbench Pro, whose more complex problems require longer, drift-prone trajectories. The exact behavioral drifts in Vanilla runs may not be observed in L IVE P LAN. However, Table VI shows that L IVE P LAN observes many similar behavioral drifts during execution and attempts to steer the agent away. Specifically,

7

TABLE VI. Observed behavioral drifts by L IVE P LAN. Setting Dataset

Advisor

Executor

SWE-bench Pro

GPT-5.2 MiniMax-M3

DeepSeek-V3 Gemini-2.5-Flash MiniMax-M2.5

SWE-bench Verified

GPT-5.2 MiniMax-M3

DeepSeek-V3 Gemini-2.5-Flash MiniMax-M2.5

Plan Violation

14.6 15.7 47.8

37.0 78.3 6.8

12.8 37.8 47.0

26.1 31.8 48.7

1.5 5.3 5.3

11.4 20.5 1.1

28.2 7.7 6.2

22.8 34.9 42.4

7.4 13.8 21.2

0.6 8.4 0.6

TABLE VII. Resolution transition matrix relative to Vanilla. R: Resolved, U: Unresolved, DS: DeepSeek-V3, GF: Gemini2.5-Flash, and MM: MiniMax-M2.5.

LCS Similarity LCS Similarity

28.8 59.5 44.1

80.7 43.9 1.9

SWE-bench Verified SWE-bench Pro

0.5 0.0

95.5 73.5 10.3

Validation

Dataset

1.0

11.0 20.8 21.3

Patching

0.5 0.0

Repeated Action

Reproduction

SWE-bench Pro

1.0

Oscillation

Long Stagnation Navigation

R→U ↓ U→R ↑ DS GF MM DS GF MM SAGE 26 12 22 19 28 16 Predefined Advice 16 13 15 25 28 15 Periodic Advisor 16 11 17 35 42 22 L IVE P LAN 2 2 7 33 38 21

Method

SWE-bench Verified L IVE P LAN

DeepSeek-V3

Gemini-2.5-Flash Prefix Phase

MiniMax-M2.5

12

7

7

56 68

28

ments, the post-intervention drop reflects the intervention itself rather than task variation. While we discuss the impact of nondeterminism in detail later (§III-E), these results indicate that L IVE P LAN’s performance gains stem from targeted behavioral corrections rather than run-to-run variation. D. RQ4: Analysis of Improvements and Regressions Table VII reports resolution transitions relative to Vanilla, computed over instances where each method is triggered. L IVE P LAN consistently achieves the fewest R→U regressions while producing many U→R improvements5 , particularly on SWE-bench Pro. These results show that L IVE P LAN not only resolves notable previously unsuccessful instances but also preserves the Vanilla successes. We manually analyzed improvement (U→R) and regression R→U cases to understand root causes. Two authors independently inspected the trajectory, intervention(s), resulting patch, and golden patch to inductively identify root causes, with categories emerging from and refined during analysis rather than defined in advance. The authors then discussed findings to reach consensus on final root-cause labels6 . 1) Improvements (U→R): Our analysis identifies the following reasons for L IVE P LAN’s success in converting previously unsuccessful Vanilla runs: (1) Correcting behavioral drift. Weaker executors, particularly DeepSeek-V3 and Gemini-2.5-Flash, frequently exhibit blocking behavioral drifts (Table VI); L IVE P LAN detects them promptly and redirects execution before they persist, eliminating Oscillations in 104/128 (81.3%) of affected trajectories and Plan Violation cases in 37/72 (51.4%). (2) Refocusing on the repair task. As trajectories lengthen, executors may drift from the main repair task due to accumulated context or intermediate failures (context poisoning [3]); by conditioning on the issue description and recent trajectory steps, Advisor redirects the executor away from

Full Phase

Fig. 10: Trajectory similarity between Vanilla and L IVE P LAN before the first intervention and over complete trajectories. P LAN, which we attribute to Advisor suggesting beneficial outof-plan actions, e.g., additional regression testing to validate the patch. Since these trajectories succeed, the out-of-plan behavior appears productive rather than aimless. 3) Strategy Divergence Analysis: The plan-compliance analysis showed that Advisor interventions affect how closely the executor follows its prescribed plan. Independent of that plan, we next perform an apples-to-apples comparison between Vanilla and L IVE P LAN trajectories to understand whether they are similar before the intervention and how they change after it. We compute normalized longest common subsequence (LCS) similarity between L ANGUTORY of Vanilla (X) and )| L IVE P LAN (Y ) for each instance as 2×|LCS(X,Y . Larger |X|+|Y | values indicate more similar execution behavior. We evaluate similarity on (1) the L ANGUTORY prefixes before the first intervention and (2) the L ANGUTORY of complete trajectories. The former captures divergence from execution nondeterminism, since both runs evolve independently before any intervention. The latter captures the combined effect of nondeterminism and online intervention. Figure 10 reports the results. Prefix similarity is high, often 0.8–0.9, indicating agents initially follow similar strategies despite execution non-determinism. After intervention, similarity consistently decreases, showing that L IVE P LAN redirects the subsequent problem-solving strategy. A paired one-sided Wilcoxon signed-rank test [36] comparing prefix and full-trajectory similarity per instance confirms the significance: in every model-benchmark pair, prefix similarity is significantly higher (median paired difference > 0, p < 1e−10 ). Since prefix and full-trajectory similarities are paired by instance, and difficulty is held fixed across the two measure-

5

Although Periodic Advisor achieves slightly more U→R transitions in some settings, it also introduces substantially more R→U regressions, leading to a lower overall success rate than L IVE P LAN. 6 Identified root causes and corresponding instances are available in artifacts [21].

8

ansible-0fd88717, SWE-bench Pro, Deepseek-V3

Navigation

Steps 1 - 14 Initial incomplete patch 15

Thought: Let’s create a script to test the look up plugin …

cat << ‘EOF’ > /app/test_password_lookup.py {TEST_SCRIPT_1} EOF 16

cat

Thought: There was an issue. Let me rephrase and execute the test script properly to verify the fix

17

18

19 20

echo

python

cat << ‘EOF’ > /app/test_password_lookup.py {TEST_SCRIPT_2} EOF

cat

Thought: Let’s execute the test to verify the fix

⚠Prolonged Validation 23

Thought: Let’s carefully review the ‘run’ method logic

24

Thought: Follow Guidance: edit run() control flow

test_script.py √

view /app/password.txt Observation: salt=... ident=2b

test_password _lookup.py

str_replace password.py::run() + if file_ident is not None and ident is None: + ident = file_ident + elif ident is not None and (..): + changed = True

17 19 18 python

Repeat the same test execution python /app/test_password_lookup.py Observation: ValueError … Thought: fix parsing logic to separate salt and ident str_replace password.py::parser_content() -salt=password[salt_sep+len(salt_slug):] +remain=password[salt_sep+len(salt_slug):] … (Repeat Failed Edit-Test Cycles)

55

…

str_replace

test_password _lookup.py ×

… 25

22

password.py × <str1, str2>

Thought: The changes looks correct. Now let's create a simpler test script that doesn't require mocking the entire Ansible environment. create test_parse_format.py

… 26

55

Thought: Let’s run the test: python test_parse_format.py

Exit

test_password _lookup.py

21

16

str_replace 23

test_script.py

General

Prolonged Validation

Action cycle: edit test_password_lookup.py python test_password_lookup.py

Pause ad-hoc tests. Manually inspect password.py::run() … Adjust the control flow…

Validation

Patch

python

16

Thought: Here's an updated test script that includes the required setup:

python /app/test_password_lookup.py Observation: ValueError

test_password _lookup.py

15

echo “{TEST_CONTENT}” > test_script.py && python test_script.py

Reproduction

Steps 1 - 22 - Initial incomplete patch - Spends 7 consecutive steps editing and rerunning test_passwor_lookup.py (Similar to Vanilla run)

Advisor redirect

test_password _lookup.py ×

22

view

password.py::run()

23

str_replace

password.py::run() √ <str3, str4>

24

create

test_parse_format.py

25

python

test_parse_format.py √

… 32

Submit

Steps 27 - 33 Focused validation + clean-up + submit

Exit (auto submitted)

(a) Vanilla (Unresolved)

(b) LivePlan (Resolved)

Fig. 11: Case study of a U→R transition. Intervention redirects the executor from ineffective validation to successful repair. due to memorization/data contamination [1] rather than proper reasoning, making corrective intervention confusing. (4) Incomplete compliance. The executor may fail to faithfully follow or fully implement a correct advice. (5) Secondarytask distraction. Advice may intentionally redirect the executor toward secondary issues (e.g., environment configuration), distracting it from the primary repair task. In nearly all L IVE P LAN R→Us, the root cause is not incorrect advice or a failure to intervene on time, but rather the executor’s inability to follow the correct guidance. We confirmed the advice’s accuracy by referring to the golden patch. Figure 12 shows such a case (ansible-8127abbc from SWE-bench Pro under MiniMax-M2.5). Monitor detects a prolonged navigation without generating a patch and triggers Advisor (step 14). The Advisor correctly identifies all five key repair elements (GP1–GP5), consistent with the gold patch, and directs the executor to begin editing. The executor follows part of the advice, e.g., modifying GP5 at step 14. Later, it (incorrectly) reasons that the GP3 edit was incorrect, reverting it at step 56, and ultimately submits a partial patch. In contrast, the Vanilla run remains in navigation until step 24, but collects all the context to generate a successful patch. Guidelines for Future Research. Inference-time intervention should be complemented by intervention-aware post-training. While L IVE P LAN identifies when and how to steer execution, successful recovery depends on the executor faithfully carrying out the advice. Future work could train on interventionconditioned trajectories using process-level rewards that encourage executors to satisfy explicit guidance constraints across following steps, since optimizing only for final task success may encourage brittle, under-validated solutions. Posttraining complements this work: L IVE P LAN determines when and what advice to give, while the trained executor learns to reliably translate that advice into a correct solution. Given the execution non-determinism observed here, the evaluation should use repeated paired rollouts to estimate how an intervention changes the probability of resolution, rather than

unproductive exploration and back toward the core problem. (3) Progressive online correction. 182/244 (74.6%) of U→R cases receive multiple interventions, averaging 3.70 across all 244 cases, letting L IVE P LAN continually correct newly emerging drifts rather than relying on a one-time redirection. L IVE P LAN consistently intervenes before recovery becomes difficult. The first intervention occurs within the first third of the trajectory in all 244 U→R cases, leaving sufficient opportunity to steer the remaining execution. Figure 11 illustrates such a case (ansible-0fd88717 from SWE-bench Pro under DeepSeek-V3). Both Vanilla and L IVE P LAN runs initially generate an incomplete patch and then enter validation. As shown in Figure 11a, the Vanilla run gets stuck in ad-hoc test script creation and execution after several test setup failures, drifting away from the main repair task. The trajectory therefore exhibits Prolonged Validation (steps 15–21) and Action Oscillation (self-loop edges at step 18 and 19) before terminating unsuccessfully at step 55. In contrast, Figure 11b shows that L IVE P LAN detects the Prolonged Validation at step 227 and invokes Advisor, which recommends pausing the adhoc tests and instead inspecting and modifying the core logic directly. The executor follows this guidance, edits the correct control flow, performs focused validation, and successfully submits the final patch at step 33. 2) Regressions (R→U): Regressions across all techniques arise from several sources: (1) Misleading or low-quality guidance. Each method fails differently. SAGE may generate an incorrect global plan that redirects the executor toward a non-existent problem (Figure 1) Predefined Advice is often insufficient for task-specific repair. Periodic Advisor may misdiagnose the trajectory state and generate unhelpful guidance (Figure 2). (2) Execution non-determinism. The rerun may diverge from Vanilla execution before any intervention, causing failures absent in the original trajectory. (3) Brittle Vanilla successes. Some Vanilla runs succeed by chance despite limited or missing navigation and validation—likely 7

Note that trajectories are not identical and similar behavioral drifts may occur at different steps.

9

ansible-8127abbc, SWE-bench Pro, MiniMax-M2.5

Navigation

🏅 Golden Patch Diff (relevant excerpt)

Prolonged … navigation, no patch (GP1 ✗, GP2 ✗, GP3 ✗, GP4 ✗, GP5 ✗)

worker.py +display.set_queue(self._final_q) # GP1: queue+detach +self._detach() # GP1: queue+detach +context.CLIARGS = self._cliargs # GP2:restore CLI context +paths = context.CLIARGS.get('collections_path') or [] +paths = paths if is_sequence(paths) else [paths] +init_plugin_loader(paths) # GP3: initialize plugin loader connection/__init__.py +class ConnectionKwargs(t.TypedDict): … # GP5: apply ConnectionKwargs -new_stdin: io.TextIOWrapper | None = None # GP4: remove new_stdin +**kwargs: t.Unpack[ConnectionKwargs] # GP5: apply ConnectionKwargs

Advisor correctly identifies GP1-GP5

LivePlan (Unresolved)

14

… str_replace connection/ __init__.py

16

Reproduction … view connection/ __init__.py

Partial compliance. Let me implement the changes.. (GP5 ✔)

Incorrect reasoning The new_stdin is optional and there is no need to change it (GP4 ✗)

Validation

Patch 56

General

… str_replace

Exit

worker.py

Implements only part of the worker setup (GP1 ✔,GP2 ✔, GP3 ✔, GP4 ✗)

Drops plugin-loader setup keeps only CLI-context initialization. 🚨 (GP3 ✗)

Unresolved 🚨 (GP1 ✔, GP2 ✔, GP3 ✗, GP4 ✗, GP5 ✔)

Fig. 12: Case study of a R→U transition under L IVE P LAN. TABLE VIII. Effectiveness in resolving instances of SWEbench Pro that deterministically pass or fail in Vanilla runs. Model

#Instances

Advisor

Executor

GPT-5.2 GPT-5.2 GPT-5.2

DeepSeek (V3)

GPT-5.2 GPT-5.2 GPT-5.2

Gemini (2.5-flash)

MiniMax-M3 MiniMax-M3 MiniMax-M3

MiniMax (M2.5)

Method

IV. R ELATED W ORK Trajectory analysis and failure diagnosis. Liu et al. [17] propose process-centric trajectory representations via structured graphs and language abstractions. Chen et al. [4] and Liu et al. [19] conduct process-oriented error analysis on GitHub issue-resolution trajectories. TRAIL [8] and MAST [27] develop failure taxonomies, while FALAT [28] and AgentRx [2] offer step-wise failure localization frameworks. These provide post-hoc analysis but do not intervene during execution; L IVE P LAN instead monitors execution continuously and intervenes upon detecting drift, correcting behavior before it propagates. Online steering, planning, and self-correction. SelfRefine [22] and LATS [40] use iterative self-reflection but target stateless reasoning rather than long-horizon agent execution. Planning-based approaches, Plan-and-Act [9], EAGLET [32], and ReCAP [39], decouple planning from execution or learn stronger planners. For software engineering agents, SAGE [11] regenerates a plan from a completed trajectory and reruns the executor; SWE-PRM [10] and Wink [25] periodically invoke an LLM evaluator over recent windows; TrajEval [13] compares fine-grained trajectories against reference patches. L IVE P LAN instead avoids periodic LLM judgment and unnecessary interventions while providing targeted next-step advice, achieving the highest resolution rate across all settings, with substantially fewer Advisor interventions.

Success Rate (%)

222

Vanilla 19.82 SAGE 16.67 (-3.15) Predefined Advice 20.27 (+0.45) Periodic Advisor 23.87 (+4.05) L IVE P LAN 27.03 (+7.21)

213

Vanilla 8.92 SAGE 11.74 (+2.82) Predefined Advice 12.68 (+3.76) Periodic Advisor 18.31 (+9.39) L IVE P LAN 18.78 (+9.86)

234

Vanilla 52.99 SAGE 50.00 (-2.99) Predefined Advice 52.99 (+0.00) Periodic Advisor 54.27 (+1.28) L IVE P LAN 56.41 (+3.42)

drawing conclusions from a single trajectory. E. RQ5: Impact of Non-determinism We re-evaluate all methods on the subset of SWE-bench Pro with deterministic outcomes across two repeated Vanilla runs. This reduces the number of instances but retains the majority of instances: 84%, 80%, and 88% of the study in §III-A for each model. The distribution of instances with consistent outcomes is similar to that of the complete set (20.3% easy, 41.0% medium, 36.8% hard, 1.9% very hard): DeepSeek-V3 (19.4% easy, 40.1% medium, 38.3% hard, 2.3% very hard); Gemini-2.5-Flash (21.1% easy, 39.4% medium, 37.6% hard, 1.9% very hard); and MiniMax-M2.5 (21.8% easy, 40.2% medium, 36.3% hard, 1.7% very hard). Because this subset is defined solely by Vanilla’s consistency, it cannot selectively favor or penalize any. As shown in Table VIII, L IVE P LAN preserves the method ranking from Table II, achieving the highest success rate across all evaluated models, followed by Periodic Advisor, Predefined Advice, and Vanilla. It does so with an advantage comparable in magnitude to that of the full study, indicating that the gain is not an artifact of baseline volatility. We further repeat the SWE-bench Verified Vanilla runs for DeepSeek-V3 (limiting this validation to one model to contain experimental cost), where L IVE P LAN again substantially outperforms Vanilla, improving the success rate from 36.58% to 46.56%, a margin consistent with its full-set gain. These results confirm that the gains of L IVE P LAN are robust to execution nondeterminism rather than favorable run-to-run variation.

V. T HREATS TO THE VALIDITY External Validity. To ensure the generalizability of the results, we evaluate L IVE P LAN across three executors and two advisor LLMs on two widely used benchmarks of realworld GitHub issues. We compare L IVE P LAN with four baselines and ablated versions that resemble related work. To account for execution non-determinism, we further reevaluate L IVE P LAN on a subset of instances with consistent outcomes across two repeated Vanilla executions. Similar improvements are observed over the Vanilla baseline, suggesting that our approach is robust to run-to-run variation. Internal Validity. We investigate the impact of the proposed technique not just on the outcome but also on the process. Construct Validity. Our pipeline is built on peer-reviewed artifacts and validated with well-vetted tools. We analyze the results quantitatively and qualitatively, aligning the two analyses and avoid incorrect metric implementation or measurement.

10

VI. C ONCLUDING R EMARKS

“Plan-and-act: Improving planning of agents for longhorizon tasks,” in International Conference on Machine Learning (ICML), Jul. 2025. [Online]. Available: https://openreview.net/forum?id=ybA4EcMmUZ [10] S. Gandhi, J. Tsay, J. Ganhotra, K. Kate, and Y. Rizk, “When agents go astray: Course-correcting SWE agents with PRMs,” in Workshop on Scaling Environments for Agents (SEA@NeurIPS), Dec. 2025. [Online]. Available: https://openreview.net/forum?id=wyrcoDNaGO [11] H. Hayashi, B. Pang, W. Zhao, Y. Liu, A. Gokul, S. Bansal, C. Xiong, S. Yavuz, and Y. Zhou, “Selfabstraction from grounded experience for plan-guided policy refinement,” arXiv preprint arXiv:2511.05931, 2025. [12] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https:// openreview.net/forum?id=VTF8yNQM66 [13] M. Kim, D. Wang, S. Cui, F. Farmahinifarahani, S. Garg, B. Ray, T. Y. Zhuo, R. Mukherjee, and V. Kumar, “Trajeval: Decomposing code agent trajectories for fine-grained diagnosis,” arXiv preprint arXiv:2603.24631, 2026. [14] X. Lin, Y. Ning, J. Zhang, Y. Dong, Y. Liu, Y. Wu, X. Qi, N. Sun, Y. Shang, K. Wang et al., “Llm-based agents suffer from hallucinations: A survey of taxonomy, methods, and directions,” arXiv preprint arXiv:2509.18970, 2025. [15] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseekv3 technical report,” arXiv preprint arXiv:2412.19437, 2024. [16] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 02 2024. [Online]. Available: https://doi.org/10.1162/tacl a 00638 [17] S. Liu, Y. Chen, R. Krishna, S. Sinha, J. Ganhotra, and R. Jabbarvand, “Process-centric analysis of agentic software systems,” Proc. ACM Program. Lang., vol. 10, no. OOPSLA1, Apr. 2026. [Online]. Available: https: //doi.org/10.1145/3798271 [18] S. Liu, S. Dehghan, J. Ganhotra, M. Hirzel, and R. Jabbarvand, “Evaluating plan compliance in autonomous programming agents,” arXiv preprint arXiv:2604.12147, 2026. [19] S. Liu, F. Liu, L. Li, X. Tan, Y. Zhu, X. Lian, and L. Zhang, “An empirical study on failures in automated issue solving,” arXiv preprint arXiv:2509.13941, 2025. [20] X. Liu, X. Yang, Z. Li, P. Li, and R. He, “Agenthallu: Benchmarking automated hallucination attribution of llm-based agents,” arXiv preprint arXiv:2601.06818, 2026. [21] Liu, Shuyang and Dehghan, Saman and Kim, Jiyoung and Ganhotra, Jatin and Hirzel, Martin and Jabbar-

We propose L IVE P LAN for lightweight monitoring and corrective steering of programming agents. L IVE P LAN relies on two abstract representations of raw trajectories, namely G RAPHECTORY and L ANGUTORY, to monitor and determine behavioral drifts. It benefits from predefined advice relevant to the drift or custom LLM-generated advice when needed. In the next step, we pursue intervention-aware post-training. VII. DATA AVAILABILITY The artifacts of this paper are publicly available at [21]. ACKNOWLEDGMENTS This work is supported by NSF CCF-2238045 and IBMIllinois Discovery Accelerator Institute (IIDAI) grants. R EFERENCES [1] Anthropic, “Introducing claude opus 4.7,” https://www. anthropic.com/news/claude-opus-4-7, 2026. [2] S. Barke, A. Goyal, A. Khare, A. Singh, S. Nath, and C. Bansal, “Agentrx: Diagnosing ai agent failures from execution trajectories,” arXiv preprint arXiv:2602.02475, 2026. [3] Y. Chen, A. Ahmad, Y. Zhou, and R. Jabbarvand, “Unlocking model potentials through adaptive multi-agent scaffolding for efficient issue resolution,” arXiv preprint arXiv:2606.25514, 2026. [4] Z. Chen, W. Ma, and L. Jiang, “Beyond final code: A process-oriented error analysis of software development agents in real-world github scenarios,” arXiv preprint arXiv:2503.12374, 2025. [5] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen et al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” arXiv preprint arXiv:2507.06261, 2025. [6] H. Deng, H. Zhang, J. Ou, and C. Feng, “Can llm be a good path planner based on prompt engineering? mitigating the hallucination for path planning,” in International Conference on Intelligent Computing. Springer, 2025, pp. 3–15. [7] X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler, “SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks?” Sep. 2025. [Online]. Available: https://arxiv.org/abs/2509.16941 [8] D. Deshpande, V. Gangal, H. Mehta, J. Krishnan, A. Kannappan, and R. Qian, “Trail: Trace reasoning and agentic issue localization,” arXiv preprint arXiv:2505.08638, 2025. [9] L. E. Erdogan, N. Lee, S. Kim, S. Moon, H. Furuta, G. Anumanchipalli, K. Keutzer, and A. Gholami,

11

vand, Reyhaneh, “Artifact repository,” https://github.com/ W. Hu, M. Schroder, B. Mutlu, D. Song, and R. D. Intelligent-CAT-Lab/Agent-Planner.git, 2026. Nowak, “The long-horizon task mirage? diagnosing [22] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, where and why agentic systems break,” arXiv preprint S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, arXiv:2604.11978, 2026. Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, [36] F. Wilcoxon, “Individual comparisons by ranking S. Welleck, A. Yazdanbakhsh, and P. Clark, methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, “Self-refine: Iterative refinement with self-feedback,” in 1945. [Online]. Available: http://www.jstor.org/stable/ Conference on Neural Information Processing Systems 3001968 (NeurIPS), Dec. 2023. [Online]. Available: https: [37] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, //proceedings.neurips.cc/paper files/paper/2023/hash/ K. Narasimhan, and O. Press, “Swe-agent: agent91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference. computer interfaces enable automated software engineerhtml ing,” in Proceedings of the 38th International Conference [23] MiniMax, “Minimax m2.5,” https://www.minimax.io/ on Neural Information Processing Systems, ser. NIPS models/text, 2025. ’24. Red Hook, NY, USA: Curran Associates Inc., 2024. [24] ——, “Minimax m3,” https://www.minimax.io/blog/ [38] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. minimax-m3, 2026. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning [25] R. Nanda, C. Maddila, S. Jha, E. M. Khan, M. Paltenghi, and acting in language models,” in International and S. Chandra, “Wink: Recovering from misbehaviors in Conference on Learning Representations (ICLR), May coding agents,” arXiv preprint arXiv:2602.17037, 2026. 2023. [Online]. Available: https://openreview.net/forum? [26] OpenAI, “Introducing gpt-5.2-codex,” https://openai. id=WE vluYUL-X [39] Z. Zhang, T. Chen, W. Xu, A. Pentland, and com/index/introducing-gpt-5-2-codex/, 2025. J. Pei, “ReCAP: Recursive context-aware reasoning [27] M. Z. Pan, M. Cemri, L. A. Agrawal, S. Yang, and planning for large language model agents,” B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, in Conference on Neural Information Processing K. Ramchandran, D. Klein, J. E. Gonzalez, M. Zaharia, Systems (NeurIPS), Dec. 2025. [Online]. Available: and I. Stoica, “Why do multiagent systems fail?” in https://openreview.net/forum?id=r2ykUnzuGt ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025. [Online]. Available: [40] A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y.-X. Wang, “Language agent tree search unifies https://openreview.net/forum?id=wM521FqPvI reasoning, acting, and planning in language models,” in [28] M. N. Rafi, M. Ahasanuzzaman, D. J. Kim, Z. Wang, International Conference on Machine Learning (ICML), and T.-H. Chen, “Falat: Tracing failures in llm agent traJul. 2024, pp. 62 138–62 160. [Online]. Available: jectories via dependency-guided search,” arXiv preprint https://proceedings.mlr.press/v235/zhou24r.html arXiv:2606.00765, 2026. [29] D. Rao and C. Callison-Burch, “Autorubric: Unifying rubric-based llm evaluation,” arXiv preprint arXiv:2603.00077, 2026. [30] M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. Bowman, E. Durmus, Z. Hatfield-Dodds, S. Johnston, S. Kravec et al., “Towards understanding sycophancy in language models,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 110–144. [31] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” Advances in neural information processing systems, vol. 36, pp. 8634–8652, 2023. [32] S. Si, H. Zhao, K. Luo, G. Chen, F. Qi, M. Zhang, B. Chang, and M. Sun, “A goal without a plan is just a wish: Efficient and effective global planner training for long-horizon agent tasks,” Oct. 2025. [Online]. Available: https://arxiv.org/abs/2510.05608 [33] SWE-agent, “Swe-agent documentation,” https://swe-agent.com/latest/usage/batch mode/, 2025. [34] Teresa Torres, “Context rot: Why ai gets worse the longer you chat,” https://www.producttalk.org/ context-rot/, 2026. [35] X. J. Wang, H. Bai, Y. Sun, H. Wang, S. Zhang,

12

Related documents

• Terminal Agents: A Survey of AI Agents in Command-Line Environments#659053
• Terminal Agents: A Survey of AI Agents in Command-Line Environments#656917
• Terminal Agents: A Survey of AI Agents in Command-Line Environments#480138
• Elements of a Distributed Programming System#501877
• SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systems#1116311