CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents

CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents Wuya Chen1,∗ and Yihao Yang2 and Yang Cao2 and Yue Lin1 {chenwuya@corp.netease.com, 13590078189@163.com} {yihao786, cy1835208239}@gmail.com gzlinyue@corp.netease.com 1 Netease Guangzhou AI Lab 2 Independent Researcher

Abstract

observation on a concrete instance: a 30B OpenHands agent on SWE-Bench Verified (Jimenez et al., 2023) averages 23 rounds and 631K tokens per resolved issue, with a large fraction consumed by grep, glob, and view_file calls issued in ambiguous exploration loops—often 20+ rounds in a wrong direction before the agent gives up. Can a specialized retrieval submodule shorten this exploration phase? Two hypotheses have not been jointly tested. H1 (effectiveness): better filelevel retrieval steers the agent past dead-end explorations and raises the resolve rate. H2 (efficiency): the agent resolves roughly the same set of issues but with a shorter rollout. Distinguishing them requires evaluating a retriever both on its intrinsic quality and on its downstream effect within the same agent stack—a comparison absent from prior open code retrievers, from lexical (BM25 (Robertson and Zaragoza, 2009)) to static dense encoders (Feng et al., 2020; Guo et al., 2021; Jina AI, 2024). We train CodeGrep, a 14B retrieval agent optimized end-to-end with GRPO (Shao et al., 2024), and evaluate it against BM25 and Jina on SWEBench Verified. Neither H1 nor H2 holds uniformly: retrieval quality translates downstream through a precision threshold, not a linear payoff. Below the threshold, BM25 (precision 0.375) degrades the agent; near it, Jina (0.445) is neutral; above it, CodeGrep (0.677) cuts 15% rounds and 19% tokens on resolved instances, while adding a small but reproducibly positive +1.2pp to the resolve rate. Three engineering ingredients enable this study. First, CATM (Code Agent Trajectory Mining) mines relevance labels from 67K open-source OpenHands trajectories without human annotation (§3.4). Second, a Docker-free Git-worktree sandbox reduces per-rollout environment setup from minutes to milliseconds, making multi-turn agent RL tractable on a single 8×B200 node (§3.3).

arXiv:2608.05886v1 [cs.SE] 6 Aug 2026

Modern LLM coding agents—Claude Code, OpenHands—share a common inefficiency: they spend most of their token budget finding the file to patch, not patching it. On SWEBench Verified, a 30B OpenHands agent averages 23 rounds and 631K tokens per resolved issue, dominated by grep, glob, and view_file calls. No open, systematically-studied agentstyle code retriever exists at scale to shorten this phase. We introduce CodeGrep, a 14B retrieval agent trained end-to-end with GRPO to issue multi-turn parallel grep/glob/read tool calls and return candidate files to a frozen downstream coding agent. On all 500 SWE-Bench Verified instances, CodeGrep delivers a small but reproducibly positive resolve-rate lift (+1.2pp: 25.8% → 27.0%) alongside a much larger efficiency gain (−15% rounds, −19% tokens on resolved instances). Across three retrievers, downstream utility follows a precision threshold: BM25 (precision 0.375) degrades the agent, Jina (0.445) is neutral, and only CodeGrep (0.677) crosses the threshold at which retrieval begins to buy efficiency. To enable this study, we mine supervision from 67K open-source agent trajectories (CATM) and build a Gitworktree environment for multi-turn agent RL. In our setting, applying the efficiency signal at the advantage layer, rather than the reward layer, holds KL drift to one-third and translates cleanly into downstream efficiency. We will release the model, training pipeline, RL environment, and evaluation harnesses.

1

Introduction

Modern LLM coding agents—general-purpose assistants like Claude Code (Anthropic, 2025) and open GitHub-issue-solving frameworks like OpenHands (Wang et al., 2024)—share a common inefficiency in how they allocate their token budget: much of it is spent finding the file to patch, not writing the patch itself. In this work, we ground this 1

Third, a systematic reward study across three trained iterations (v1 →v2 →v3 ) yields two observations about GRPO reward design: in our setting, applying an efficiency signal at the advantage layer is markedly more stable than at the reward layer, and removing an auxiliary component absent from the downstream tool improves both training stability and downstream efficiency (§3.5).

of sequential calls per issue. For search specifically, Search-R1 (Jin et al., 2025) and ReasonIR (Shao et al., 2025) train LLMs to use search engines with RL, but target indexed web corpora rather than repository file systems. Reinforcement learning for tool policies. GRPO (Shao et al., 2024) has been adapted to multi-turn tool use in ToRL (Li et al., 2025), RAGEN (Wang et al., 2025), and concurrent Cognition SWE-grep (Pan et al., 2025). A recurring difficulty is reward-length exploitation when the reward mixes task accuracy with efficiency signals. Our v1 →v2 →v3 study (§3.5) traces one manifestation and its resolution: moving the efficiency signal from the reward to the advantage layer.

Contributions. • CodeGrep, an open 14B agent-style code retriever trained end-to-end with GRPO; injected into a frozen OpenHands agent, it cuts rounds by 15% and tokens by 19% on resolved SWEBench Verified instances while lifting resolve rate by +1.2pp.

Trajectory mining as supervision. Learning “what is relevant” from past agent reads is behavioural supervision with a long history in classical IR. LRAT (Zhou et al., 2026) applied this to web-search agents with one-to-one browserelevance pairs. CATM (§3.4) adapts the template to code retrievers that emit multiple parallel reads per turn, requiring one-to-many attribution and pertrajectory aggregation (Table 4); our source data are open-source OpenHands trajectories (Nebius AI, 2025).

• A precision-threshold characterisation of retrieval–agent coupling: across three retrievers (BM25 0.375, Jina 0.445, CodeGrep 0.677), downstream utility exhibits three regimes—hurts, neutral, buys efficiency (§5.1). • A reward-design study showing that (i) reward-layer multiplicative scaling of a toolcall penalty triples GRPO policy drift vs. advantage-layer scaling (KL 0.31 vs. 0.09); (ii) in our final iteration, dropping an auxiliary line-range component from the base reward improves both training stability and downstream efficiency (§3.5). • Full open release: model, CATM pipeline, RL environment, and all evaluation harnesses.

2

3

Method

3.1

System Overview

The system runs in two decoupled phases (Fig. 1). During training, CodeGrep is optimized with GRPO in a custom RL environment (§3.3) against relevance labels mined by CATM (§3.4). During inference, the trained CodeGrep takes the issue description, emits a short list of candidate files, and those files are injected into the prompt of a frozen OpenHands downstream agent. Only the retriever is trained; the downstream agent is untouched, so any downstream measurement (§4) is unambiguously attributable to what the retriever contributes to the prompt.

Related Work

Code retrieval for SWE-Bench-style agents. Static retrievers—dense code encoders (Feng et al., 2020; Guo et al., 2021, 2022; Jina AI, 2024) and lexical baselines such as BM25 (Robertson and Zaragoza, 2009)—struggle when issue vocabulary does not overlap with the target code, a common failure mode in GitHub bug reports (§4.2). Pipeline systems such as Agentless (Xia et al., 2024), Moatless (Örwall, 2024), and SWE-Fixer (Xie et al., 2025) pair lightweight retrieval with LLM-based reranking, but keep the retrieval component nontrainable.

3.2

CodeGrep: The Retrieval Model

CodeGrep is a 14B-parameter retriever built on Qwen3-14B-Instruct (Yang et al., 2025). It exposes three read-only primitives—grep (regex search), glob (path matching), and read (file contents)— mirroring how a developer navigates an unfamiliar codebase. At each turn it emits up to 8 tool calls executed concurrently; observations are appended

Agent-style search and tool use. Building on ReAct (Yao et al., 2023) and Toolformer (Schick et al., 2023), coding agents such as OpenHands (Wang et al., 2024) and SWE-agent (Yang et al., 2024) operate over rich tool interfaces at the cost of dozens 2

Training

Inference

Agent Trajectories (67K) GRPO Training (CodeGrep, 14B)

SWE-Bench Issue CodeGrep (14B) weights

grep / glob / read files

SWE-Bench Agent

CodeGrep Checkpoint

Qwen3-30B, frozen

Code Patch Figure 1: System overview. Training (blue) produces CodeGrep weights from open-source agent trajectories via GRPO. Inference (orange) uses the trained CodeGrep as a retrieval submodule whose output is injected into a frozen downstream agent’s prompt.

to the context, and exploration terminates on a final answer or after 4 turns (3 exploration + 1 answer, 24 effective reads). The answer conforms to a fixed JSON schema:

limit, and per-tool timeouts. Environment interaction per rollout drops from minutes to milliseconds, making the training runs in §3.6 feasible on a single 8×B200 node. Full plumbing, rollout scheduler, and the three-layer architecture diagram are in Appendix A.

<answer>{"files": [. . . ], "line_ranges": [. . . ]}</answer>

the line_ranges field is dead information downstream (the OpenHands editing tool consumes only file paths), so we drop it from the training objective in v3 (§3.5). 3.3

3.4

Training Data: CATM (Code Agent Trajectory Mining)

Ground-truth problem and fix. The natural “relevant files” label—files touched by the SWEBench gold patch—is incomplete: it omits auxiliary files that must be read to understand the fix but are not themselves edited. Directly optimizing against the patch signal therefore biases the retriever away from the behaviour we want. We instead adopt a behavioural notion of relevance: a file is relevant if some past agent, while solving the issue, opened it and produced non-trivial reasoning grounded in its contents. CATM (Code Agent Trajectory Mining) instantiates this notion for code retrieval and for agents that emit parallel tool calls; a related notion has been used by LRAT (Zhou et al., 2026) in the web-search setting (see Appendix Table 4 for a detailed comparison).

RL Environment: A Lightweight Multi-turn Tool Sandbox

An RL environment for CodeGrep must expose the target repository at a specific commit so that grep/glob/read calls can execute and their observations feed back into the rollout. The default of reusing official SWE-Bench Docker images is intractable at single-node training scale: each image is 1–3 GB, pulls take minutes, and thousands of them need terabyte-scale disk. Because CodeGrep training only reads code, the Python environment inside each image is dead weight. We replace Docker with a worktree-based sandbox whose per-rollout setup completes in milliseconds. From any SWE-Bench-style dataset we extract unique (repo, commit) pairs; for each repo we keep a single git clone –bare and use git worktree add to instantiate one lightweight tree per commit (disk drops from N · repo_size to repo_size + N · worktree_size, with no network I/O). At rollout time, grep/glob/read run as native subprocesses inside the target worktree, under path-traversal protection, a 64-way concurrency

Three-stage pipeline. CATM operates on 67,074 open-source OpenHands trajectories (Nebius AI, 2025). Stage 1 (mining): extract every file-read tool call (str_replace_editor with view), normalize paths, and discard directories, documentation (README.md, issue.md), and extensionless files. For each surviving file f record the following assistant message as the post-reasoning, with 3

token length l(f ). Stage 2 (judge filtering): an LLM judge classifies each file’s post-reasoning as RELEVANT or NOT_RELEVANT; we adopt a conservative bias treating any non-explicit-NOT_RELEVANT output as RELEVANT. Stage 3 (intensity-aware weighting): following LRAT (Zhou et al., 2026), each file receives an exponential-saturation score of its reasoning length,
w̃(f ) = 1 − exp − ln 2 · l(f )/β , (1) w(f ) = w̃(f )/µraw ,

Rollout-level efficiency signal. Let Ctotal be the total tool calls a rollout issues and T its number of turns. The average tool calls per turn is c̄ := Ctotal /T

(3)

with the 8 from the per-turn parallel-call budget (§3.2). We treat c̄ = 4 as the natural saturation point. Iteration v1 : reward-layer efficiency scaling. Average the two component scores and multiply by a reward-level efficiency scale:

(2)

where β is the median l(f ) and µraw is the mean of w̃; the normalization gives E[w] ≈ 1 so a single threshold generalizes across issues. Ground truth structure. CATM-mined files enter the training signal not as an auxiliary recall term but through the ground-truth set itself. For each issue x, the reward’s target set is G(x) = Gpatch (x) ∪ {f ∈ L(x) : wf ≥ 0.15}, where Gpatch (x) are SWE-Bench gold-patch files and L(x) is the CATM-mined set. The weights act as a noise filter; every surviving file is a hard positive in the Fβ reward, on equal footing with patch files. Line ranges from the patch are used as an auxiliary scoring input for v1 /v2 and dropped in v3 (§3.5). The pipeline yields 31,977 effective training samples (47.7% retention); the discarded remainder is dominated by misdirected reads and documentation lookups, confirming that judge filtering is necessary. 3.5

(T ≥1, 0≤c̄≤8),

Rv1 = 21 Fβfile + Fβlr · σ v1 (c̄)

(4)

1 ∈ (0, 1]. max(1, c̄/4)

(5)

σ v1 (c̄) =

σ v1 is piecewise: no penalty for c̄ ≤ 4, a 1/(c̄/4) discount above. Trained v1 checkpoints exhibit two coupled failure modes: policy drift is elevated and per-instance downstream efficiency is negligible. Section 5.2 provides the numerical diagnosis. Iteration v2 : move scaling to the advantage layer. Keep the base score, drop the reward-level scale to a hard mask on degenerate rollouts, and move the discount into the GRPO advantage estimator:
Rv2 = 21 Fβfile + Fβlr · 1c̄>0 Avi 2 = Ai · s(c̄i ) ,

(6)

p min(c̄/4, 1). (7) The square root softens the discount: rather than penalizing low-c̄ rollouts linearly, s(c̄) preserves a non-trivial fraction of the gradient signal even at c̄ = 1 (s = 0.50), while flattening once c̄ passes the half-budget saturation point c̄ = 4 (s = 1). Two properties motivate the split: (i) the group ranking of raw rewards is preserved so GRPO’s intra-group comparison stays anchored to task performance; (ii) the discount applies at the gradient step for a specific rollout, not to other group members’ rewards. v2 resolves the drift symptom but develops a new one (length exploitation), taken up in §5.2.

Reward Design and Three Iterations

Let x be an issue and P(x) the file set predicted by a rollout. The target G(x) is the CATM merge (§3.4). Component scores. Two precision-biased Fβ statistics (β = 0.5) appear across the iterations. File-level Fβfile (Pf , Gf ) is a standard Fβ over predicted and target files (with the natural degenerate cases: 1 if both are empty, 0 if exactly one is empty). The precision bias mirrors the downstream constraint: false positives pollute the agent’s context (§4.3), while a miss is usually recoverable. Line-range Fβlr (Pr , Gr ) is defined analogously, but matching predicted against gold ranges: a prediction p matches a gold g iff they share a filename and p covers at least 50% of g; matching is greedy and one-to-one. Under merge mode, predicted ranges for CATM-only files are dropped before scoring. The Fβ formula, edge cases, and matching algorithm are in Appendix B.

s(c̄) =

Iteration v3 : remove line-range from the base reward. Drop the line-range component entirely: Rv3 = Fβfile (Pf , Gf ) · 1c̄>0

(8)

with the same s(c̄) advantage scaling (Eq. 7). Rationale: (i) Interface mismatch: the downstream 4

Table 1: Summary of the three reward-design iterations. All variants share the same target G and c̄ signal (Eq. 3). Component Base task score Reward-layer scale Advantage-layer scale

v1 1 (Fβfile +Fβlr ) 2 1 max(1,c̄/4)

—

v2

resolve rate plus efficiency measures (average rounds and total tokens on resolved instances; average rounds on unresolved). Without any retrieval, this baseline resolves 25.8% of instances at 23.0 rounds and 631K tokens per resolved issue—within one standard deviation of the publicly reported 25.2 ± 0.7% for the same configuration (Nebius AI, 2025), reproducing the reference number all subsequent comparisons anchor to. We compare six configurations: baseline, BM25 (Appendix E), Jina-1.5B (Jina AI, 2024) (dense embedding retriever), and the three CodeGrep iterations v1 /v2 /v3 . Each retriever is evaluated first on its own retrieval quality (§4.2), then on downstream injection (§4.3).

v3

= v1 Ffile β 1c̄>0 1c̄>0 s(c̄) = v2

editor has no view_range argument, so predicted line ranges are dead information at inference time. (ii) Systematic label bias: CATM-only entries have no gold line ranges by construction, so Fβlr = 0 for a large fraction of training data. The concurrent Cognition system (Pan et al., 2025) retains a line-range term; a component-level comparison (Appendix B, Table 6) is necessarily partial as their reward formulation is not fully disclosed. Table 1 summarizes our three iterations. 3.6

4.2

The internal evaluation set is constructed from heldout swe-rebench instances with CATM-mined labels, subsequently audited by senior software engineers on our team to filter cases in which the LLM judge over-accepts marginally-relevant reads. Each retriever is scored with Fβ (β=0.5), together with file-level precision and recall. We choose β<1 because a false positive (an irrelevant file injected into the downstream agent) is more costly than a false negative: the former inflates the agent’s context and dilutes attention with distractor code, whereas the latter can often be recovered by the agent’s own tool calls. Table 2 consolidates the five retrievers. Three observations follow. (i) CodeGrep dominates both BM25 and Jina, with the gap driven primarily by precision. v3 ’s mean Fβ is 1.6× BM25’s and 1.35× Jina’s, and the three retrievers form a strict precision ordering: BM25 (0.375) < Jina (0.445) < CodeGrep v3 (0.677). The gap is even more pronounced in the tail: CodeGrep v3 produces high-quality retrievals (Fβ ≥0.8) on 43.0% of instances, a 6.1× increase over both BM25 and Jina (each at 7.0%)—the injection regime that matters most for downstream token savings. (ii) Among CodeGrep variants, v3 Paretodominates on both quality and cost. Median Fβ improves to 0.714 (+28% over v1 and v2 ), the Fβ ≥0.8 rate reaches 43.0% (+36% relative to v2 ), and the mean turn count decreases monotonically along the iteration sequence 3.8 → 2.9 → 2.3. (iii) v1 and v2 are quality-comparable in aggregate. Their median Fβ coincide at 0.556 and their recall values differ by less than 0.003; the mean Fβ gap (0.562 vs. 0.526) is consistent with v2 ’s advantage-layer efficiency signal (§5.2) trading a small amount of retrieval quality for a substantially

Training Recipe

We train CodeGrep with GRPO on top of msswift (Zhao et al., 2024) on a single 8×B200 node, using LoRA (Hu et al., 2022) plus optimizer offload and vLLM-colocated rollout to fit the 14B run on one node. End-to-end wall-clock is ∼ 27 hours per run; we early-stop at step 897 of a planned 2000step budget. Full hyperparameters (LoRA rank, batch/rollout config, LR schedule, sampling) are in Appendix A.4. Training dynamics. Fig. 2 tracks reward, KLto-reference, and average rollout length across v1 /v2 /v3 . Three signatures stand out: v3 converges highest on reward (0.60–0.65 vs. 0.45–0.48 for v1 /v2 ); v1 ’s KL drifts to ∼ 0.31 by step 900 while v2 /v3 stay at 0.09/0.15; average tool-use turns drop to ∼ 2.1 under v3 and remain there, while v1 /v2 rebound to ∼ 2.6 after step 500. Together these are the training-side evidence for the reward-design principles unpacked in §5.2. Additional diagnostics (completion length, gradient norm, clipping ratio) are in Appendix D.

4

Experiments

4.1

Setup

Retrieval evaluation

We evaluate on SWE-Bench Verified (Jimenez et al., 2023) (500 instances), using OpenHands (Wang et al., 2024) with Qwen3-30B-A3BInstruct-2507 (Yang et al., 2025) as the downstream agent (temperature 0, 100-round max). We report 5

Figure 2: Training dynamics of the three reward-design iterations. (a) Reward: v3 climbs faster and levels off higher; v1 and v2 plateau together. (b) KL-to-reference: v1 ’s reward-layer scaling drives KL to ∼ 0.31; v2 /v3 hold at 0.09/0.15. (c) Tool-use turns: v3 stabilizes at ∼ 2.1; v1 and v2 rebound to ∼ 2.6 after step 500. Table 2: Retrieval quality on the internal evaluation set. The “Turns” column is specific to the agentic retriever and does not apply to non-agentic baselines. Retriever

Fβ mean

Fβ median

Precision

Recall

Fβ ≥ 0.8

Turns

BM25 Jina-1.5B CodeGrep v1 CodeGrep v2 CodeGrep v3

0.359 0.427 0.562 0.526 0.576

0.455 0.500 0.556 0.556 0.714

0.375 0.445 0.641 0.589 0.677

0.386 0.468 0.486 0.483 0.435

7.0% 7.0% 36.7% 31.7% 43.0%

– – 3.8 2.9 2.3

more stable training run (§3.5). The line-range component does not impede file localization; rather, it consumes optimization capacity along a direction the downstream agent never consumes (§5.2).

v1 hits 27.0% resolve but only −1.3% rounds and −0.6% tokens—essentially flat; v2 sits between, reaching 26.6% resolve with −7% rounds and −7% tokens.

4.3

5

Downstream evaluation

Analysis

We now unpack two non-obvious patterns visible across Tables 2–3: (i) how retrieval quality translates to downstream utility via a precision threshold, and (ii) how the three reward iterations induce three qualitatively distinct downstream regimes. A methodological caveat on efficiency accounting follows in §5.3.

We now inject each retriever’s output into the frozen OpenHands agent and evaluate end-to-end. Table 3 consolidates the six configurations. Four observations follow. (i) v3 delivers a small but reproducibly positive resolve-rate lift alongside a much larger efficiency dividend. Resolve rate lifts +1.2pp (25.8% → 27.0%), while resolved rounds drop 23.0 → 19.6 (−15%) and resolved tokens 631K→ 514K (−19%). (ii) BM25 injection degrades downstream. Resolve drops 0.6pp and resolved tokens inflate 21% (631K→ 763K); within the 94 instances both configs resolve, BM25 rollouts spend 6.6% more rounds and 38.6% more tokens. (iii) Jina injection is roughly neutral. Resolve rate is identical to baseline (25.8%) and resolved tokens drop 7% (631K→ 587K)—a modest efficiency shift with no resolve lift, placing Jina between BM25 and CodeGrep. (iv) v1 matches v3 ’s resolve rate but not its efficiency.

5.1

From retrieval quality to downstream utility: a precision threshold

A joint reading of Tables 2 and 3 reveals that retrieval quality does not translate linearly into downstream utility. Instead, we observe a monotone precision gradient partitioned into three regimes. (i) Below threshold, retrieval hurts. BM25, at file-precision 0.375, lowers resolve rate by 0.6pp and inflates resolved tokens by 21% relative to baseline; restricted to the 94 instances both configurations resolve, BM25 rollouts spend 6.6% more 6

Table 3: Downstream comparison of the baseline, BM25 top-2, Jina-1.5B top-2, and CodeGrep v1 /v2 /v3 on SWEBench Verified. Efficiency columns (Resolved r., Resolved tok., Unresolved r.) are averaged over each configuration’s own resolved / unresolved set. Config

Resolve

Resolved r.

Resolved tok.

Unresolved r.

Baseline BM25 Jina-1.5B CodeGrep v1 CodeGrep v2 CodeGrep v3

25.8% 25.2% 25.8% 27.0% 26.6% 27.0%

23.0 22.9 23.2 22.7 21.4 19.6

631K 763K 587K 627K 584K 514K

32.0 29.7 27.8 26.2 26.4 27.5

rounds and 38.6% more tokens. Low-precision candidates function as distractors from which the agent must recover. (ii) Near threshold, injection is roughly neutral. A dense embedding retriever (Jina-1.5B) at precision 0.445 leaves the resolve rate at baseline (25.8%) and yields only a modest efficiency shift (−7% resolved tokens). The direction of change is consistent with CodeGrep, but the magnitude is an order of magnitude smaller: a moderately better retriever than BM25 is insufficient to unlock a downstream dividend. (iii) Above threshold, marginal quality gains are absorbed by rollout efficiency. Both CodeGrep variants already lift resolve rate above baseline (§4.3); comparing further along the CodeGrep axis, v2 → v3 improves median Fβ by +28% (0.556 → 0.714) and the Fβ ≥0.8 rate by +36%, yet the resolve rate advances by only +0.4pp while resolved rounds and tokens fall 8% and 12% respectively. Once the retriever is precise enough to be net-useful, additional precision no longer expands the resolvable set; it is absorbed as rollout compression. Two mechanisms are consistent with this gradient: (a) file localization is not the binding bottleneck on resolve rate—the downstream agent already recovers much of the task-relevant context through its own grep/view tools on the instances it can resolve, and on the instances it cannot, the failure is typically in patch synthesis rather than in locating the right files; sharper retrieval therefore compresses rollouts rather than enlarging the resolvable set; (b) the marginal harm of a false positive is comparable in magnitude to the marginal benefit of a true positive, so a low-precision retriever can be net-negative in aggregate. Takeaway. Retrieval quality translates to downstream performance through a precision threshold

rather than a linear payoff. The three data points— BM25 (0.375, hurts), Jina (0.445, neutral), and CodeGrep (0.677, buys efficiency)—trace a monotone gradient, placing the transition from neutral to net-positive somewhere between precision 0.45 and 0.68. Whether the exact threshold transfers to other downstream agents remains open (§7). 5.2

Reward iterations: three failure modes, three regimes

We now unpack the failure modes underlying the reward transitions introduced in §3.5. Evidence is drawn from the training curves (Fig. 2, Appendix D) and Tables 2–3. v1 → v2 : reward-layer scaling destabilises training and fails to translate into downstream efficiency. (i) Distorted advantage estimates. GRPO advantages are group-relative, so multiplicative scaling on the reward reshapes intra-group variance and distorts the sign structure of the resulting advantage estimates. The observable consequence is a tripled policy drift: v1 ’s KL divergence to the reference policy reaches ∼ 0.31 by step 900 (Fig. 2b), 3.4× that of v2 (∼ 0.09). (ii) Higher inference cost. v1 requires 3.8 mean turns per inference, 1.65× v3 ’s 2.3. (iii) Downstream efficiency does not materialise. v1 matches v3 ’s 27.0% resolve rate but achieves only −1.3% rounds and −0.6% tokens relative to baseline, versus −15% and −19% under v3 . v2 → v3 : fixing drift exposes length exploitation. (i) Win: v2 ’s final KL settles at ∼ 0.09 (Fig. 2b), roughly one-third of v1 ’s. (ii) New failure mode: mean completion length peaks near ∼ 2000 tokens around steps 400–500 (Fig. 4a), with a clipping ratio approaching 20%; advantage-layer scaling implicitly rewards long completions whose token totals happen to score well. (iii) No net retrieval7

quality gain: the file-level Fβ mean under v2 is 0.526 against 0.562 under v1 ; the reward-side stability improvement is expended on completion length rather than file localization.

db/migrations/operations/fields.py, which controls how AddField runs rather than why OneToOneField declares itself unique. Injection: 15 actions, 307K tokens, patch passes. The injected <retrieved_context> lists three candidate files, including related.py. The same agent deliberates for five turns, opens related.py at action 6, greps for OneToOneField at action 8 to locate the class definition, applies a minimal edit at action 12 conditioning unique=True on null=False, and calls finish at action 14. Result: one workspace file touched, 10× fewer tokens, patch passes the harness. CodeGrep does not supply the fix—the agent still greps to locate the exact definition. Instead, it compresses the exploratory prefix the frozen agent otherwise burns on repository-wide find/grep sweeps for issue-prose tokens that never appear in the codebase. This prefix is where the token savings originate.

v3 closes the loop. Removing Fβlr eliminates a signal that has no downstream consumer (the code editor takes no line ranges) and that is systematically zero on CATM-only entries. Under matched hyperparameters, v3 converges higher and faster on reward (Fig. 2a: 0.60–0.65 vs. 0.45–0.48), stabilises rollout turns at ∼ 2.1 (Fig. 2c, from ∼ 3.5), and delivers the headline downstream gains reported in §4.3. 5.3

Pooled vs. paired-instance efficiency

The efficiency numbers in Table 3 are pooled over each configuration’s own resolved set (baseline 129, v3 135), so they mix a genuine efficiency effect with a selection effect. Restricting to the 96 instances both configurations resolve, rounds fall 20.1 → 18.3 (−9%) and tokens 525K→ 448K (−15%)—preserving 60–80% of the pooled effect and confirming the improvement as per-issue acceleration.

6

7

Limitations and Future Work

CodeGrep’s contribution is concentrated on rollout cost, not resolve rate; a natural next step is end-toend co-training against a downstream resolve-rate signal, testing whether retrieval can be made to buy both efficiency and effectiveness.

Case Study: Where the Token Savings Come From

We complement the aggregate numbers with an action-level trace on a representative issue where injection converts a stalled 74-action rollout into a 15-action success.

8

Conclusion

We introduced CodeGrep, an open 14B RL-trained agent-style code retriever. Injected into a frozen OpenHands downstream agent, it lifts SWE-Bench Verified resolve rate by +1.2pp and cuts 15% of rounds and 19% of tokens on resolved instances. Our study surfaces a precision threshold governing when retrieval helps downstream, and—in our training setting—identifies the advantage layer, not the reward layer, as the appropriate site for efficiency signals in GRPO-style multi-turn training. All artifacts (model, training pipeline, RL environment, evaluation harnesses) will be released.

The issue: django-15278. Adding a nullable OneToOneField via migration crashes on SQLite with Cannot add a UNIQUE column: the OneToOneField constructor in django/db/ models/fields/related.py sets unique=True unconditionally, forcing a UNIQUE constraint that SQLite forbids on a nullable column. Baseline: 74 actions, 3.2M tokens, no patch. Without a retrieval hint, the frozen agent stalls after 74 actions: 35 terminal calls dominated by repo-wide find and grep, 17 chain-of-thought turns, and 18 file-editor operations touching 13 unique files, none of which contains the fix. Its first 18 tool calls (Table 8 in Appendix F) expose the failure mode: the agent searches for oauth2—a token from the issue’s third-party reproducer, absent from Django itself—inspects tests/migrations/test_state.py for a related test, and eventually attempts an edit in django/

References Anthropic. 2025. Claude Code: Anthropic’s official CLI for Claude. https://www.anthropic.com/ claude-code. Anthropic product page and documentation. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin,

8

Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. Findings of EMNLP.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems (NeurIPS).

Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified crossmodal pre-training for code representation. Annual Meeting of the Association for Computational Linguistics (ACL).

Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, and Luke Zettlemoyer. 2025. ReasonIR: Training retrievers for reasoning tasks. arXiv preprint arXiv:2504.20595.

Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training code representations with data flow. International Conference on Learning Representations (ICLR).

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR).

Xingyao Wang et al. 2024. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741.

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.

Zihan Wang, Kangrui Wang, Qineng He, Linjie Zhang, Ye Zheng, Xujie Zhou, Xinyu Deng, Bingxuan Liu, Chi Zhang, Heng Ji, and Manling Li. 2025. RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073.

Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516.

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying LLM-based software engineering agents. arXiv preprint arXiv:2407.01489. Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. 2025. SWE-Fixer: Training open-source LLMs for effective and efficient GitHub issue resolution. arXiv preprint arXiv:2501.05040.

Jina AI. 2024. Jina Code Embeddings 1.5B: A multilingual code embedding model. https://huggingface.co/jinaai/jina-codeembeddings-1.5b.

An Yang et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.

Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025. ToRL: Scaling tool-integrated RL. arXiv preprint arXiv:2503.23383.

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793.

Nebius AI. 2025. OpenHands trajectories with Qwen3-coder-480b on SWE-bench verified. https://nebius.com/blog/posts/openhandstrajectories-with-qwen3-coder-480b. Public release of ∼67K OpenHands agent trajectories and baseline numbers on SWE-Bench Verified.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR).

Ben Pan, Carlo Baronio, Albert Tam, Pietro Marsella, Mokshit Jain, Daniel Chiu, Swyx, and Silas Alberti. 2025. Introducing SWE-grep and SWE-grep-mini: RL for multi-turn, fast context retrieval. https:// cognition.ai/blog/swe-grep. Cognition AI blog post; commercial product overview.

Yuze Zhao et al. 2024. ms-swift: A framework for scalable training and inference of large language models. https://github.com/modelscope/ms-swift.

Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389.

Yuqi Zhou, Sunhao Dai, Changle Qu, Liang Pang, Jun Xu, and Ji-Rong Wen. 2026. Learning to retrieve from agent trajectories. arXiv preprint arXiv:2604.04949.

9

the context window. All searches use the native grep -rn –include=*.py ... rather than a pure-Python implementation, gaining roughly an order of magnitude in throughput.

Albert Örwall. 2024. Moatless tools: A framework for LLM-assisted code editing. https://github.com/ aorwall/moatless-tools.

A

Infrastructure Details

This appendix preserves the complete engineering realization of the RL environment (§3.3) and the CATM pipeline (§3.4). A.1

Rollout scheduler. SweGrepScheduler is a subclass of ms-swift’s MultiTurnScheduler. It maintains state across turns, masks tool responses out of the loss, injects the correct worktree path per rollout, and dispatches early termination. In total the class is roughly 400 lines of Python.

RL Environment Implementation

A rollout begins with the model emitting up to eight parallel grep/glob/read tool calls, which the sandbox executes concurrently inside the target worktree. Their responses are appended to the conversation with a loss mask of zero, so that only the tokens generated by the model itself contribute to the gradient. The model then sees the accumulated context and emits either the next round of tool calls or a final <answer> block. The rollout terminates when the model produces an answer, when no tool call is generated, when the completion budget is exhausted, or after the configured maximum of four turns (3 exploration +1 answer, matching §3.2).

A.2

CATM Pipeline Implementation

Stage 1. For each trajectory we traverse the message stream, identify role="assistant" messages containing tool_calls, and extract command="view" operations along with their target paths. We auto-detect the repository prefix (/workspace/owner__repo__commit/) and normalize to repo-relative paths. After a view we skip over any subsequent role="tool" messages and locate the next role="assistant" message, whose content is recorded as the file’s postreasoning. If a file is viewed multiple times, we keep the instance with the longest post-reasoning.

Layer 2 engineering. A single bare repository (e.g. _bare/django__django/) holds the shared .git object database, and each target commit is materialized as a lightweight worktree (django__django__abc12345/, ..._def67890/, etc.). The initial git clone --bare is a onetime network operation guarded by a 600-second timeout, and every subsequent git worktree add is a local file-system operation completing in milliseconds. To make the clone pipeline robust in the presence of private repositories or unusual GitHub configurations, we disable interactive prompts with GIT_TERMINAL_PROMPT=0 and GIT_ASKPASS=echo, fall back from fetch <commit> to fetch --all when a commit is not directly fetchable, and pre-validate every commit’s existence with git cat-file -e <commit> before invoking worktree add.

Stage 2. We use GLM-5.1-FP8 as the judge, prompted with the file path, the post-reasoning, and the task description. The judge returns either RELEVANT or NOT_RELEVANT; when the model output is ambiguous we default conservatively to RELEVANT. The judge runs with 16 parallel worker threads under a 60-second per-call timeout, and the pipeline is resumable by checkpointing processed instance IDs to disk with real-time flushing (append mode). End-to-end the judge stage completes in about five hours. Stage 3. The global constants β (the median postreasoning length across all mined candidates) and µraw (the mean of the unnormalized weights over the whole dataset) require two passes over the data: the first estimates them; the second assigns perfile weights via Eq. (2). Files with w < 0.15 are discarded.

Layer 3 engineering. The tool executor (swe_grep_tools.py) canonicalises every argument path and refuses any access that escapes the repository root. A global asyncio semaphore (Semaphore(64)) caps the number of concurrent subprocesses; per-tool timeouts (10s for grep and glob, 5s for read) shield the rollout from pathological queries. Tool outputs are truncated to 4096 characters, with 50 matches per grep and 200 lines per read, so that a single call cannot exhaust

A.3

Key Engineering Numbers

Table 5 lists the salient scale parameters and throughput figures for both the environment and the data pipeline. 10

Figure 3: Three-layer RL environment. Layer 1 extracts unique (repo, commit) pairs; Layer 2 uses git clone –bare and git worktree add to share the .git object database across commits; Layer 3 executes grep/glob/read through native subprocesses with safety guards. The right-hand sidebar compares against a Docker sandbox. Table 4: Comparison of CATM (ours) with LRAT (Zhou et al., 2026).

A.4

Dimension

LRAT

CATM (ours)

Target scenario

Web retrieval (Wikipedia QA)

Code retrieval (SWE-Bench)

Action pattern

Sequential, 1 browse per turn

Parallel, up to 8 reads per turn

Reasoning attribution

One reasoning trace per document (one-toone)

One reasoning trace shared across N parallel reads (one-to-many)

Sample granularity

One (query, doc) pair per browse; many pairs per trajectory

One (issue, filtered file set) per trajectory

Judge model

Qwen3-30B-A3B-Thinking-2507

GLM-5.1-FP8

Data scale

91K pairs from 26K trajectories

32K samples from 67K trajectories (47.7% retention)

Training method

Weighted InfoNCE (dense retriever)

GRPO (multi-turn tool-calling RL)

prompts (per-device batch 1, gradient accumulation 8, across 8 GPUs) and samples 8 GRPO rollouts per prompt, for 512 effective rollouts per step. Rollouts cap at 4096 generation tokens and 4 tooluse turns (3 exploration +1 answer); sampling uses temperature 1.0, top-p 0.9, top-k 50. We early-stop at step 897 of a planned 2000-step budget when reward improvement plateaus and KL-to-reference stays within safe bounds; end-to-end wall-clock is ∼ 27 hours per run.

Training Configuration

We train CodeGrep with GRPO on ms-swift on a single 8×B200 node. Parameter-efficient fine-tuning. LoRA (rank 32, α = 64) is applied to all linear projections of the 14B backbone; the base model weights stay frozen. Optimizer and model states are offloaded to CPU when idle, and training and rollout share GPUs via vLLM’s colocated mode (tensor-parallel 8, 35% of GPU memory allocated to the vLLM engine, prefix caching on).

B

Optimization. Peak learning rate 5×10−6 with cosine decay and 5% linear warmup; GRPO KL coefficient β = 0.02. Each step processes 64

Reward Design Details

Edge cases for Fβfile and Fβlr . Both scores are precision-biased Fβ with β = 0.5, with prec · rec/(β 2 prec + rec) · (1 + β 2 ) as the general for11

Table 5: Key engineering scale and throughput. Item

Value

Environment: unique (repo, commit) pairs Environment: disk saving (bare + worktree) Environment: tool concurrency cap Environment: max parallel tools per turn Environment: max tool execs per rollout Data: raw trajectories Data: judge model Data: judge thresholds Data: effective training samples Data: judge stage runtime Training: total wall-clock

∼thousands (after dedup) N ×repo_size → repo_size + N ×worktree 64 (asyncio semaphore) 8 32 (4 turns × 8) 67,074 GLM-5.1-FP8 (16-way parallel) reasoning_tokens ≥ 30, w ≥ 0.15 31,977 (47.7% retention) ∼5 hours 27 hours (8× B200)

mula, plus two degenerate cases: if both predicted and target sets are empty the score is 1; if exactly one is empty it is 0.

F1 scores over file retrieval and line retrieval tasks” with advantages scaled “by the average number of tool calls used per turn.” Their blog does not specify β, the line-range matching rule, the toolscale functional form, or the training data pipeline, so a component-level comparison is partial by necessity. The most consequential difference is our v3 ’s removal of the line-range term. Its outcome (higher final reward, Fig. 2a; more stable rollouts, Appendix D; 19% downstream token reduction, §4.3) suggests the line-range signal is not universally beneficial. We conjecture the driver is the downstream tool interface: agents whose editor consumes line ranges (plausibly Windsurf’s Cascade) may benefit; agents whose editor operates on full files (OpenHands, Cursor) will not, and the extra reward dimension becomes a training-time distractor.

Line-range matching algorithm. Each answer emits a list of line ranges Pr of the form (filename, start, end). The target set Gr comes from the SWE-Bench gold patch (patch files only). A predicted p matches a gold g iff: (a) p.filename = g.filename; and (b) |[p.s, p.e] ∩ [g.s, g.e]|/(g.e − g.s + 1) ≥ 0.5. Matching is greedy over Pr and each side consumes at most one match: iterating predictions in order, each pairs with the first still-unpaired gold range it overlaps; both are then removed from further consideration. TP is the number of matched pairs; prec = TP/|Pr |, rec = TP/|Gr |. Under merge mode, we drop from Pr any predicted range whose filename is a CATM-only file before scoring (such files have no gold range to match against, and would otherwise appear as unrecoverable false positives).

C

On merge mode and the “no CATM recall bonus” choice. In principle the pipeline supports an auxiliary CATM-recall term with a mixing weight α ∈ [0, 1] that would blend the base task score with a soft weighted recall over CATM files. Under merge mode this branch is inactive (α = 0), because every surviving CATM file is already inside G and contributes to Fβfile directly. Keeping α = 0 makes the three iterations cleanly comparable: they differ only in how task and efficiency signals are combined. Contrast with concurrent work. Cognition’s concurrent commercial system (Pan et al., 2025) describes its reward as “an average of weighted 12

Positioning: Agent RL vs Traditional RLHF

For readers coming from RLHF, Table 7 highlights the main structural differences between the setting we work in and standard single-turn preferencebased RL. This is a positioning aid rather than a methodological claim, and is included here for context.

D

Training Stability Diagnostics

Figure 4 reports three additional diagnostics tracked over the same ∼ 900-step training run as Figure 2: mean completion length, gradient norm, and the fraction of clipped tokens per batch. The three panels together characterize the trainingstability regime of all three reward configurations and complement the reward / KL / rollout picture

Table 6: Reward-design comparison with concurrent commercial work (Pan et al., 2025). Their blog describes a single final configuration; where a component’s mathematical form is not disclosed in the blog we mark it n/s (not specified). We list our final (v3 ) configuration. Reward component

Cognition SWE-grep

File-score β (precision bias)

Precision-biased (β not speci- β = 0.5 (precision-biased) fied) n/s (“F1 over line retrieval”) Jaccard with ≥ 50% overlap → Fβ=0.5 Avg. of file F1 and line F1 File-only (Fβ ); line-range removed Proprietary; construction not Open unsupervised CATM pipeline disclosed (§3.4) Not described (advantage-layer v1 : σ v1 (c̄) = 1/max(1, c̄/4); v2 /v3 : scaling only) mask only, 1c̄>0 p “Scale by avg tool calls per turn” s(c̄) = min(c̄/4, 1) via monkey(functional form n/s) patch on GRPO advantage None reported (single final con- v1 → v2 → v3 , three trained runs fig) (§3.5)

Line-range scoring rule Base score composition Training data / labels Efficiency signal at reward layer Efficiency signal at advantage layer Explicit reward-design study

This work (v3 )

Table 7: Comparison of traditional RLHF with the Agent RL setting used in this work. Dimension

Traditional RLHF

Agent RL (ours)

Rollout structure Action space Reward source Ground truth

Single-turn Single completion Human preference model Prompt–response pair

Environment Loss mask Data-construction cost

None Entire completion Low

Multi-turn (up to 8) Up to 8 parallel tool calls per turn Environment + rule-based Fβ Structured multi-signal (patch + trajectory + weights) Filesystem sandbox with real code execution Tool responses must be masked out High (judge inference + global β statistics)

in the main text.

Taken together, the three configurations trace a clean design trade-off: v1 (reward-layer scaling) contains the length problem but at the cost of policy drift (KL climbing to ∼ 0.31; Figure 2b); v2 (advantage-layer scaling) controls policy drift but opens the length channel; only v3 (no line-range) resolves both, producing training that is simultaneously the most reward-productive, the shortest, and among the cleanest on clipping. This picture supports our conclusion that v3 ’s reward is not only more effective on downstream metrics (§4.3) but also structurally the most stable during training.

The completion-length trajectory (a) is the most revealing panel and tells a two-story diagnostic. Under v2 , mean completion length inflates from roughly 400 tokens to a peak of ∼ 2,000 tokens around step 400–500 before slowly retreating, and the completion clipping ratio (c) peaks near 20% during the same window—clear evidence that v2 ’s advantage-layer scaling implicitly rewards relatively short-and-well-scoring completions within a group in a way that also permits long-completion excursions, opening a length-exploitation channel. v1 shows a milder version of the same inflation (peak ∼ 880 smoothed / ∼ 1500 raw around step 300–400) but recovers earlier, consistent with its reward-layer scaling directly penalising longcompletion / high-turn rollouts on the reward side. v3 removes the line-range component and stays near ∼ 300–400 throughout, with clipping ratio consistently below 5%. Gradient norms (b) remain bounded throughout for all three runs, with a single brief excursion near step 400 under v1 and v2 .

E

BM25 Baseline Configuration

This appendix documents the BM25 baseline used for both the retrieval-quality comparison in Table 2 and the downstream-injection comparison in Table 3, so that both are fully reproducible. Corpus. For each SWE-Bench-Verified instance we index the target repository’s application Python files. We skip __pycache__/, virtual-environment directories (venv/, .venv/, env/), and build arte13

Figure 4: Training-stability diagnostics over the same ∼ 900 GRPO steps as Figure 2, for all three reward iterations. (a) Mean completion length: v2 inflates dramatically (smoothed peak ∼ 1,150 tokens near step 450; raw peak ∼ 2000), v1 shows a milder mid-training bump, and v3 stays near 300–400 throughout. (b) Gradient norms remain bounded throughout for all three. (c) Completion-clipping ratio mirrors the length curve: v2 peaks at ∼ 20%, v1 at ∼ 14%, and v3 stays below 6% throughout.

facts (build/, dist/, .eggs/). Our default includes tests/ and test/ directories in the index; the ablation below shows that excluding them yields nearly identical results.

effect a naive path-aware BM25 could offer. Top-K selection. We report BM25 at K = 2 as the default because it approximately matches CodeGrep v3 ’s mean output length of 1.31 files. Larger K is not free: injecting more candidate files into the agent’s prompt inflates the per-rollout context. On retrieval quality, top-2 achieves Fβ mean 0.359 (Table 2). For completeness we also measured K = 5: it raises recall (each list is longer) at the cost of precision, and on downstream evaluation larger K carries the additional cost of longer agent-side prompts, producing qualitatively similar downstream failure to help. We report K = 2 as the main comparison because it is the fairest applesto-apples match to CodeGrep’s output cardinality.

Tokenization. Documents and queries are tokenized identically: split on whitespace, strip characters outside [a-zA-Z0-9_./], lower-case, drop tokens shorter than 2 characters, drop tokens beginning with http (URL fragments in issue text). This is a deliberately simple tokenizer intended to model a “classical BM25” baseline rather than a code-specialized one. BM25 parameters. We use BM25 Okapi with default parameters k1 = 1.5 and b = 0.75 (the rank_bm25 library defaults). We did not tune these on the evaluation set to avoid overfitting.

Test-directory scope. Our default indexes tests/ and test/ directories alongside application code (via the --include_tests flag), maximizing BM25’s fair shot at the ground truth (approximately 36% of evaluation instances have at least one test file in the ground truth). Ablating this filter (i.e., skipping test directories) yields a comparable Fβ within 0.01, indicating that testdirectory inclusion neither meaningfully helps nor hurts BM25 on this task. The default configuration is therefore not artificially disadvantaging BM25.

Path signal. We initially expected a file-path signal to help, since SWE-Bench issue reports frequently mention directory or file-name fragments (e.g. “django admin form”, “the ec2 metadata module”). We therefore built a second BM25 index over file-path tokens and added its scores to the content BM25 scores with a 2× weight. Empirically this augmentation had no measurable effect: the pathaugmented variant (B) and the content-only variant (A) produce identical mean Fβ on our evaluation set. The reason is that our tokenizer preserves the path separators . and / within tokens, so file-path fragments already appear as long tokens inside the content index. We report variant B as our default for clarity, but the finding is that a simple BM25 over tokenized code content already captures the

Retrieval. At query time, we take the tokenized issue text (the user turn from the eval prompt, which contains the bug report and error trace), compute the combined BM25 score for every indexed file, and return the top-K by score (K = 2 by default). Ties are broken by insertion order; if fewer 14

than K files have non-zero score, we still return the top-scoring 1–K as a fallback. Downstream injection. For the downstream evaluation (Table 3), we inject BM25 top-2 predictions into the same OpenHands prompt template used for CodeGrep. Predictions are formatted as a bulleted <candidate_files> block appended to the issue description, byte-identical to the CodeGrep injection format. The agent then executes its standard code-repair loop without any BM25specific prompting. Scoring. Predicted files are compared against the SWE-Bench gold patch file set using file-level Fβ=0.5 , identical to §4.2. We use forward-slash paths on both sides; any residual path-separator differences are normalized before set-intersection. Runtime. On CPU, indexing and querying the full evaluation set completes in under 10 minutes on a single machine. Building the index dominates cost; the retrieval step is under 10 ms per query. The BM25 downstream evaluation on all 500 SWEBench Verified instances completes with the same wall-clock budget as the CodeGrep downstream evaluations described in §4.1.

F

Case Study: Action-Level Trajectory

Table 8 shows the first 18 tool calls of the baseline vs. CodeGrep-injected trajectories on django-15278, illustrating the failure mode discussed in §6.

15

Table 8: Action-level trajectory on django-15278. Baseline spends its first 18 actions searching for oauth2 (a third-party name in the issue) and reading tangential files; CodeGrep injection sends the same agent directly to related.py. Baseline (unresolved, 74 actions) # Tool 1–2 3 4–9 10 11–14 15 16–17 18 19–20 21 ···

think task terminal think terminal file_editor terminal file_editor think file_editor ···

CodeGrep (resolved, 15 actions)

Target analyze issue plan find | grep oauth2×6 rewrite plan grep AddField tests/.../test_state.py grep -n for line hits test_state.py l. 627 propose migration-op fix operations/fields.py (never converges)

# Tool 1–5 6 7 8 9 10–11 12 13 14

16

think file_editor think terminal file_editor think file_editor think finish

Target analyze + read hints related.py locate OneToOneField grep OneToOneField related.py propose fix edit __init__ verify —

Related documents

• Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First#429408
• CodeSkill: Latent Skill Abstraction for Long-Horizon Code Agents#1117955
• SWE-Explore: Benchmarking How Coding Agents Explore Repositories#266239
• Learning Globally Reusable Skills for Coding Agents#432578
• MRCoder: An Efficient Context Selecting Approach for Repository-Level Code Generation#411106