Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills Chuan Xiao2,∗ , Zhengbo Jiao1,2,∗ , Shaobo Wang2 , Wei Wang1 , Bing Zhao1 , HU WEI1,† , Linfeng Zhang2,† , Lin Qu1 1
arXiv:2606.07412v1 [cs.SE] 5 Jun 2026
2 ∗
AI Data, Alibaba Group Shanghai Jiao Tong University
Equal contribution, † Corresponding authors
Abstract LLM-driven software engineering agents have become a central testbed for real-world languagemodel capability, yet their training remains limited by the availability of high-quality SWE tasks. Existing synthetic data methods typically create tasks through fixed mutation or bug-injection procedures, making the resulting distributions largely independent of the agent’s own weaknesses and training progress. We introduce Socratic-SWE, a closed-loop self-evolution framework that reuses the agent’s historical solving traces as a source of training signal. Rather than treating traces only as evidence for reward computation, Socratic-SWE distills them into structured agent skills that summarize recurring failures and effective repair patterns. These skills then guide the generation of targeted repair tasks in real repositories. Candidate tasks are checked through execution-based validation and scored with a solver-gradient alignment reward, so that the retained tasks are both verifiable and useful for improving the Solver. The updated Solver produces new traces, enabling the task curriculum to adapt over successive rounds. Across SWE-bench Verified, SWE-bench Lite, SWE-bench Pro, and Terminal-Bench 2.0, Socratic-SWE consistently improves over self-evolving baselines under the same compute budget, reaching 50.40% on SWEbench Verified after three iterations. These results suggest that solving traces can serve as a scalable substrate for self-evolving SWE agents. Date: June 8, 2026
1
Introduction
Software engineering is one of the most impactful application domains for large language models (LLMs), and the capability of coding agents has emerged as a critical measure of real-world intelligence. Unlike mathematical reasoning or short-form code generation, SWE tasks require agents to complete a full loop of bug localization, repair, and verification through long-horizon interaction with real code repositories, making SWE a natural setting for training capable language agents in realistic environments. Reinforcement learning (RL) has shown strong promise in this setting: SWE-RL [1] and SWE-Gym [2] demonstrate substantial gains by embedding agents in executable repository environments. However, RL training fundamentally depends on large quantities of high-quality tasks, and such data remains scarce in the SWE domain. Existing synthetic pipelines attempt to alleviate this bottleneck through AST-level mutation, LM-guided rewriting, or learned bug injection [3, 4], but they operate independently of the agent’s own training experience, resulting in largely 1
(a) Traditional SWE Data Synthesis Paradigm
Static Rule
Synthesis Engine
Scarce & Costly
Model-Agnostic
Static Task Supply Non-adaptive
reward only
(b)Socratic-SWE (Ours)
Agent Skills Grounded & Reusable
Evolving Task Supply
SWE-Agent Model-Aware
Dynamic & Evolving reward signals
Task-Trace-Skill
Figure 1 Comparison of SWE data synthesis paradigms. (a) Traditional: an open-loop, single-pass pipeline where static rules synthesize a fixed task supply. Evaluation traces are used only post hoc for rewards, not fed back into task construction, leaving the supply scarce, model-agnostic, and non-adaptive. (b) Socratic-SWE: a closed-loop, self-evolving process that distills grounded skills from solving traces to guide repository-grounded task construction, while feeding rewards into subsequent skill and task updates.
static task distributions that may be poorly aligned with the model’s actual capability gaps. At the same time, each round of RL training produces a valuable byproduct: solving traces. These traces record the agent’s behavior throughout repository interaction, including code search, file editing, command execution, and test runs. They reveal where the agent repeatedly fails, which repair strategies tend to induce regressions, and which repository patterns lead to ineffective exploration. Yet existing methods use traces primarily for reward extraction or credit assignment. GRPO [5] reduces a trajectory to a scalar reward, while process-level approaches such as GiGPO [6], iStar [7], and process reward models [8] assign finer-grained credit within trajectories. Regardless of granularity, traces are discarded once supervision has been computed. As the model improves, the fraction of tasks in a fixed distribution that still provides useful training signal becomes increasingly sparse, and learning eventually stagnates. We argue that these discarded traces contain the signals needed to evolve the curriculum. Because the traces are generated by the agent itself, they provide a direct view of the model’s current capability boundary. This suggests a self-evolving loop: the agent distills capabilities from historical solving traces and uses them to construct the next round of training tasks, without requiring external annotation. Related ideas have shown promise in adjacent domains. R-Zero [9] and Socratic-Zero [10] adapt task proposers to the solver’s frontier, Absolute Zero [11] achieves zero-data self-improvement through execution-based verification, and SkillRL [12] and SKILL0 [13] distill reusable skills from interaction traces. However, these settings involve simple traces, such as symbolic reasoning chains, short programs, or finite-step games, where pass/fail feedback is often sufficient to drive adaptation. SWE traces are richer: a single trajectory may span dozens of tool invocations across search, editing, execution, and testing, exposing diverse and diagnosable failure modes. Combined with deterministic execution-based verification from repository test suites, this makes SWE a particularly suitable domain for trace-based self-evolution. To address this challenge, we propose Socratic-SWE, a self-evolution framework for software engineering. Figure 1 summarizes the closed-loop design of Socratic-SWE. Socratic-SWE operates in 2
Figure 2 Main performance comparison on four benchmarks under a fixed 36k-instance training budget. SocraticSWE achieves 50.40% on SWE-bench Verified, 36.67% on SWE-bench Lite, 22.85% on SWE-bench Pro, and 14.61% on Terminal-Bench 2.0, outperforming all adapted baselines across both repository-level software repair and terminalagent tasks.
a three-stage loop. First, it distills recurring failure modes and successful repair behaviors from historical solving traces into an Agent Skill Registry, a structured representation of the model’s capability boundary. Second, a Generator uses these skills as constraints to construct targeted repair tasks in real code repositories. Candidate tasks are filtered through a staged execution-based validation pipeline to ensure reproducibility and non-triviality, and are further scored by a solver-gradient alignment reward that favors tasks whose induced Solver updates align with trusted validation gradients. Third, the Solver trains on the accepted tasks and produces new traces, which support the next round of skill distillation. In this way, traces are transformed into skills, skills guide task generation, and generated tasks produce the traces needed for continued self-evolution. Empirically, we validate Socratic-SWE on SWE-bench Verified, SWE-bench Lite, SWE-bench Pro, and Terminal-Bench 2.0. As summarized in Figure 2, Socratic-SWE consistently outperforms self-evolving baselines under the same compute budget, reaching 50.40% on SWE-bench Verified after 3 iterations (+7.80 over the base agent and +3.40 over SSR). These results show that trace-derived curricula provide stronger training signals, support sustained self-evolution without external annotation, and transfer beyond repository repair to terminal-native tasks in realistic SWE environments. Our contributions are as follows: 1. Trace-Driven Self-Evolution Paradigm. We show that solving traces, typically discarded after reward computation, can serve as a reusable substrate for self-evolving SWE agents. Socratic-SWE converts historical Solver behavior into structured agent skills, which guide subsequent repository-grounded task construction in a closed trace-skill-task loop. 2. Gradient-Aware Curriculum Optimization. We introduce a curriculum mechanism that combines skill-conditioned task generation, execution-based task validation, and a solver-gradient alignment reward. Generated tasks are first verified in real repository environments to ensure reproducibility and solvability, and are then prioritized by whether their induced Solver updates align with trusted validation gradients. 3. Superior Empirical Performance. We evaluate Socratic-SWE on SWE benchmarks, including SWEbench and Terminal-Bench. Under the same compute budget, it consistently outperforms self-evolving baselines, demonstrating strong self-evolution and transfer capabilities.
3
2
Related Work
Agentic Reinforcement Learning. Reinforcement learning is a central post-training paradigm for LLM reasoning and decision-making. OpenAI o1 popularized large-scale reasoning RL [14], and DeepSeekMath’s GRPO provided an open-source framework for verifiable rewards [5]. DAPO [15], GSPO [16], SAPO [17], and GDPO [18] improve RL efficiency and stability, but still assume immediately verifiable rewards. Agentic tasks violate this assumption: feedback is sparse and spread over long trajectories. GiGPO [6], ARPO [19], and SkyRL-Agent [20] extend RL to multi-turn tool use and long-horizon optimization. OPUS [21], GradAlign [22], and OptimSyn [23] use optimizer feedback, gradient alignment, or synthetic data construction as dynamic signals. Integrating verifiable feedback, failure analysis, and long-horizon agent training for SWE remains underexplored. SWE Coding Agents. SWE agents must navigate executable repositories, edit code, run commands, and preserve behavior under regression tests. SWE-bench introduced repository-level issue resolution from GitHub issues [24]. SWE-agent [25], OpenHands [26], and Agentless [27] show that LLMs can solve tasks with tools, code execution, and verification feedback. SWE-Gym [2], SWE-RL [1], and SWE-Master [28] build training settings from executable environments, software evolution data, and post-training pipelines. BugLab [4], SWE-smith [3], and SSR [29] expand data through bug injection, synthetic task generation, or self-play. Yet data construction emphasizes scale and executability, while using repository understanding, executable feedback, and model failures only weakly for targeted task generation. Self-Evolving LLMs. Self-play reduces human labels by assigning proposer, solver, evaluator, or teacher roles. TTRL [30], R-Zero [9], and Socratic-Zero [10] form self-improvement signals from unlabeled data, Challenger-Solver co-evolution, or Teacher-Solver-Generator loops. Absolute Zero [11], SPIRAL [31], SocraticGeo [32], and SpatialEvo [33] reduce noisy feedback through verifiable environments, including code execution, zero-sum games, programmatic geometry, and deterministic spatial tasks. SkillRL [12] and SKILL0 [13] distill interaction experience into reusable skills. Agentic Proposing [34] uses modular reasoning skills to synthesize harder verifiable examples. Overall, self-evolution is shifting toward environment-grounded feedback and skill-driven synthesis; for SWE, the key challenge is transforming long-horizon repair failures into future task distributions that keep pushing model capabilities.
3 3.1
Methodology Socratic-SWE Framework
We introduce Socratic-SWE, a co-evolutionary self-play framework for software engineering agents. A shared policy πθ alternates between two roles: a Generator that constructs repository-grounded repair tasks and a Solver that produces patches for them. The framework learns from two external signals: an Agent Skill Registry S distilled from historical interaction traces, and a staged execution-grounded validation pipeline that filters generated tasks before they enter training. Figure 3 illustrates the overall loop. Let R denote the repository corpus and S the Agent Skill Registry. At iteration t, the system maintains a curriculum of tasks Dt = {(r, τ, v)}, where r ∈ R is a sandboxed repository, τ is a repair task grounded in r, and v is an executable verification signal. The Solver trains on Dt and produces trajectories that expose capability gaps. The Generator then uses skills s ∈ S to propose tasks targeting these gaps, and the validation pipeline filters proposals so that only executable, reproducible, and solvable tasks enter the curriculum: (τ ′ , v ′ ) ∼ πθ (· | r, s, Et , role=G), Dt+1 = Dt ∪ {(r, τ ′ , v ′ ) | Valid (τ ′ , v ′ , r) = 1},
(1) (2)
where Et denotes Solver-side evidence collected on Dt , and Valid (·) is a staged validation function defined in §3.3. Algorithm 1 details the procedure.
3.2
Agent Skill Registry
A central challenge in skill-guided task generation is obtaining skills that are both structured for retrieval and grounded in real agent behavior. We distill an Agent Skill Registry from historical interaction traces 4
Role-based Self-play Loop Trace Success trace
Capability Gaps
Failure trace
pass
Solver
fail
Skill: Verifier Design
Skill: Gap-to-Task
Partial trace regression
Gap-to-task
name
Verifier Design
name
when
Weakness in 𝑬𝝉
when
Needs check
when
Task Mutation Frontier target
ops
Oracle to 𝒗
ops
𝝉 to harder 𝝉
ops
Gap to 𝝉
Generator
Repository Sandbox/Code World
Accepted Task
Inputs
File Tree
edit
task
patch
test
Verifier Gate
Generated
Solver in Repository Sandbox inspect
Skill: Task Mutation
name
Format
skills Construct task
Grounding
Verifier
Execution
Execution Feedback Fixed failing tests Preserved passing rates
Regression avoidance
Terminal
…
Patch
run tests
Execution feedback
Results
$ pytest -q ..F. FAIL: test_get_item
Outputs
Evidence
Curriculum Task Pool
1 failed
Design verifier
repos
Semantics
Generator Reward
Task Evolution(capability-gap targeting)
Repository Context(shared)
evidences
5 passed
Step 1
Step 2
Step 3
local fix
edge case
no regression
Alignment Reward
Held-out Validation Step 4
⚫ Trusted tasks ⚫ Diverse tasks ⚫ Human verified
multi-file
Target gradient 𝐺𝑣
(usefulness signal)
Accepted Task 𝝉′ , 𝒗′ , 𝒓 Candidate signal 𝒈𝝉
𝑅𝐺 = 𝑉𝑎𝑙 𝑖𝑑 ሶ 𝜏, 𝑣, 𝑟 ⋅ cos 𝑔𝜏 , 𝐺𝑣
usefulness signal validation direction 𝐺𝑣
Figure 3 An overview of the Socratic-SWE framework, illustrating the co-evolution of the Solver and the Generator. On the Solver side, the model consumes tasks from the task pool, solves them in the repository sandbox, and produces solving traces. These traces are distilled into the Agent Skill Registry, where recurring model gaps and repository-understanding patterns are captured as reusable skills. On the Generator side, the model uses the Skill Registry to synthesize targeted tasks in the repository sandbox. Candidate tasks are then filtered by the Verifier Gate along format, grounding, execution, and semantics to ensure that they are well-formed, reproducible, solvable, and meaningful. Accepted tasks receive an alignment reward from the held-out validation gradient, enter the task pool as the next generation of tasks, and are consumed again by the Solver, forming a closed task-trace-skill-task loop.
through a three-stage pipeline. We define a skill as a structured document with four fields: a name, a natural-language description, a set of applicability conditions, and an ordered list of operations. This representation lets the Generator retrieve and condition on skills programmatically rather than through free-form text. Stage 1: Trace Collection. We deploy πθ at its current checkpoint on the seed task set and collect a trace corpus T = {τ1 , . . . , τN }. Each trace records repository inspection, code edits, command execution, and verification outcomes. We split T into successful traces T + = {τ : r(τ ) = 1} and failed traces T − = {τ : r(τ ) = 0}, and retain both since failure traces expose capability gaps. Stage 2: Skill Extraction. A distillation model Mdistill processes the trace corpus and extracts recurring behavioral patterns as candidate skills. For successful traces, it identifies generalizable strategies; for failed traces, it summarizes failure lessons and corrective principles. Formally: Ŝ = Mdistill T + , T − , (3) where Ŝ is the set of candidate skills. Stage 3: Registry Construction. Candidate skills are deduplicated by semantic similarity and filtered by trace coverage to form the Skill Registry: S = Dedup Ŝ, δsim = {s1 , . . . , sM }, (4) where δsim is the similarity threshold and M is the number of retained skills. During training, the Generator samples s ∼ S and uses it to bias task proposal toward the behavioral pattern.
5
3.3
Skill-Guided Task Generator
When acting as Generator, πθ constructs executable SWE repair tasks that expose Solver weaknesses. Given a repository r ∈ R and a skill s ∈ S, the Generator produces a candidate task τ and its verification signal v: (τ, v) ∼ πθ (· | r, s, Et , role=G).
(5)
Here, τ specifies a repository-grounded repair objective and v specifies the executable tests or commands used to evaluate patches. Conditioning on Solver evidence Et makes generation adaptive as the Solver improves. Task Validation. Not all generated tasks are suitable for training. Before entering the curriculum, each candidate (τ, v) passes four checks in the repository sandbox r: format, grounding, execution, and semantics. A candidate is accepted iff all stages pass: 1. Format (f1 ): τ and v are well-formed, parseable, and syntactically valid. 2. Grounding (f2 ): τ references artifacts that actually exist in r. 3. Execution (f3 ): v runs without infrastructure errors and is stable across repeated runs. 4. Semantics (f4 ): v separates failing from repaired states, and at least one valid repair exists. We write: Valid (τ, v, r) =
4 Y
fl (τ, v, r) ∈ {0, 1},
(6)
l=1
where each fl is evaluated only if all preceding stages pass. Only accepted candidates enter Dt+1 . Generator Reward. Validation ensures a task is executable and solvable, but not useful. We reward the Generator by whether the Solver update by a task aligns with a validation direction. |V
|
val We maintain a held-out set of trusted validation tasks Vval = {(τjv , vjv , rjv )}j=1 . For each validation task, we roll out K Solver trajectories, compute executable feedback rewards, and estimate the per-task policy gradient: PK 1 v v v (7) gjv = K k=1 Âj,k ∇θ log πθ (ŷj,k | τj , rj , vj ). P v 1 Averaging over the validation set yields the target gradient direction Gv = |Vval | j gj .
For each candidate task (τ, v) in repository r, we estimate the Solver policy gradient gτ from K rollouts. The Generator reward is: RG (τ, v, r) = Valid (τ, v, r) · cos(gτ , Gv ). (8) The validation factor zeros out invalid tasks, and the cosine term favors tasks whose induced updates align with the validation gradient. We recompute Gv periodically as the Solver evolves.
3.4
Repository Repair Solver
When acting as Solver, πθ solves accepted tasks by interacting with the repository environment. The Solver sees the task specification and repository feedback, but not the reference solution or verifier internals used by the Generator. Patch Generation. Given a task (τ, v) and repository r, the Solver samples a trajectory: ŷ ∼ πθ (· | τ, r, v, role=S).
(9)
The trajectory may include inspection, code localization, file edits, and validation attempts. Executable Feedback Reward. For each trajectory ŷ, we apply the generated patch and run the verification suite. Let F denote the set of originally failing tests and P the set of originally passing tests; let F✓ ⊆ F and P✓ ⊆ P be the subsets that pass after patching. The Solver reward is: rS = λ1 1[F✓ =F ∧ P✓ =P] + λ2
|F✓ | |P✓ | + λ3 , |F| |P|
where the three terms reward full-suite pass, partial repair rate, and regression avoidance. 6
(10)
3.5
Training with Role-Specific Objectives
We optimize both roles jointly with shared weights: J(θ) = Er,s E(τ,v)∼πG[RG (τ, v, r)] + Eτ,r,v Eŷ∼πS[rS (ŷ, τ, v, r)] ,
(11)
where πG (· | r, s) := πθ (· | r, s, role=G) and πS (· | τ, r, v) := πθ (· | τ, r, v, role=S). We optimize a clipped surrogate objective: h P i K 1 L(θ) = E K i=1 L̂i − β DKL [πθ ∥ πref ] ,
(12)
is the importance ratio. where L̂i = min ρi Âi , clip(ρi , 1−ϵ, 1+ϵ) Âi and ρi = πθπθ (o(oi |x) i |x) old
i For the Generator, which receives a single scalar reward RG (Eq. (8)), we apply GRPO with Ri := RG . The Solver reward rS (Eq. (10)) combines three heterogeneous components (pass, repair, regr) on different scales. We adopt GDPO [18], which normalizes each component within its own group before aggregation: (m,i)
(m)
Âi
=
rS
(m,j) K }j=1 )
− mean({rS (m,j)
std({rS }K j=1 ) + δ P (m) 3 Â , ÂiS = BatchNorm m=1 i
,
(13) (14)
where m ∈ {1, 2, 3} indexes reward components and BatchNorm rescales advantages across the batch. This normalization lets the Solver distinguish partial repair from full fix and yields informative gradients from a single summed reward. Both roles share the clipped objective in Eq. (12), with Âi := ÂiS for the Solver. The total loss is L(θ) = LG (θ) + LS (θ).
4 4.1
Experiments Experimental Setup
Models. Qwen3.5-9B [35] was used for both the Generator and the Solver. The Generator was optimized with GRPO [5] from gradient feedback. Qwen3.6-27B [36] was used to distill skills. The Solver was trained with GDPO [18] on data generated by Socratic-SWE, with weights shared with the Generator. Datasets and Benchmarks. All self-evolving methods ran for 3 iterations, generating 12k validated training instances per iteration (36k total). For baselines that require seed tasks, 10% of SWE-smith [3] was used as the seed dataset; Socratic-SWE requires seed repositories. The trusted validation set Vval was a fixed held-out subset of BeyondSWE [37], used for generator-gradient alignment. All methods were evaluated on four benchmarks: SWE-bench Verified [24] (500 human-verified repository-level issues), SWE-bench Lite [24] (300 filtered issues), SWE-bench Pro Public [38] (731 complex enterprise-level software problems), and Terminal-Bench 2.0 (TB2) [39] (terminal-native task completion in sandboxed environments). Agent Harness. mini-swe-agent [40] was used as the execution harness for SWE benchmarks, with only Bash exposed to reduce confounds from tool design. For Terminal-Bench 2.0, we used little-coder [41] as the terminal-agent evaluation harness. Evaluation. For SWE-bench Verified, Lite, and Pro, each agent interacted with the repository and submitted a patch validated by benchmark tests. For TB2, each agent was given an instruction and a sandboxed terminal, then required to reach a final state that passed the verifier. TB2 was evaluated using little-coder [41]. Pass rate was reported as the primary metric across all benchmarks. Baseline Methods. We compare Socratic-SWE with the base agent and five self-evolving baselines: SPIRAL [31], R-Zero [9], Absolute-Zero [11], Socratic-Zero [10], and SSR [29]. All methods use the same Solver architecture (Qwen3.5-9B), agent harness (mini-swe-agent), Terminal-Bench harness (little-coder), benchmarks, and interaction budget. Details of the SWE adaptations and baseline initialization are provided in Appendix B. 7
Table 1 Main results on software engineering benchmarks. All self-evolving methods use identical Solver (Qwen3.59B), harness (mini-swe-agent), and training budget (12k instances × 3 iterations). TB2 is evaluated with little-coder. Overall is the mean score across four benchmarks. Green and red arrows indicate improvements and degradations relative to the Base Agent, respectively. Benchmarks Method Qwen3.5-9B + Base Agent
Overall
SWE-bench SWE-bench SWE-bench Terminal-Bench Verified Lite Pro 2.0
∆ vs. Base
24.91
42.60
29.67
17.24
10.11
—
25.29 24.25 24.02
43.20 ↑ 0.60 42.00 ↓ 0.60 41.80 ↓ 0.80
30.33 ↑ 0.66 29.33 ↓ 0.34 29.00 ↓ 0.67
17.51 ↑ 0.27 16.69 ↓ 0.55 16.28 ↓ 0.96
10.11 — 8.99 ↓ 1.12 8.99 ↓ 1.12
↑ 0.38
25.67 26.29 25.72
43.80 ↑ 1.20 44.40 ↑ 1.80 44.00 ↑ 1.40
31.00 ↑ 1.33 31.33 ↑ 1.66 31.00 ↑ 1.33
17.78 ↑ 0.54 18.19 ↑ 0.95 17.78 ↑ 0.54
10.11 — 11.24 ↑ 1.13 10.11 —
↑ 0.76
26.21 26.75 26.01
44.20 ↑ 1.60 45.00 ↑ 2.40 44.40 ↑ 1.80
31.33 ↑ 1.66 32.00 ↑ 2.33 31.33 ↑ 1.66
18.06 ↑ 0.82 18.74 ↑ 1.50 18.19 ↑ 0.95
11.24 ↑ 1.13 11.24 ↑ 1.13 10.11 —
↑ 1.30
26.83 27.87 27.57
45.20 ↑ 2.60 46.40 ↑ 3.80 45.80 ↑ 3.20
32.00 ↑ 2.33 33.00 ↑ 3.33 32.67 ↑ 3.00
18.88 ↑ 1.64 19.70 ↑ 2.46 19.43 ↑ 2.19
11.24 ↑ 1.13 12.36 ↑ 2.25 12.36 ↑ 2.25
↑ 1.92
26.31 27.63 28.51
44.20 ↑ 1.60 45.80 ↑ 3.20 47.00 ↑ 4.40
31.33 ↑ 1.66 32.67 ↑ 3.00 34.00 ↑ 4.33
18.47 ↑ 1.23 19.70 ↑ 2.46 20.66 ↑ 3.42
11.24 ↑ 1.13 12.36 ↑ 2.25 12.36 ↑ 2.25
↑ 1.40
27.82 29.64 31.13
46.20 ↑ 3.60 48.40 ↑ 5.80 50.40 ↑ 7.80
33.00 ↑ 3.33 35.33 ↑ 5.66 36.67 ↑ 7.00
19.70 ↑ 2.46 21.34 ↑ 4.10 22.85 ↑ 5.61
12.36 ↑ 2.25 13.48 ↑ 3.37 14.61 ↑ 4.50
↑ 2.91
R-Zero + Iteration 1 + Iteration 2 + Iteration 3
↓ 0.66 ↓ 0.89
SPIRAL + Iteration 1 + Iteration 2 + Iteration 3
↑ 1.38 ↑ 0.81
Absolute-Zero + Iteration 1 + Iteration 2 + Iteration 3
↑ 1.84 ↑ 1.10
Socratic-Zero + Iteration 1 + Iteration 2 + Iteration 3
↑ 2.96 ↑ 2.66
SSR + Iteration 1 + Iteration 2 + Iteration 3
↑ 2.72 ↑ 3.60
Socratic-SWE (Ours) + Iteration 1 + Iteration 2 + Iteration 3
4.2
↑ 4.73 ↑ 6.22
Main Results
Overall Performance. As shown in Table 1, Socratic-SWE achieves the strongest results across all four benchmarks after 3 iterations. With a fixed 36k-instance budget and zero pre-existing SWE training tasks, it improves over the base agent by +6.22 points overall, reaching 50.40% on SWE-bench Verified (+7.80), 36.67% on Lite (+7.00), 22.85% on Pro (+5.61), and 14.61% on Terminal-Bench 2.0 (+4.50). Its Verified gain also grows steadily across iterations (+3.60, +5.80, +7.80), while several baselines saturate or regress. Self-Play Methods Struggle on SWE Tasks. General self-play methods are brittle when adapted to SWE. R-Zero, which uses majority vote as reward, briefly improves at Iteration 1 (+0.60 on Verified) but falls below the base agent (−0.80), suggesting that vote-based rewards are too noisy for partial repairs. SPIRAL and Absolute-Zero peak at Iteration 2 (+1.80 and +2.40 on Verified) and regress at Iteration 3, indicating that self-play without execution-grounded validation can lead to mode collapse or reward hacking. SSR Improves Steadily but Remains Bounded. SSR is the strongest baseline, reaching +4.40 on SWE-bench Verified after 3 iterations. Its bug-injection mechanism and execution checks keep generated tasks valid, but its gains lack the acceleration of Socratic-SWE. This suggests that, without skill-guided targeting of capability gaps, SSR exhausts low-hanging bug patterns. Teacher-Guided Co-evolution Saturates Early. Socratic-Zero, despite using a 397B-parameter Teacher, peaks at Iteration 2 (+3.80 on Verified) and slightly declines at Iteration 3. This suggests that teacher-guided task generation remains bounded by the Teacher’s domain understanding and may lose information during
8
SWE-bench Verified (%)
Socratic-SWE
52
52.00
50 48
48.00
SSR
46 44
Socratic-SWE SSR
Base agent
42 0
1
2
3
Training Iterations
4
5
Figure 4 SWE-bench Verified scaling beyond the first three iterations. Socratic-SWE improves faster and reaches a higher ceiling, while SSR plateaus earlier.
Generator distillation. Terminal-Native Generalization. On Terminal-Bench 2.0, only SSR and Socratic-SWE show meaningful gains (+2.25 and +4.50). Socratic-SWE’s stronger transfer suggests that trace-derived skills capture general agent behaviors across tasks such as file manipulation, command chaining, and environment navigation, rather than only issue-specific repair patterns.
5
Analysis and Ablation Studies
5.1
Iteration Scaling and Saturation
To study long-term scaling, we extend training to 5 iterations for Socratic-SWE and SSR. Figure 4 shows three regimes: Socratic-SWE improves rapidly in Iterations 1–3 (+3.60, +2.20, +2.00) and still gains in Iteration 4 (+1.20, reaching 51.60%), while SSR improves more slowly (+1.60, +1.60, +1.20, then +0.80). By Iteration 5, both methods nearly plateau, at 52.00% for Socratic-SWE and 48.00% for SSR. This saturation reflects the closed-world setup. The Agent Skill Registry increasingly covers the seedrepository space, leaving fewer gaps to target, while the fixed repository pool yields redundant training signal. Even so, Socratic-SWE saturates 2 iterations later and at a higher ceiling than SSR, showing that skill-guided curriculum generation extracts more signal from the same data. Expanding the repository pool or enabling cross-repository transfer may extend this trend.
5.2
Ablation of Framework Components
We ablate three components on SWE-bench Verified at Iteration 3 (Table 2): removing the Skill Registry, replacing trace distillation with manual skills, and replacing GDPO with GRPO. Removing the Skill Registry causes the largest drop, showing that curriculum design is the main driver of Socratic-SWE. Replacing trace distillation with manual skills hurts performance because hand-written taxonomies miss behaviors such as iterative grep-then-edit patterns. Replacing GDPO with GRPO also degrades results, suggesting that decomposed reward learning better handles partially correct patches than binary pass/fail signals.
9
Table 2 Ablation results on SWE-bench Verified at Iteration 3. The upper block removes key Socratic-SWE components, and the lower block varies the skill extractor. The results show that each component helps and that performance is robust across extractor scales. Variant
5.3
Verified
Socratic-SWE w/o Skill Registry w/o Trace Distillation w/o GDPO, using GRPO
50.40 46.20 48.00 48.60
Skill extraction model Qwen3.5-9B self-extraction Qwen3.6-27B default Claude Opus 4.5
49.80 50.40 51.00
∆ vs. Full –
↓ 4.20
↓ 4.20
↓ 2.40
↓ 2.40
↓ 1.80
↓ 1.80
↓ 0.60
↓ 0.60
—
—
↑ 0.60
↑ 0.60
Robustness of Skill Extraction
Socratic-SWE is not sensitive to the skill extractor. Even the smallest extractor, Qwen3.5-9B, reaches 49.80% on Verified, only 0.60 points below the full system. Qwen3.6-27B gives a small gain, and Claude Opus 4.5 adds another +0.60. This suggests that the core advantage comes from the framework itself—skill-gap-targeted generation plus execution-grounded validation—rather than from a particularly strong extractor. As long as skills come from real traces and are validated by tests, coarse descriptions are sufficient to guide effective curriculum generation.
6
Conclusion
We presented Socratic-SWE, a practical closed-loop framework for self-evolving software engineering agents under limited access to high-quality SWE task data. By reusing historical solving traces as training signal and distilling them into an Agent Skill Registry, Socratic-SWE generates targeted repair tasks that address capability gaps and track the model’s frontier. Across four benchmarks, it gains +7.80 points on SWE-bench Verified and +4.50 points on Terminal-Bench 2.0 after three iterations, outperforming five self-play baselines under identical compute budgets. Analyses show that skill-guided curricula delay saturation and better exploit finite repository pools, while ablations confirm the roles of the skill registry, trace distillation, and GDPO. Future work may explore dynamic repository augmentation, cross-domain skill transfer, and online skill discovery.
Limitations Socratic-SWE is evaluated in a closed-world setting with a fixed pool of seed repositories. As the Agent Skill Registry becomes more complete, later iterations have fewer novel capability gaps to target, which makes task generation increasingly redundant. Our scaling analysis therefore reflects curriculum behavior under a fixed repository distribution rather than fully open-ended improvement with continuously refreshed data. The method also relies on a held-out validation set to define the generator-gradient alignment reward. While this design improves curriculum quality, it introduces dependence on the choice of trusted validation tasks and may limit robustness if the validation distribution is not representative of the target deployment setting. In addition, Socratic-SWE assumes executable verification and sandboxed repository interaction. Its gains may therefore not transfer directly to settings without reliable tests, deterministic execution, or clear tasklevel validation. Finally, our evaluation covers four benchmarks in SWE and terminal-agent settings; broader transfer to other programming languages, repository styles, and software engineering workflows remains to be established.
10
References [1] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. SWE-RL: Advancing LLM reasoning via reinforcement learning on open software evolution. In Advances in Neural Information Processing Systems 38, 2025. [2] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-Gym. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 47717–47737. PMLR, 2025. [3] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents. In Advances in Neural Information Processing Systems 38, 2025. [4] Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. Self-supervised bug detection and repair. In Advances in Neural Information Processing Systems 34, 2021. [5] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. [6] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. Advances in Neural Information Processing Systems, 38:46375–46408, 2026. [7] Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, and Jianbin Jiao. Agentic reinforcement learning with implicit step rewards. arXiv preprint arXiv:2509.19199, 2025. [8] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, volume 2024, pages 39578–39601, 2024. [9] Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-Zero: Self-evolving reasoning LLM from zero data. In International Conference on Learning Representations, 2026. [10] Shaobo Wang, Zhengbo Jiao, Zifan Zhang, Yilang Peng, Xu Ze, Boyu Yang, Wei Wang, Hu Wei, and Linfeng Zhang. Socratic-Zero: Bootstrapping reasoning via data-free agent co-evolution, 2025. [11] Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. In Advances in Neural Information Processing Systems 38, 2025. [12] Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. SkillRL: Evolving agents via recursive skillaugmented reinforcement learning. In ICLR 2026 Workshop on Lifelong Learning Agents, 2026. [13] Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. SKILL0: In-context agentic reinforcement learning for skill internalization, 2026. [14] OpenAI. OpenAI o1 system card, 2024. [15] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026. [16] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. [17] Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin. Soft adaptive policy optimization. arXiv preprint arXiv:2511.20347, 2025. [18] Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, YuChiang Frank Wang, Kwang-Ting Cheng, et al. Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization. arXiv preprint arXiv:2601.05242, 2026.
11
[19] Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Agentic reinforced policy optimization. In International Conference on Learning Representations, 2026. [20] Shiyi Cao, Dacheng Li, Fangzhou Zhao, Shuo Yuan, Sumanth R. Hegde, Connor Chen, Charlie Ruan, Tyler Griggs, Shu Liu, Eric Tang, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. SkyRL-Agent: Efficient RL training for multi-turn LLM agent, 2025. [21] Shaobo Wang, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, Jialin Liu, Guo Chen, Tianyu Zhang, Junhao Zheng, Kexin Yang, Xingzhang Ren, Dayiheng Liu, and Linfeng Zhang. Opus: Towards efficient and principled data selection in large language model pre-training in every iteration, 2026. [22] Ningyuan Yang, Weihua Du, Weiwei Sun, Sean Welleck, and Yiming Yang. GradAlign: Gradient-aligned data selection for LLM reinforcement learning, 2026. [23] Zhiting Fan, Ruizhe Chen, Tianxiang Hu, Ru Peng, Zenan Huang, Haokai Xu, Yixin Chen, Jian Wu, Junbo Zhao, and Zuozhu Liu. OptimSyn: Influence-guided rubrics optimization for synthetic data generation. In International Conference on Learning Representations, 2026. [24] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, 2024. [25] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37, 2024. [26] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations, 2025. [27] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying LLM-based software engineering agents. Proceedings of the ACM on Software Engineering, 2(FSE):801–824, 2025. [28] Huatong Song, Lisheng Huang, Shuang Sun, Jinhao Jiang, Ran Le, Daixuan Cheng, Guoxin Chen, Yiwen Hu, Zongchao Chen, Yiming Jia, Wayne Xin Zhao, Yang Song, Tao Zhang, and Ji-Rong Wen. SWE-Master: Unleashing the potential of software engineering agents via post-training, 2026. [29] Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, Gabriel Synnaeve, Daniel Fried, Lingming Zhang, and Sida I. Wang. Toward training superintelligent software agents through self-play SWE-RL, 2025. [30] Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, Biqing Qi, Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, and Bowen Zhou. TTRL: Test-time reinforcement learning. In Advances in Neural Information Processing Systems 38, 2025. [31] Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv preprint arXiv:2506.24119, 2025. [32] Zhengbo Jiao, Shaobo Wang, Zifan Zhang, Wei Wang, Bing Zhao, Hu Wei, and Linfeng Zhang. Socratic-Geo: Synthetic data generation and geometric reasoning via multi-agent interaction, 2026. [33] Dinging Li, Yingxiu Zhao, Xinrui Cheng, Kangheng Lin, Hongbo Peng, Hongxing Li, Zixuan Wang, Yuhong Dai, Haodong Li, Jia Wang, Yukang Shi, Liang Zhao, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. SpatialEvo: Self-evolving spatial intelligence via deterministic geometric environments, 2026. [34] Zhengbo Jiao, Shaobo Wang, Zifan Zhang, Xuan Ren, Wei Wang, Bing Zhao, Hu Wei, and Linfeng Zhang. Agentic proposing: Enhancing large language model reasoning via compositional skill synthesis, 2026. [35] Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. [36] Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026.
12
[37] Guoxin Chen, Fanzhe Meng, Jiale Zhao, Minghao Li, Daixuan Cheng, Huatong Song, Jie Chen, Yuzhi Lin, Hui Chen, Xin Zhao, Ruihua Song, Chang Liu, Cheng Chen, Kai Jia, and Ji-Rong Wen. Beyondswe: Can current code agent survive beyond single-repo bug fixing?, 2026. [38] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks?, 2025. [39] Mike A. Merrill et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. [40] SWE-agent Team. mini-SWE-agent: SWE-agent/mini-swe-agent, 2025.
The minimal AI software engineering agent.
https://github.com/
[41] Itay Inbar. little-coder: A coding agent optimized for small local language models. https://open.substack.com/ pub/itayinbarr/p/honey-i-shrunk-the-coding-agent, April 2026. White paper. [42] Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. Spice: Self-play in corpus environments improves reasoning, 2025.
13
Algorithm 1 Socratic-SWE: Self-Play in Repository Environments Require: Shared policy πθ ; repository corpus R; Agent Skill Registry S; curriculum D0 ; batch size B; group size K; iterations T ; held-out validation set V 1: for t ← 1 to T do 2: Generator Phase: Construct skill-guided SWE repair tasks 3: Et ← Solver outcomes on Dt 4: for b ← 1 to B do 5: Sample r ∼ R, retrieve skill s ∼ S 6: {(τk , vk )}K ▷ K candidate tasks k=1 ← πθ (· | r, s, Et , role=G) 7: for k ← 1 to K do 8: Validate (τk , vk ) in sandbox r ▷ Format → grounding → execution → semantics 9: if Valid(τk , vk , r) = 1 then 10: Add (τk , vk , r) to Dt+1 11: end if 12: end for 13: end for 14: Solver Phase: Solve accepted tasks in repositories 15: for accepted task (τ, v, r) ∈ Dt+1 do 16: {ŷi }K ▷ K patch trajectories i=1 ∼ πθ (· | τ, r, v, role=S) 17: for i ← 1 to K do 18: Apply patch ŷi , run verifier v, compute rS (ŷi , τ, v, r) via Eq. (10) 19: end for 20: end for 21: Update Phase: Role-specific policy optimization 22: Compute gτ and gval (recomputed on V periodically) 23: Compute RG (τk , vk , r) via Eq. (8) for Generator group 24: Compute ÂkG from RG and update πθ via LG using Eq. (12) 25: Compute ÂiS via Eq. (14), update πθ via LS 26: end for 27: return Trained SWE agent πθ
Appendix A
Training Algorithm
Algorithm 1 summarizes the role-conditioned Generator–Solver training loop used in Socratic-SWE.
B
Baseline Adaptation Details
We compare Socratic-SWE with the base agent and five representative self-evolving methods adapted to the SWE setting. All methods use the same Solver architecture (Qwen3.5-9B), the same SWE benchmark harness (mini-swe-agent), the same Terminal-Bench harness (little-coder), and the same interaction budget. Each method runs for 3 iterations, generating 12k validated training instances per iteration (36k total). Baselines that require seed tasks are initialized from 10% of SWE-smith task instances; Socratic-SWE uses only seed repositories and no pre-existing SWE task instances or repair trajectories. • SPIRAL [31]: A multi-agent multi-turn self-play framework originally designed for zero-sum language games. We adapt it to the SWE setting by modeling repository-level repair as a two-player zero-sum game: one agent injects code defects while the other repairs them, trained with role-conditioned advantage estimation (RAE). • R-Zero [9]: A Challenger–Solver co-evolution framework that requires zero external data. The Challenger proposes SWE tasks and the Solver generates repair patches. Unlike execution-grounded methods, R-Zero uses majority vote across multiple Solver rollouts as the reward signal for both roles. 14
• Absolute-Zero [11]: A single-model self-play paradigm where the agent simultaneously proposes and solves coding tasks with execution-based verification. We adapt it to repository environments, where the model freely proposes code-level repair tasks and solves them using the code executor as verifiable reward, without structured bug artifacts or skill guidance. • Socratic-Zero [10]: A Teacher–Solver–Generator tri-role co-evolution framework originally designed for mathematical reasoning. We adapt it to SWE by using Qwen3.5-397B [35] as the Teacher to construct increasingly challenging SWE tasks from the seed set, with the Generator distilling the Teacher’s taskdesign strategy for scalable curriculum generation. • SSR [29]: Self-play SWE-RL, a native SWE self-play method where a single LLM alternates between injecting bugs (via code removal or history reversion) and repairing them in real repository environments. Bug artifacts are validated through execution-based consistency checks and the Solver receives binary pass/fail rewards.
C
Theoretical Justification of Gradient-Aligned Generator Reward
In §3.3, the Generator is rewarded via the cosine similarity between the policy gradient induced by a candidate task and the aggregated validation gradient. We provide a formal justification for this design.
C.1
Setup
Let πθ be the shared policy, Vval a held-out set of trusted SWE tasks, and Jval (θ) = E(τ,v,r)∼Vval , ŷ∼πθ [rS (ŷ, τ, v, r)] the expected solve rate on the validation set. The RL objective follows GRPO [5].
C.2
Validation-Gradient Estimation
Theorem C.1 (GRPO gradient as solve-rate estimator). Under on-policy sampling with binary episodic reward R(ŷ, τ ) = 1[all tests pass] and without clipping or KL regularization, the GRPO gradient on Vval is unbiased for ∇θ Jval (θ). Proof. The policy gradient theorem gives: ∇θ Jval (θ) = Eτ, ŷ∼πθ R(ŷ, τ ) − b(τ ) ∇θ log πθ (ŷ | τ ) ,
(15)
where b(τ ) = Eŷ′ ∼πθ [R(ŷ ′ , τ )] is any state-dependent baseline (variance reduction without bias). GRPO P 1 R(ŷ , τ ), which is an unbiased estimate of b(τ ) since rollouts estimates b(τ ) with the group mean r̄ = K k k are i.i.d. on-policy. Substituting yields an unbiased Monte Carlo estimator: ĝval =
1 X 1 X Âj,k ∇θ log πθ (ŷj,k | τj ), |Vval | j K k
(16)
E[ĝval ] = ∇θ Jval (θ).
C.3
Direction Preservation Under Normalization
Theorem C.2 (Advantage normalization preserves direction). Dividing advantages by their within-group standard deviation σA > 0 rescales the gradient by a positive scalar, preserving its direction. Proof. Ânorm = Âk /(σA + ϵ) with σA fixed per group implies ĝ norm = ĝraw /(σA + ϵ). A positive scalar does k not change cosine: cos(ĝ norm , ∇θ Jval ) = cos(ĝraw , ∇θ Jval ).
15
Table 3 Generator reward ablation on SWE-bench Verified (Iteration 3). Generator Reward Hardness (1 − p) Uncertainty (1 − 2|p − 0.5|) Variance (Gaussian) Gradient-aligned (Ours) Gradient + Difficulty hybrid
C.4
Verified (%)
∆
47.40 48.20 48.80 50.40 50.60
↓ 3.00 ↓ 2.20 ↓ 1.60
– ↑ 0.20
Why Cosine Similarity Ranks by Validation Improvement
A first-order Taylor expansion of the validation objective after one gradient step on candidate τ yields: ∆Jval ≈ η ⟨gτ , Gv ⟩ = η ∥gτ ∥ ∥Gv ∥ cos(gτ , Gv ),
(17)
where Gv is the aggregated validation-gradient direction and η is the learning rate. Among candidates, η and ∥Gv ∥ are constant. Although ∥gτ ∥ varies across tasks, in practice it can be confounded by task length and patch complexity. Normalizing out the magnitude via cosine similarity yields a scale-normalized proxy for validation-aligned improvement. We do not require exact rank preservation; the reward is designed to prefer candidates whose update directions are better aligned with held-out validation gradients. This motivates Eq. (8). Empirically, Table 6 confirms that cosine (50.40%) outperforms the unnormalized inner product (49.40%), validating the directional focus.
D
Discussion: Generator Reward Strategies
The Generator reward determines which candidate tasks enter the training curriculum. We compare several reward design philosophies.
D.1
Reward Families
These rewards depend on the Solver’s group pass rate p on a candidate: • Variance reward [42]: rG = exp −(Var(l1 , ..., lK ) − 0.25)2 /2σ 2 , peaking at p = 0.5. Difficulty-Aware Rewards.
• Uncertainty reward [9]: rG = 1 − 2|p − 0.5|, a triangular peak at p = 0.5. • Hardness reward [11]: rG = 1 − p, favoring maximal difficulty. All three share the assumption that difficulty ≈ learning value. However, in SWE self-play, many “hard” tasks are hard for irrelevant reasons (e.g., requiring domain knowledge absent from the training distribution), providing no signal for downstream benchmarks. Gradient-Aligned Reward (Ours). Rather than proxying learning value through difficulty, we directly measure the degree to which a candidate task’s optimization direction matches the direction that improves validation performance: RG (τ ) = Valid (τ, v, r) · cos(gτ , Gv ). (18) This decouples difficulty from utility: a moderate-difficulty task that teaches transferable repair patterns scores higher than a near-impossible one whose gradient is orthogonal to Gv . Hybrid: Gradient × Difficulty. We also test gating gradient alignment by a Gaussian difficulty prior: hybrid RG = Valid · cos(gτ , Gv ) · exp −(p − 0.5)2 /0.08 .
16
Table 4 Validation set size sensitivity (SWE-bench Verified, Iteration 3). Gradient stability is measured as the Pearson correlation between Gv vectors from two independent rollout sets. |Vval | 20 50 100 (default) 200
D.2
Verified (%) 49.20 50.00 50.40 50.60
Gv Stability
∆
0.34 0.52 0.61 0.74
↓ 1.20 ↓ 0.40
– ↑ 0.20
Ablation Results
Analysis. The Hardness reward (−3.00) produces the largest degradation because near-impossible tasks yield vanishing advantages (all rollouts fail) and thus uninformative gradients. The Variance reward performs best among difficulty-only baselines, confirming the intuition from SPICE [42] that targeting the frontier (p ≈ 0.5) is preferable to maximizing difficulty. However, it still trails our gradient-aligned reward by 1.60 points because being at the frontier does not guarantee relevance to the target distribution. The hybrid barely improves over pure gradient alignment (+0.20), indicating that the cosine score already encodes appropriate difficulty implicitly: tasks that are trivial (Â ≈ 0 since all pass) or impossible (Â ≈ 0 since all fail) produce near-zero gradients and thus receive low cosine scores by construction. Qualitative Insight. We inspected tasks selected under each reward at Iteration 2. The Variance reward frequently selects tasks requiring obscure library internals (e.g., C-extension edge cases) where the Solver occasionally gets lucky but learns no transferable skill. The gradient-aligned reward preferentially selects tasks involving common repair patterns (e.g., off-by-one in iteration bounds, incorrect argument ordering) that appear frequently in the validation set, producing broadly useful gradient signal.
E
Validation Set Design and Sensitivity
E.1
Design Principles
The validation set Vval provides a stable reference direction for the Generator reward. It is never trained on—its sole purpose is to produce Gv , the target gradient. We construct Vval from a held-out subset of BeyondSWE, selecting 100 tasks that span diverse repositories and difficulty levels. To avoid distribution bias, we stratify by repository language (Python 40%, JavaScript 30%, TypeScript 20%, Go/Rust 10%). The validation set is never used for Solver training or final evaluation.
E.2
Sensitivity to Validation Set Size
With only 20 tasks the gradient estimate is too noisy (correlation 0.34) and performance drops 1.20 points. At 100 tasks the signal stabilizes (correlation 0.61); doubling to 200 yields diminishing returns (+0.20). We therefore fix |Vval | = 100 throughout.
E.3
Recomputation Frequency
We refresh Gv once per iteration (every 12k training instances). Ablating this choice: recomputing every 4k instances gives only +0.20 at 3× validation cost, while using a single fixed Gv across all iterations degrades by −1.40 as the policy’s optimization landscape drifts.
F
Computational Overhead
F.1
Cost Breakdown
The gradient-alignment mechanism adds two compute steps: (i) rolling out Solver trajectories on 100 validation tasks to form Gv (once per iteration), and (ii) scoring each candidate task’s gradient against Gv . The gradient-alignment overhead totals 1.3 h/iteration (8.4%). Key observations:
17
Table 5 Wall-clock time per iteration (8×A100-80G). Stage
Time (h)
%
Generator: task proposal + validation Solver: rollout + execution feedback Policy update (GRPO/GDPO)
4.2 7.8 1.7
28.0 52.0 11.3
Gradient alignment: Gv computation (800 rollouts) Per-candidate scoring
0.5 0.8
3.3 5.1
Total
15.0
100
Table 6 Cosine vs. inner product for gradient scoring. Metric
Verified (%)
∆
49.40 50.40
↓ 1.00
Inner product Cosine similarity
–
• Gv computation requires 800 rollouts (100 tasks × K = 8), negligible next to the ∼96k Solver rollouts for training. • Per-candidate scoring reuses rollouts already generated during the semantic validation gate (f4 ). The cosine itself is a single dot product (<1s per candidate). • Across 3 iterations, cumulative overhead is ∼3.9 wall-clock hours (< 9% of the 45h wall-clock budget). Compared to SSR [29] which trains on all validated tasks indiscriminately, Socratic-SWE achieves +3.40 higher Verified at comparable total FLOPs, demonstrating that the modest selection cost is recovered through better data efficiency.
F.2
Alignment Metric Ablation
The inner product conflates direction with magnitude: multi-file patches produce large gradients irrespective of their utility, biasing selection toward complex but irrelevant tasks. Cosine isolates the directional signal (cf. Theorem C.2).
G
Prompt Templates and Implementation Details
This section provides shortened versions of the prompts and key configuration.
G.1
Generator System Prompt
Generator System Prompt (Shortened) <ROLE> You are a bug injection agent for the Socratic-SWE training pipeline. Your job is to introduce exactly one realistic, targeted bug into a repository located at /testbed/. </ROLE> <INTERFACE> You interact with the repository only through a Linux Bash shell. Use ordinary shell commands to inspect files, search code, edit files, run tests, and inspect git diff. Do not assume access to any structured file editor or search tool. Issue at most one shell command per turn and wait for the command output before continuing. </INTERFACE> <BUG_INJECTION_RULES> 1. Inject exactly one atomic semantic mistake in one production file. 2. Do not modify tests. 3. Do not introduce syntax errors, import errors, or changes that prevent the module from loading. 4. The bug must be reversible: the original code is the reference fix. 5. Keep the diff minimal and free of unrelated cleanup.
18
6. Do not add comments, logs, TODOs, or variable names that reveal the bug. </BUG_INJECTION_RULES> <WORKFLOW> 1. Inspect the repository and identify a plausible target location. 2. Identify visible tests or behavior that should expose the injected bug. 3. State the intended semantic change before editing. 4. Make one contiguous source-code edit using Bash-accessible file operations. 5. Run the relevant visible test(s) and confirm that the target behavior fails. 6. Run collateral checks when feasible to avoid broad breakage. 7. Inspect git diff and stop once a clean single-bug diff is obtained. </WORKFLOW>
G.2
Mini-SWE-agent Prompt
System Prompt You are a helpful assistant that can interact with a computer shell to solve programming tasks.
Base Task Instructions (Shortened) Given a task description, the agent interacts with a Linux shell in /testbed to make the required source-code changes. The agent is instructed to: 1. inspect the repository and identify relevant files; 2. reproduce or understand the issue when possible; 3. modify only source files needed for the task; 4. verify the change by running visible checks when available; 5. test edge cases when feasible; 6. leave a clean git diff containing only the intended changes; 7. finish according to the configured mini-swe-agent completion protocol. The agent should not modify tests, generated files, build artifacts, or unrelated configuration files unless they are directly required by the task.
G.3
Base Mini-SWE-agent Prompt
Solver Task Prompt Template Fix the issue described below. <ISSUE> {{ problem_statement }} </ISSUE> <TASK> You are in the repository root at /testbed/. Use Bash commands to inspect the codebase, modify production source code, run visible tests when available, and leave the repository in a fixed state. Do not modify tests. </TASK> <TESTING> If the issue text includes a public reproduction command or visible test, run it to confirm the failure and rerun it after the fix. If no such command is provided, identify and run the most relevant visible tests you can. Hidden benchmark tests are never provided. </TESTING>
G.4
Training Hyperparameters
Table 7 lists all training hyperparameters.
19
Table 7 Training hyperparameters for Socratic-SWE.
H
Hyperparameter
Value
Model Configuration Base model Skill extractor Context length
Qwen3.5-9B Qwen3.6-27B 32,768 tokens
Training Iterations Instances per iteration Group size (K) Learning rate Optimizer KL coefficient (β) Clip ratio (ϵ) Batch size
3 12,000 8 1 × 10−6 AdamW 0.01 0.2 64
Solver Reward (GDPO) λ1 (full pass) λ2 (repair rate) λ3 (regression avoidance)
0.5 0.3 0.2
Gradient Alignment Validation set size Rollouts per validation task Recomputation frequency Alignment metric
100 8 Once per iteration Cosine similarity
Generator Validation Gate Max attempts per repository Stability reruns
8 3
Infrastructure GPUs Sandbox Wall-clock per iteration
8× A100-80G Aone Cloud Sandbox (per-task) ∼15 hours
Worked Example of Skill-Guided Task Generation
Shortened Repository-Level Skill Guidance # OAuthLib Repository Skill ## 1. Target Overview OAuthLib implements OAuth 1.0, OAuth 2.0, OpenID Connect, and related RFC extensions. High-value tasks in this repository are usually small semantic regressions in protocol helpers, endpoint constructors, and grant wrappers. They require exact repository-specific semantics rather than broad rewriting. Relevant modules: - oauthlib/oauth2/rfc6749/utils.py - oauthlib/oauth2/rfc8628/endpoints/device_authorization.py - oauthlib/oauth1/ - oauthlib/openid/connect/core/grant_types/ Common conventions: - Scope helpers preserve exact conversion behavior. - Constructors store protocol configuration without transformation. - OAuth1 parameters are string-like but semantically distinct. - OpenID Connect grants extend or wrap OAuth2 grant behavior. ## 2. Solver Weakness Analysis Historical Solver traces show four recurring weaknesses. 1. Scope conversion: the Solver localizes list_to_scope or scope_to_list but uses generic filtering, sorting, or normalization that violates OAuthLib's helper contracts. 2. Constructor storage: endpoint and client constructors contain same-typed parameters, causing the Solver to swap fields, transform values, or store parameters under the wrong private attributes. 3. OAuth1 plumbing: nonce, timestamp, realm, and callback_uri are all
20
string-like but semantically different, so type information alone is insufficient. 4. OIDC inheritance: OpenID Connect grants inherit from or delegate to OAuth2 grant logic. The Solver may replace inherited behavior instead of extending it, or forward attributes to the wrong object. Representative traces include failures in scope utility tests, device endpoint tests, OAuth1 client/signature tests, and OpenID Connect grant-type tests. ## 3. Bug Injection Playbook ### Pattern A: Scope Helper Contract Targets: - oauthlib/oauth2/rfc6749/utils.py - list_to_scope, scope_to_list, params_from_uri Mutation ideas: - Convert only some scope elements instead of all elements. - Reorder tokens during list/string conversion. - Mishandle None, empty strings, or empty collections. - Replace repository-specific conversion with generic filtering. ### Pattern B: Device Endpoint Constructor Storage Target: - oauthlib/oauth2/rfc8628/endpoints/device_authorization.py Mutation ideas: - Swap verification_uri and verification_uri_complete. - Store interval or expires_in under the wrong attribute. - Transform a constructor value that should be preserved unchanged. ### Pattern C: OAuth1 Parameter Plumbing Target: - oauthlib/oauth1/ Mutation ideas: - Confuse nonce and timestamp. - Confuse callback_uri and realm. - Keep most fields correct so the bug remains localized. ### Pattern D: OIDC Inheritance and Delegation Target: - oauthlib/openid/connect/core/grant_types/ Mutation ideas: - Drop inherited OAuth2 response fields. - Forward a method to the wrong wrapped object. - Override a method without preserving parent behavior. ## 4. Validation and Anti-Patterns Use existing targeted pytest tests, not synthetic tests or full-suite execution. A candidate is accepted only if the bug patch applies cleanly, target tests fail, the reversed patch restores passing tests, repeated runs are stable, and the issue text does not leak the oracle patch. Do not generate tasks that modify tests, introduce syntax/import errors, require network access, touch many unrelated modules, reveal the patch in the issue text, or produce failures that cannot be reproduced by targeted pytest commands.
21