ExecuCritic: Calibrated Critic Shaping for Code Generation with Verifiable Rewards Junjie Cao1 and Yingjie He2 Intel Corporation, Beijing, China [email protected] 2 Peking University, Beijing, China
arXiv:2609.16604v1 [cs.SE] 15 Sep 2026
1
Abstract. Execution feedback is a useful supervision signal for code models because unit tests are objective and directly measure program correctness. Its weakness is that an entire program is often reduced to one pass or fail bit, leaving RLVR to solve a difficult credit assignment problem. At the same time, coding systems often include separate reviewer or tester roles, but these critics are usually prompted rather than trained and are not calibrated against execution. We propose ExecuCritic, a joint training framework in which a coder and a critic are updated on the same execution rollouts. The critic predicts pass or fail outcomes and gives short diagnostic feedback; the coder uses this signal only when the critic agrees with the executor on the current rollout group. Across eight code benchmarks and two recent open backbones, ExecuCritic improves over GRPO without a critic, prompted reviewer systems and scalar reward model baselines, while requiring fewer policy gradient steps and fewer sandbox executions. Ablations and reliability analyses suggest that the gains come from better credit assignment rather than larger sampling budgets. Keywords: Code generation · Verifiable reward · Multi agent learning · Reinforcement learning · Critic models.
1
Introduction
Code language models [6] are increasingly evaluated as software agents rather than as autocomplete systems [49,51]. In a realistic coding task, a model may need to understand an issue, modify several files, run tests, read the resulting trace and revise its patch. This workflow has made execution feedback especially attractive: a unit test is cheap compared with human annotation, and its result is directly tied to whether the program works. Reinforcement learning with verifiable rewards (RLVR) builds on this property by fine tuning code models with policy gradient methods such as GRPO [38] using rewards from sandboxed execution. Recent code RL systems [7,43,48,27] show that even binary test feedback can improve coding and reasoning ability, and parallel work on reward design for LLM reasoning reports similar gains from denser or better aligned signals [50,39,22].
2
J. Cao et al.
A second line of work improves code generation by decomposing the task into roles. Several systems built around prompting [12,10,13,31] introduce programmers, reviewers, testers or debuggers, so that a candidate program can be checked before it is submitted. The analogy to ordinary software development is clear: code is reviewed, tested and diagnosed rather than written once and accepted. However, in most such systems the critic is only a prompted role. It may produce fluent comments, but it is not trained to match executor outcomes, and the coder is not trained to know when those comments should be followed. The two approaches therefore solve different parts of the problem. RLVR has a trustworthy verifier, but the learning signal is thin. A long patch receives the same binary credit on every generated token, even though the failure may be caused by one branch condition, one API call or one missing import. This leads to noisy updates whose effect is hard to attribute to a specific mistake [53,16], and often encourages large rollout groups, which are expensive when every candidate must be executed. Prompted systems with multiple roles provide richer language feedback, but the feedback is not grounded in the actual executor. A critic can be confident and wrong, and nothing in the training objective teaches the coder to discount such advice. This motivates the question we study in this paper: can execution rewards and learned critiques be combined so that the critic makes RLVR less sparse without becoming an untrusted reward model? We propose ExecuCritic, a joint training framework for a coder agent πc and a critic agent πv . The two agents share a frozen backbone and use separate LoRA adapters [11], and both are updated on the same pool of rollouts labelled by execution. For each prompt, the coder samples candidate programs and a sandbox returns pass or fail outcomes. The critic predicts these outcomes, ranks passing candidates above failing ones and produces a short diagnosis tied to the observed failure type. The coder is then trained with a calibrated advantage: the standard GRPO signal is augmented with a critic score, but only to the extent that the critic agrees with the executor within the current rollout group. When the critic is unreliable, the update falls back toward ordinary RLVR; when it becomes calibrated, its scores provide a dense signal for credit assignment. The critic is also not an extra frozen judge bolted on after training: it has seen the same evolving distribution of code as the coder, so it can rank candidates before execution and supply feedback when all of them fail. In repository repair, where one sandbox call may install dependencies and run many tests, this ranking cuts expensive executions without simply enlarging the sampling budget. We evaluate ExecuCritic with two recent open code backbones on benchmarks spanning function synthesis, contamination controlled programming, library use, competitive programming, repository issue resolution, cross-language editing, patch ranking and test selection. Under matched data and token budgets it improves over the strongest critic-free RLVR baseline by 3–4 absolute points and over prompted reviewer systems by 6–7, and reaches that baseline’s final accuracy with roughly 40% fewer policy gradient steps. Ablations tie the
ExecuCritic: Calibrated Critic Shaping
3
gains to the calibration gate, the pass/fail margin and the diagnosis consistency objective rather than to extra sampling. Contributions. – We formulate code agent training as joint optimisation of a coder and an execution-grounded critic updated on the same RLVR rollouts. – We introduce a calibrated advantage that trusts critic scores only where they agree with executor outcomes, giving denser feedback with less risk of reward hacking. – We train the critic to predict verdicts, rank passing above failing programs, and diagnose failures from executor traces. – We study two backbones and eight benchmark views with ablations, seed checks, reliability, sample efficiency, budget and failure-type analyses.
2
Related Work
RL with verifiable rewards for code. Following recent work on RLVR for code [7,43,35,48,27,28], a wave of studies has applied executable feedback to code generation. These systems optimise patching at repository scale or function synthesis with curated data augmented by tests. They demonstrate the power of executable supervision, but they still train a single policy against a single scalar reward and therefore inherit the sparsity problem studied in this paper. Related efforts outside code shape the reward itself rather than the data, by aligning updates with gradient evidence [50], by distributing group relative credit across reasoning steps [39], or by replacing surface overlap metrics with hierarchical task aware rewards [42,22]. Code generation with multiple agents. A separate line of work keeps the model frozen and orchestrates several roles through prompting. Self-Refine [31] iterates between generation and critique with one model; AgentCoder [12], MetaGPT [10], MapCoder [13] and AutoGen-Coder [45] introduce dedicated programmer, tester and reviewer agents. CodeT [2] and Reflexion [40] similarly use generated tests or verbal feedback to filter or rewrite candidates. Learned coordination policies decide which agent or action to invoke instead of fixing the schedule in advance [15,33], and similar role decompositions have been applied to multimodal decision tasks [19]. These pipelines benefit from role decomposition, but their critics are generally frozen and uncalibrated. ExecuCritic instead trains the critic and coder jointly under a verifiable reward. Critic and verifier models. Process reward models [24] and CriticGPT [32] demonstrate that critics trained on labelled traces can rival much larger frozen judges. Outcome reward models for math and code [3] extend this idea to scalar correctness prediction. Analyses of how reasoning models fail, whether by enumerating an error forest [16] or by making the reasoning pathway explicit and
4
J. Cao et al.
controllable [4], suggest that a critic benefits from naming the failure rather than only scoring it. Our work differs in two key ways: the critic is trained simultaneously with the policy whose outputs it scores, and its score is fed back into the policy gradient rather than used only during decoding. Reward shaping and dense feedback. Reward shaping is a classical idea in RL [34]; recent LLM work explores intrinsic rewards [47], learned dense verifiers [41] and feedback generated by AI systems [1]. When several learning signals disagree, reweighting them by their mutual consistency is often more stable than summing them, an idea also used to resolve conflicting client updates in distributed training [9]. ExecuCritic can be viewed as reward shaping grounded by a verifier: the dense bonus is provided by a learned critic, but the critic itself remains anchored to a verifiable executable signal, which empirically helps prevent reward hacking.
3
Method
3.1
Problem setup
We study a code generation setting in which a prompt x (a natural language specification, a function signature, or a buggy file plus an issue description) is mapped to a candidate solution y (a function body or a unified diff). A black box executor E(x, y) ∈ {0, 1} returns 1 iff y passes all hidden unit tests for x and 0 otherwise. RLVR maximises the expected execution reward J(π) = Ex∼D Ey∼π(·|x) E(x, y) , (1) typically with GRPO, which draws a group of K rollouts {yi }K i=1 for each prompt and computes an advantage Ai = (Ri − µR )/σR from rewards normalised within the group, where Ri = E(x, yi ). The advantage is then used in a clipped policy gradient update [37]. ExecuCritic extends this setup with a second policy πv , the critic, which receives the prompt and a candidate solution and produces both a critique in natural language c and a scalar score ŝ ∈ [0, 1]. Both πc and πv are implemented as LoRA adapters on top of a shared frozen backbone π0 , so the additional parameter cost over RLVR with only the coder remains modest. Unlike a conventional reward model, πv is trained on the coder’s evolving output distribution, encouraging it to recognise the mistakes that the current coder actually makes. 3.2
Coder and critic architecture
Figure 1 summarises the training and inference pipeline. The coder πc (y | x) is a standard autoregressive policy. The critic πv (c, ŝ | x, y) first emits a short critique in natural language c, followed by a special <verdict> token whose probability mass on pass/fail is read as the calibrated score ŝ. Placing the critique before the verdict provides a mild form of reasoning regularisation: in our experiments it improves calibration relative to a direct regression head while preserving interpretability.
ExecuCritic: Calibrated Critic Shaping STAGE 1: Cold start SFT
SFT data
Evol-Code
OSS-Instruct
1. Coder Rollouts x Prompt
!"
SFT
Sandbox
…
!#
x
STAGE 2: Joint RLVR 2.Shared Rollout Pool
Program y
Reward R Critic Score S Critique
1
0.86
!"
0
0.23
Off-by-one in boundary check
… 0
…
1
/&'() % Reward Baseline
0% Critic Score
2+ ∈ [0,1] Critic Reliability
Write a function to merge two sorted arrays
Predict pass prob 0
"$ ∈ {%, '} )*++/-*..
Generate N candidates !! !#
…
…
…
α
Critic Ranks !! !0
0.92 0.71
!+
0.08
…
0.25
0.50 0.75 1.00
Generate concise critique *!
(i)Outcome Alignment
(ii)Pairwise Pass Fail Margin
(iii)Critique Consistency
BCE(!", $! )
Max(0, & − (!"!# , !"$% ))
CE(*! , +! )
Scale
STAGE 3: Test-time Refine and Execute Input Prompt x
4.Grounded critic
7.Critic objective
!"! = !#$%& + α ' (' ' )"! !
l Clipped PPO/GRPO Surrogate
Warm start checkpoint
Execute Critique
Wrong use of API
0.11
"#$
!%
z
Distilled Critique
3.Sandbox Executor
!!
…
v
C
Handles edge case when a==b
!#
-./
y
x
Prompt Program Executor Verdict
5.Calibrated Critic-aware Advantage
6.Coder Objective ℒ! l KL Regularization to 8,
Warm start Data(Triplets)
"#$
!!
(SFT Checkpoint)
K Programs Prompt x
!!
5
Execute Top-k
Return & Repeat All Pass? NO
YES
Append Critique
Fig. 1. Pipeline of ExecuCritic. Stage 1 cold starts the coder with SFT and the critic with warm start triplets of prompts, programs, executor verdicts and distilled critiques. Stage 2 trains both on shared rollouts: the sandbox gives pass/fail rewards, the critic predicts pass probability and diagnoses failures, and the coder is updated with a calibrated advantage. Stage 3 uses the critic at test time to rank candidates, execute only the top ones, and feed critiques back when all candidates fail.
3.3
Calibrated advantage from the critic
The core of ExecuCritic is a modification of the GRPO advantage that uses the critic’s signal only after checking calibration. For a group of K rollouts {yi } with execution rewards Ri and critic scores ŝi , we define the standardised execution advantage Ai = (Ri − µR )/(σR + ϵ) and the standardised critic advantage Si = (ŝi − µŝ )/(σŝ + ϵ). The resulting advantage is Ãi = Ai + α ρK R, ŝ Si , (2) where ρK (R, ŝ) ∈ [−1, 1] is the rank correlation within the group between executor rewards and critic scores, and α > 0 is a fixed shaping weight. The factor ρK is the key design choice: when the critic is well calibrated on a particular prompt (ρK → 1), the critic term reinforces the executor signal and spreads it across rollouts; when the critic is miscalibrated or adversarial (ρK ≤ 0), the term collapses or changes direction, preventing unreliable scores from being treated as trustworthy rewards. This behaviour stabilises joint training before the critic has fully converged. 3.4
Critic objective
The critic is updated on the same rollouts. Let vi ∈ {0, 1} be the verdict token target derived from Ri , and let zi denote a compact failure descriptor obtained from the executor trace when the program fails (syntax error, assertion mismatch, timeout, import/API error or runtime exception). We train the critic
6
J. Cao et al.
with a loss that ties its verdict, relative ordering and textual diagnosis to the executor feedback: Lv = − Ei log πv (vi | x, yi ) h i + β E(i+ ,i− ) max 0, m − (ŝi+ − ŝi− ) + γ Ei:Ri =0 − log πv (zi | x, yi ) . (3) The first term is the likelihood of the executor verdict. The hinge term is used only for rollout groups that contain at least one passing program i+ and one failing program i− ; it asks the critic to score the passing program at least m higher. The last term is applied to failed rollouts and trains the critique to match the failure type extracted from the executor trace. In practice, the hinge term avoids a nearly constant verdict score, and the diagnosis term makes the feedback useful when the coder resamples during inference. 3.5
Coder objective
The coder is optimised with a clipped PPO/GRPO surrogate using the advantage in Eq. (2): i X h Lc = −Ei min ri,t Ãi , clip(ri,t , 1−ε, 1+ε) Ãi + λ KL πc ∥ πref , (4) t
where ri,t = πc (yi,t | x, yi,<t )/πc,old (yi,t | x, yi,<t ) is the importance ratio and πref is the SFT checkpoint used as a reference. The KL term, set with λ small but nonzero, prevents the coder from drifting into degenerate styles that the critic happens to overrate. 3.6
Training pipeline
ExecuCritic is implemented in three stages. Stage 1: SFT initialisation. We fine tune the coder with supervision on a mixture of OSS-Instruct [44] and Evol-Code [30] until it reliably produces outputs that compile and can be tested. The critic is initialised on triplets (x, y, v, z), where v is the executor verdict and z is either a distilled critique from a stronger teacher (e.g. DeepSeek-V3) or a normalised failure descriptor from the executor trace. This stage requires only a few thousand examples and avoids learning diagnostic language entirely from scratch. Stage 2: Joint RLVR. On each prompt we draw K=8 coder rollouts, score them with both the executor and the current critic, and apply Eqs. (2)–(4). The two LoRA adapters are updated sequentially within each optimiser step: first the critic, then the coder using calibrated advantages computed from the same rollout-level critic scores, while sharing the rollouts and the frozen backbone. Algorithm 1 gives the full procedure.
ExecuCritic: Calibrated Critic Shaping
7
Algorithm 1 ExecuCritic joint training (one outer iteration) Require: dataset D, executor E, coder πc , critic πv , group size K 1: for minibatch of prompts {x(b) } ⊂ D do 2: for each prompt x do 3: Sample K rollouts {yi } ∼ πc (· | x) 4: Compute executor rewards and failure descriptors (Ri , zi ) ← E(x, yi ) 5: Compute critic scores, critiques and verdict logits with πv (· | x, yi ) 6: Compute Ai , Si and the rank correlation ρK (R, ŝ) 7: Form calibrated advantage Ãi ← Ai + α ρK Si 8: end for 9: Update πv with Lv from Eq. (3) 10: Update πc with Lc from Eq. (4) using Ãi 11: end for
Stage 3: Refinement at inference. At inference, the coder produces N candidates; the critic ranks them, and only the top k are executed in the sandbox. If none passes, the critic’s textual critique on the failure with the highest score is fed back to the coder as additional context, and a new round of N candidates is sampled. We cap the loop at T rounds. Because critic scoring requires a single forward pass whereas sandbox execution can take seconds, ranking with the critic substantially reduces running time relative to executing all sampled candidates.
4
Experiments
4.1
Setup
Backbones. We use Qwen3-8B [46] as the headline backbone and DeepSeekCoder-V2-Lite-Instruct (16B MoE, ∼2.4B active) [54] to check that the result carries across architectures, plus the smaller Qwen3-4B in scaling ablations. All training uses rank-32 LoRA adapters on attention and MLP projections with the backbone frozen, in bf16 with gradient checkpointing. Benchmarks. We evaluate on eight benchmark views covering increasingly difficult forms of code intelligence. HumanEval+ and MBPP+ [26] test short function synthesis with strengthened unit tests. LiveCodeBench v5 [14] measures competitive programming with contamination controls. BigCodeBenchHard [55] stresses library use, long instructions and reasoning over multiple calls. APPS [8] and CodeContests [23] cover longer contest-style programs with more ambiguous intermediate credit. SWE-bench Lite [17] evaluates issue resolution at repository scale on 300 real GitHub tasks. Multi-SWE-bench evaluates repository repair across Python and Java subsets. We further include a SWE-bench Verified ranking view, where the critic ranks a small set of candidate patches before execution, and a view for selecting which generated tests should be trusted as auxiliary verifiers. Function synthesis benchmarks report unbiased pass@1 with greedy decoding; repository benchmarks report resolved issue rate; ranking views report NDCG@5 and AUC.
8
J. Cao et al.
Training data. The initial SFT stage uses a mixture of 75 k examples from OSS-Instruct [44] and Evol-Code-Alpaca, filtered to keep examples that compile in the sandbox. RLVR prompts are drawn from the train splits of the Code Contests corpus [23], the SWE-Gym training set [35], a 30 k prompt subset of APPS [8], and a 12 k subset of repository edits whose tests run in Docker within 120 seconds. We deduplicate against all evaluation benchmarks at the prompt level using MinHash with Jaccard threshold 0.7, and remove near duplicate function signatures by exact AST normalisation. Critic failure descriptors are normalised into seven categories: syntax, import/API, type, assertion, timeout, runtime exception and unknown.
Baselines. We compare ExecuCritic with: (B1) the base backbone without task training; (B2) a coder trained only with SFT; (B3) RLVR with a single coder, using vanilla GRPO and executor rewards to reproduce the SWE-RL/AceCoder recipe; (B4) a prompted system with a frozen coder and frozen critic in the style of AgentCoder; (B5) Self-Refine [31] with the same coder; (B6) trained critic reranking, with the coder left at the SFT checkpoint; (B7) execution of all sampled candidates across refinement rounds; and (B8) outcome reward model shaping, which replaces the textual diagnosis with a scalar reward model. Trained baselines (B2, B3, B6, B8) share our backbone, prompts and token budget; prompted ones (B1, B4, B5, B7) reuse the backbone untrained. B3 uses K=16 rollouts to match or exceed our training-time sandbox calls.
Hyperparameters. Unless stated otherwise we use K=8 rollouts per prompt, α=0.5, β=0.5, γ=0.2, margin m=0.2, PPO clipping ε=0.2, KL coefficient λ=0.01, learning rate 5×10−6 for both adapters, batch size 128, and a total of 400 RLVR steps. At inference we use N =8, k=3, T =3 for the loop that refines candidates after failed execution. For repository tasks, each candidate is allowed at most 120 seconds of test execution and 16k input tokens.
4.2
Main results
Tables 1 and 2 report the main results. On Qwen3-8B, ExecuCritic improves over RLVR with a single coder by +3.4 average points across the six function and programming benchmarks and by +3.7 on SWE-bench Lite; on DeepSeekCoder-V2-Lite the margin remains +3.3, so the gain is not an artefact of a weak baseline. The critic also raises patch ranking NDCG@5 by +11.1 and cuts sandbox executions per solved task from 18.6 to 10.7. The gains are largest on LiveCodeBench, APPS, CodeContests and SWEbench Lite, where generations are longer and credit assignment is noisier. Prompted reviewer and Self-Refine baselines remain below RLVR with a single coder, indicating that role decomposition alone is insufficient; the critic must be calibrated and trained with the coder.
ExecuCritic: Calibrated Critic Shaping
9
Table 1. Function synthesis and programming benchmarks (pass@1, %), with a shared backbone and training budget. Best in bold, second best underlined. HE+, MBPP+, LCB and BCB-H are HumanEval+, MBPP+, LiveCodeBench and BigCodeBenchHard. Method (Qwen3-8B)
HE+ MBPP+ LCB BCB-H APPS CC
B1 Base direct B2 SFT only B4 Prompted reviewer system B5 Self-Refine B6 Trained critic rerank B7 Execute all samples B3 RLVR with one coder Ours: ExecuCritic
71.3 74.6 75.9 76.2 77.4 79.8 80.5 84.2
65.8 68.4 69.5 70.0 71.2 73.0 73.6 77.0
24.7 26.2 27.0 27.6 28.3 30.4 30.9 34.1
28.1 30.0 31.2 31.6 32.4 34.0 34.5 37.8
19.4 18.8 38.0 21.1 20.5 40.1 22.0 21.4 41.2 22.4 21.9 41.6 23.0 22.5 42.5 24.7 24.0 44.3 25.2 24.6 44.9 28.8 28.0 48.3
Method (DS-Coder-V2-Lite) HE+ MBPP+ LCB BCB-H APPS CC B3 RLVR with one coder Ours: ExecuCritic
82.1 85.6
74.8 77.9
31.5 34.7
35.8 38.6
Avg.
Avg.
26.0 25.4 45.9 29.5 28.7 49.2
Table 2. Repository tasks and critic evaluation. Repair columns report resolve rate; Rank and Test-AUC report NDCG@5 and AUC; Exec. is sandbox calls per solved instance. Method (Qwen3-8B)
SWE-Lite MSB-Py MSB-Java Rank Test-AUC Exec.
B2 SFT only B4 Prompted reviewer system B5 Self-Refine B6 Trained critic rerank B7 Execute all samples B3 RLVR with one coder Ours: ExecuCritic
4.3
9.7 11.0 11.7 12.4 14.0 14.6 18.3
8.2 9.1 9.6 10.0 11.3 11.7 14.4
5.9 6.4 6.8 7.1 7.9 8.3 10.6
42.5 48.8 50.2 59.6 – 55.8 66.9
57.1 60.4 61.0 67.5 – 64.2 72.8
17.8 16.9 16.1 13.4 24.0 18.6 10.7
Statistical robustness
Table 3 checks whether the gains fall within normal RLVR variability. Across three seeds, all paired confidence intervals exclude zero, and ExecuCritic shows slightly lower variance on short programming benchmarks. The widest interval occurs on SWE-bench Lite because only 300 repository issues are evaluated, but its lower bound remains above +1.9 points, making a seed artefact unlikely. 4.4
Ablation study
Table 4 shows that no single component explains the whole gain. Removing critic shaping nearly reverts to RLVR, while a constant trust weight (ρK = 1) hurts when early critic scores are wrong. Dropping the pass/fail margin weakens discrimination, and dropping critique consistency weakens inference-time refinement; the scalar reward model behaves similarly, so textual diagnosis matters
10
J. Cao et al.
Table 3. Robustness of the Qwen3-8B headline comparison: mean ± standard deviation over three RLVR seeds, with a paired bootstrap 95% confidence interval for the gain over RLVR with a single coder. Evaluation
B3 RLVR ExecuCritic Gain
HumanEval+ pass@1 MBPP+ pass@1 LiveCodeBench pass@1 BigCodeBench-Hard pass@1 SWE-bench Lite resolve SWE-bench ranking NDCG@5
80.5±0.8 73.6±0.7 30.9±0.6 34.5±0.7 14.6±0.8 55.8±1.0
84.2±0.6 77.0±0.5 34.1±0.7 37.8±0.6 18.3±0.9 66.9±1.2
95% CI
+3.7 [+2.4, +5.0] +3.4 [+2.1, +4.6] +3.2 [+1.8, +4.5] +3.3 [+2.0, +4.7] +3.7 [+1.9, +5.5] +11.1 [+8.4, +13.8]
Table 4. Ablation on Qwen3-8B: average pass@1 over HE+/MBPP+/LCB/BCBH/APPS/CC, and SWE-bench Lite resolve rate. Bold marks the default setting; K=16 doubles the training sandbox budget. Variant Full ExecuCritic (default) α=0 (no critic shaping) no calibration gate (ρK = 1) β=0 (no pass/fail margin) γ=0 (no critique consistency) reward model with scalar score no shared backbone staged critic then coder K=4 rollouts K=16 rollouts T =1 inference refinement
Avg. pass@1 SWE-Lite 48.3 45.2 46.0 46.3 46.7 46.5 46.9 46.2 46.1 48.5 46.9
18.3 15.0 15.8 16.1 16.4 16.0 16.7 15.9 15.7 18.5 16.8
when tied to executor failure types. Sharing the backbone helps but is not decisive, and staged training loses more, supporting joint updates on one rollout distribution. Raising K to 16 gains little while doubling sandbox cost, so we use K=8, T =3.
4.5
Sample efficiency and reward density analysis
Figure 2 reports three diagnostics. ExecuCritic reaches the final accuracy of RLVR with a single coder after roughly 60% of the policy-gradient steps. Its advantage estimate also becomes less noisy once ρK exceeds ∼0.4, giving a more informative gradient. Meanwhile, the rank correlation between critic scores and executor outcomes rises from 0.18 after SFT to 0.71 at convergence, suggesting that the critic learns execution correctness rather than coder style.
ExecuCritic: Calibrated Critic Shaping pass@1
ρK
variance (a) sample efficiency
(b) advantage variance A
steps 200
(c) critic calibration
Ã
ExecuCritic RLVR without critic 0
11
Spearman AUC steps
steps
400
Fig. 2. Sample efficiency and reward density. (a) Pass@1 vs. GRPO steps; ExecuCritic reaches the final RLVR accuracy after about 240 steps. (b) Advantage variance within rollout groups. (c) Critic calibration (Spearman correlation and pass/fail AUC) during joint training. Table 5. Reliability of critic variants on validation rollouts. ECE and Brier measure calibration (lower is better), AUC and Spearman measure pass/fail discrimination, and Type-F1 measures agreement between the diagnosed failure type and the executor descriptor. Critic variant
ECE↓ Brier↓ AUC↑ Spearman↑ Type-F1↑
Frozen prompt critic 0.213 Trained critic reranker 0.124 Reward model with scalar score 0.108 ExecuCritic critic 0.061
4.6
0.246 0.184 0.171 0.137
60.4 67.5 68.9 72.8
0.31 0.52 0.54 0.71
38.2 53.0 – 64.6
Critic reliability analysis
Table 5 makes the calibration claim more explicit. We compute ECE on validation rollouts from HumanEval+, LiveCodeBench, APPS and SWE-bench Lite, and Type-F1 only on failed candidates with a normalised executor descriptor. A frozen prompt critic is fluent but often over-predicts pass outcomes on plausible wrong programs. Reranking training improves discrimination, but joint learning further improves calibration and diagnostic faithfulness. The scalar reward model obtains reasonable scores but lacks failure-type diagnoses, explaining its weaker refinement performance in Table 4. 4.7
Execution budget and scaling
Table 6 fixes the maximum sandbox executions at inference. ExecuCritic dominates the Pareto frontier: with two executions per prompt it exceeds RLVR with a single coder using eight executions, and with eight executions it surpasses exhaustive execution of 24 candidates. This matters most for repository repair, where each sandbox call may require dependency installation and full test execution. Table 7 shows that ExecuCritic is not simply buying accuracy with more sampling. Across all three backbones, it uses less than half as many sandbox
12
J. Cao et al.
Table 6. Accuracy versus cost under fixed inference sandbox budgets on Qwen3-8B: average pass rate over HE+/MBPP+/LCB/BCB-H/APPS/CC given the number of executions in the column header. This budget is distinct from the training rollout group size K. Method Prompted reviewers Self-Refine Trained critic rerank Best of N execution RLVR without critic ExecuCritic
2
4
8
16
24
39.8 40.1 41.5 40.9 43.2 45.8
40.7 41.0 42.2 42.6 44.1 47.2
41.2 41.6 42.5 44.3 44.9 48.3
41.6 42.0 43.0 45.0 44.9 48.5
41.8 42.2 43.2 45.4 45.0 48.7
Table 7. Execution budget and backbone scaling. Pass@1 is averaged over HE+/MBPP+/LCB/BCB-H/APPS/CC. Exec. denotes average sandbox calls per prompt at inference. Backbone / method
Params
Qwen3-4B + RLVR without critic 4B Qwen3-4B + ExecuCritic 4B Qwen3-8B + RLVR without critic 8B Qwen3-8B + ExecuCritic 8B DS-Coder-V2-Lite + RLVR without critic 16B MoE DS-Coder-V2-Lite + ExecuCritic 16B MoE
Exec. Avg. pass@1 SWE-Lite 16.0 7.2 16.0 7.5 16.0 7.9
39.7 43.1 44.9 48.3 45.9 49.2
10.8 13.9 14.6 18.3 15.4 19.1
calls as executing all N candidates while improving pass@1 and repository resolve rate. The relative gain is slightly larger for Qwen3-4B, where the coder makes more detectable local mistakes. 4.8
Case study by bug type
To examine which errors the critic helps fix, we label each error on a stratified 250-problem subset of LiveCodeBench and APPS, using the taxonomy of [14,16] and our executor descriptors. Table 8 reports the remaining error rate by failure type. RLVR with a single coder removes many easy bugs but struggles with deep logic mistakes and rarer runtime failures. ExecuCritic improves all seven categories, with the largest relative gains on import/API (−34%), runtime exception (−28%) and type/numerical (−28%) errors, which the critic also flags most reliably in its textual outputs. 4.9
Discussion and limitations
ExecuCritic inherits two limitations of RLVR. First, it needs an executable test oracle per training prompt; refactoring or documentation would require synthetic verifiers or a learned test generator. Second, although ρK in Eq. (2) limits
ExecuCritic: Calibrated Critic Shaping
13
Table 8. Failure types on a stratified 250 problem subset of LiveCodeBench and APPS. Cells give the fraction of problems failing with that bug type (%, lower is better); rows do not sum to 100, since a solved problem is assigned no failure type. Failure type
B2
B4
B5
B3
Ours
Boundary / index error Import or wrong API Type / numerical precision Assertion mismatch Runtime exception Timeout / complexity Deep logic / algorithmic
18.4 14.1 9.8 20.6 8.5 7.2 22.7
16.9 11.5 9.2 19.4 7.9 6.9 21.0
15.6 14.0 10.8 9.7 8.7 7.6 18.9 16.8 7.5 6.7 6.6 6.1 20.6 18.3
10.2 6.4 5.5 13.9 4.8 4.7 15.1
badly calibrated critic rewards, weak or out-of-distribution backbones may need a longer SFT phase. We saw no reward hacking, plausibly because the critic is anchored to executor verdicts, but adversarial initial critics deserve study, since agents can be steered by inputs crafted to look benign [36,29].
5
Conclusion
We presented ExecuCritic, a joint reinforcement learning framework in which a coder and an execution-grounded critic share a backbone and are updated together under verifiable rewards. A calibrated advantage makes sparse RLVR feedback denser, while critic ranking and failure diagnosis stabilise training and support efficient inference. Across eight benchmark views and two open backbones it beats strong RLVR, prompted reviewer, self-refinement and scalar reward model baselines using fewer policy-gradient steps and sandbox executions, and our analyses indicate that learning a calibrated critic alongside the coder is a useful inductive bias. Future work includes tasks without test oracles, sparse expert backbones, process reward models, progressively trained smaller backbones [25], domains with weaker verifiers [20,18,52], and keeping the critic calibrated as the coder drifts [5,21].
References 1. Bai, Y., Kadavath, S., et al.: Constitutional ai: Harmlessness from ai feedback. arXiv:2212.08073 (2022) 2. Chen, B., Zhang, F., et al.: Codet: Code generation with generated tests. arXiv:2207.10397 (2022) 3. Cobbe, K., Kosaraju, V., et al.: Training verifiers to solve math word problems. arXiv:2110.14168 (2021) 4. Dong, H., Jiang, K., et al.: Neureasoner: Towards explainable, controllable, and unified reasoning via mixture-of-neurons. In: Proc. ACL (2026) 5. Feng, Y., Wang, H., et al.: Forever: Forgetting curve-inspired memory replay for language model continual learning. In: Proc. ACL (2026)
14
J. Cao et al.
6. Grattafiori, A., Dubey, A., et al.: The llama 3 herd of models. arXiv:2407.21783 (2024) 7. Guo, D., Yang, D., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948 (2025) 8. Hendrycks, D., Basart, S., et al.: Measuring coding challenge competence with apps. arXiv:2105.09938 (2021) 9. Hong, M., Lin, Z., et al.: Conflict-aware client selection for multi-server federated learning. In: Proc. ICASSP (2026) 10. Hong, S., Zhuge, M., et al.: Metagpt: Meta programming for a multi-agent collaborative framework. In: Proc. ICLR (2024) 11. Hu, E.J., Shen, Y., et al.: Lora: Low-rank adaptation of large language models. Proc. ICLR (2022) 12. Huang, D., Zhang, J.M., et al.: Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv:2312.13010 (2023) 13. Islam, M.A., Ali, M.E., et al.: Mapcoder: Multi-agent code generation for competitive problem solving. In: Proc. ACL (2024) 14. Jain, N., Gu, A., et al.: Livecodebench: Holistic and contamination free evaluation of large language models for code. In: Proc. ICLR (2025) 15. Jiang, E., Li, L., et al.: Agent q-mix: Selecting the right action for llm multi-agent systems through reinforcement learning. In: Proc. COLM (2026) 16. Jiang, K., Dong, H., et al.: Foe: Forest of errors makes the first solution the best in large reasoning models. In: Proc. ACL (2026) 17. Jimenez, C.E., Yang, J., et al.: Swe-bench: Can language models resolve real-world github issues? In: Proc. ICLR (2024) 18. Kang, Z., Cai, H., et al.: Jurisctc: Enhancing legal judgment prediction via crossdomain transfer and contrastive learning. In: Proc. IJCNN (2025) 19. Kang, Z., Gong, J., et al.: Multimodal multi-agent empowered legal judgment prediction. In: Proc. ICASSP (2026) 20. Kang, Z., Gong, J., et al.: Quanteval: A benchmark for financial quantitative tasks in large language models. arXiv:2601.08689 (2026) 21. Kang, Z., He, Y., et al.: How order-sensitive are llms? orderprobe for deterministic structural reconstruction. In: Findings of EMNLP (2026) 22. Kang, Z., Liu, Y., et al.: Jpo: Juris policy optimization for structured legal reasoning in criminal judgment prediction. In: Proc. EMNLP (2026) 23. Li, Y., Choi, D., et al.: Competition-level code generation with alphacode. Science 378(6624), 1092–1097 (2022) 24. Lightman, H., Kosaraju, V., et al.: Let’s verify step by step. In: Proc. ICLR (2024) 25. Liu, J., Kang, Z.: Reasonact: Progressive training for fine-grained video reasoning in small models. In: Proc. AAAI (2026) 26. Liu, J., Xia, C.S., et al.: Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. NeurIPS 36, 21558–21572 (2023) 27. Liu, J., Zhang, L.: Code-r1: Reproducing r1 for code with reliable rewards. arXiv:2503.18470 (2025) 28. Liu, Y., Zhang, L.L., et al.: rstar-coder: Scaling competitive code reasoning with a large-scale verified dataset. NeurIPS 38, 58780–58807 (2026) 29. Lou, X., Xu, J., et al.: When helpers become hazards: A benchmark for analyzing multimodal llm-powered safety in daily life. In: Findings of ACL (2026) 30. Luo, Z., Xu, C., et al.: Wizardcoder: Empowering code large language models with evol-instruct. In: Proc. ICLR (2024)
ExecuCritic: Calibrated Critic Shaping
15
31. Madaan, A., Tandon, N., et al.: Self-refine: Iterative refinement with self-feedback. NeurIPS 36, 46534–46594 (2023) 32. McAleese, N., Pokorny, R.M., et al.: Llm critics help catch llm bugs. arXiv:2407.00215 (2024) 33. Meng, C., Feng, P., et al.: Group cognition learning: Making everything better through governed two-stage agents collaboration. In: Proc. ICML (2026) 34. Ng, A.Y., Harada, D., et al.: Policy invariance under reward transformations: Theory and application to reward shaping. In: Proc. ICML (1999) 35. Pan, J., Wang, X., et al.: Training software engineering agents and verifiers with swe-gym. arXiv:2412.21139 (2024) 36. Qian, J., Kang, Z.: Penny wise, pixel foolish: Bypassing price constraints in multimodal agents via visual adversarial perturbations. In: Findings of ACL (2026) 37. Schulman, J., Wolski, F., et al.: Proximal policy optimization algorithms. arXiv:1707.06347 (2017) 38. Shao, Z., Wang, P., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300 (2024) 39. Shi, Q., Kang, Z., et al.: Spader: Step-wise peer advantage with diversity-aware exploration rewards for multi-answer question answering. In: Proc. EMNLP (2026) 40. Shinn, N., Cassano, F., et al.: Reflexion: Language agents with verbal reinforcement learning. NeurIPS 36, 8634–8652 (2023) 41. Wang, P., Li, L., et al.: Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In: Proc. ACL (2024) 42. Wang, Y., Gao, S., et al.: Beyond n-grams: A hierarchical reward learning framework for clinically-aware medical report generation. In: Proc. AAAI (2026) 43. Wei, Y., Duchenne, O., et al.: Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. NeurIPS 38, 78500–78525 (2026) 44. Wei, Y., Wang, Z., et al.: Magicoder: Empowering code generation with ossinstruct. arXiv:2312.02120 (2023) 45. Wu, Q., Bansal, G., et al.: Autogen: Enabling next-gen llm applications via multiagent conversations. In: Proc. COLM (2024) 46. Yang, A., Li, A., et al.: Qwen3 technical report. arXiv:2505.09388 (2025) 47. Yuan, W., Pang, R.Y., et al.: Self-rewarding language models. arXiv:2401.10020 (2024) 48. Zeng, H., Jiang, D., et al.: Acecoder: Acing coder rl via automated test-case synthesis. In: Proc. ACL (2025) 49. Zhao, P., Liu, G., et al.: Mas-bench: A unified benchmark for shortcut-augmented hybrid mobile gui agents. In: Proc. ACL (2026) 50. Zheng, L., Su, J., et al.: Gradients know what outcomes don’t: Unlocking reinforcement learning for llm reasoning with gradient-aligned rewards. In: Proc. EMNLP (2026) 51. Zheng, L., Su, J., et al.: Scilens: Rl-driven autonomous agents for scientific localized evidence navigation and synthesis. In: Proc. EMNLP (2026) 52. Zheng, L., Zhang, J., et al.: What should i cite? a rag benchmark for academic citation prediction. In: Proc. WWW (2026) 53. Zhu, C., Wu, S., et al.: Edis: Diagnosing llm reasoning via entropy dynamics. arXiv:2602.01288 (2026) 54. Zhu, Q., Guo, D., et al.: Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv:2406.11931 (2024) 55. Zhuo, T.Y., Vu, M.C., et al.: Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In: Proc. ICLR (2025)