arXiv CS open access

TIPCODER: Reinforcement Learning Boosted Test-time Instruction Proposer for Code Generation

2026 · arxiv_cs

arXiv CS · Papers · 2026

software-architecture software-engineering testing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

T IP C ODER: Reinforcement Learning Boosted Test-time Instruction Proposer for Code Generation Minyu Chen1 * , Sihao Wu2 , Ling-I Wu3 , Song Qin1,4 , Jingyang Li3 , Lei Ning1 , Jianxin Xue4 , Guoqiang Li3 * 1 Shenzhen Technology University, 2 Institute of AI for Industries, Chinese Academy of Sciences, 3 Shanghai Jiao Tong University, 4 Shanghai Polytechnic University chenminyu@sztu.edu.cn, li.g@sjtu.edu.cn Abstract

arXiv:2609.03309v1 [cs.SE] 3 Sep 2026

Test-time scaling for code generation typically explores the solution space by sampling multiple programs from a fixed instruction. We study a complementary direction: instancelevel instruction-space exploration. Our observation is that many coding failures stem from missing constraints, overlooked edge cases, or misleading reasoning paths induced by the original prompt. To address this, we propose T IP C ODER, a test-time instruction proposer that generates problem-specific auxiliary tips before code synthesis. T IP C ODER distills multi-turn debugging trajectories into proactive guidance and further optimizes the Proposer with reinforcement learning using a marginalutility reward. At inference time, it generates both a base solution and a tip-guided solution, and applies a Reward Model for post-hoc selection. This exploration-selection design allows tips to expose additional candidate potential while reducing regressions from unnecessary guidance. Across the evaluated codegeneration benchmarks and target Code LLMs, T IP C ODER provides a consistent instructionlevel test-time scaling strategy, comparing favorably with stochastic sampling and generic prompt optimization baselines under a shared reward-model-based selection protocol. †

Introduction

Code Large Language Models (Guo et al., 2024; Hui et al., 2024) have become increasingly effective at translating natural-language specifications into executable programs. However, their outputs remain highly sensitive to the initial problem instruction, which may omit implicit constraints, under-specify boundary cases, or induce a spurious reasoning trajectory. Existing test-time scaling methods, such as Best-of-N sampling (Brown et al., *

Corresponding authors. Our dataset and code are available at https://github. com/Minkow/TipCoder †

2024), improve reliability by allocating additional computation to explore the solution space (Snell et al., 2025; Muennighoff et al., 2025). Since these methods typically sample from a fixed conditioning context, they may remain concentrated around the same flawed interpretation when the original instruction is misleading. This motivates a complementary scaling axis: adapting the instruction that guides solution search. Prompt optimization demonstrates that instructions can be optimized to improve model behavior (Zhou et al., 2023; Yang et al., 2024; Agrawal et al., 2026; Hu et al., 2025). For code generation, however, the limiting factor is often not the absence of a globally better prompt, but the absence of instance-specific guidance. A failure may hinge on a particular boundary case, index convention, library behavior, numerical constraint, or ambiguity in the specification. Such localized failure modes are poorly captured by fixed dataset-level prompts or generic reminders such as “consider edge cases”. We therefore study instance-level instruction proposing: generating a problem-specific auxiliary instruction before code synthesis. We introduce T IP C ODER, a test-time instruction proposer for code generation. Given a programming problem, a Proposer generates an auxiliary tip that highlights relevant constraints, likely pitfalls, or implementation considerations. The target Code LLM then produces two candidates: a base solution conditioned on the original instruction and a tip-guided solution conditioned on the augmented instruction. The tip is not intended to specify the solution; instead, it modifies the conditioning context so that the generator explores an alternative region of the solution space. To learn such tips, T IP C ODER distills multi-turn debugging trajectories into proactive guidance. In these trajectories, an initial solution fails, feedback exposes a missing requirement or incorrect assumption, and a subsequent solution repairs the

error. This process reveals information that would have been useful before the initial generation. We first initialize the Proposer with Supervised FineTuning on distilled repair guidance, and then optimize it with RL in a frozen Code LLM environment. The reward measures the marginal utility of the generated tip relative to base generation, encouraging corrective instructions while penalizing harmful or redundant perturbations. At inference time, the utility of a tip is instancedependent. A tip-guided candidate may recover a solution that is unlikely under the original prompt, but can also perturb an already correct base trajectory. Thus, the base and tip-guided candidates define an oracle-selection upper bound for instructionguided scaling, while practical deployment requires instance-wise selection. T IP C ODER uses a Reward Model (RM) for this post-hoc selection. The RM does not create instruction-level exploration, but approximates the selector needed to realize the candidate potential exposed by the Proposer. T IP C ODER operates as an external prompt-level module and requires no access to the target Code LLM’s parameters, gradients, or execution feedback during inference. Across multiple codegeneration benchmarks and backbones, T IP C ODER consistently improves pass rates over base generation and stochastic sampling. Under the same RM reranking protocol, it also outperforms selfhinting and generic context-optimization baselines, indicating that the gains arise from more effective instance-level instruction proposals rather than from the mere availability of an additional candidate selector. Our contributions are summarized as follows: • Instance-level instruction-space exploration. We formulate test-time instruction proposing as a complementary scaling axis for code generation, where problem-specific auxiliary instructions guide solution search beyond fixed-prompt stochastic sampling and dataset-level prompt optimization.

instance-wise selection between base and tipguided candidates, and evaluate T IP C ODER against stochastic sampling, self-hinting, and context-optimization baselines under the same reranking protocol.

Methodology

2.1

Problem Formulation

Let D = {(x, T )} denote a dataset where x is the problem description and T is the unit test set. A Code LLM M generates a solution y according to P (y|x) = M(y|x). While standard one-off generation often fails due to ambiguous constraints, empirical evidence suggests that a correct solution y ∗ remains reachable (Austin et al., 2021; Larbi et al., 2025). This can be achieved through iterative debugging with execution feedback, where error messages are utilized to refine the code over multiple turns (Chen et al., 2024; Shinn et al., 2023). Our goal is to retain useful signals from this multi-turn process without requiring a runtime debugging loop. We posit that the sparse reasoning signals within the interaction history can be compressed into a concise latent instruction c (a “tip”). Consequently, we formulate the problem as learning a proposer policy πϕ (c|x) that maximizes the expected pass rate of the downstream generation: h max J (ϕ) = Ec∼πϕ (·|x) Ey∼M(·|x,c) ϕ i [⊮(Pass(y, T ))] . Intuitively, this objective drives the proposer to discover optimal shortcuts in the instruction space that functionally replicate the utility of the expensive debugging history. 2.2

Tip Supervision from Debugging Trajectories

• Debugging-distilled and utility-optimized Proposer. We distill multi-turn debugging trajectories into proactive tips that capture latent constraints, edge cases, and failure modes. We then optimize the Proposer with RL using a marginal-utility reward.

Training the Proposer requires high-quality pairs (x, c) where the auxiliary instruction c captures information that is missing from the original problem instruction but useful for code generation. We construct such supervision by distilling successful multi-turn debugging trajectories into single-turn proactive tips. The construction pipeline consists of three stages: mining repair trajectories, extracting debugging observations through masked distillation, and verifying the causal utility of the tips.

• Exploration-selection decomposition. We use RM-based reranking to approximate

Mining Repair Trajectories. We use AceCoder87K (Zeng et al., 2025), a large-scale code instruc-

Summary

SFT Tips

Masked Distillation

Tips

Code LLM (With Tips)

Codetips

Debug History

GRPO Reward

Problem Instruction

Proposer Model

Reward Model

Best Code

Code Environment Code Generate

Training Phase

Tips Generate Tips Rollout

Code LLM

Codebase

Inference Phase

Figure 1: Overview of T IP C ODER. The training phase (left) consists of two stages: initializing the Proposer Agent via SFT, followed by Reinforcement Learning using pass rates from the code environment. The inference phase (right) employs the Proposer Model to generate guiding tips, and a Reward Model is used to select the best solution between the base and guided codes.

tion dataset constructed through evolutionary instruction generation from open-source repositories, as the seed corpus. To obtain dense reasoning signals, we focus on challenging samples where the student model exhibits low one-pass performance, filtering for problems with Pass@1 < 50%. For each retained problem, we employ Gemini-3-Pro as the teacher model to perform iterative debugging with execution feedback. For a problem x with unit tests T , we denote a debugging trajectory as τ = {(y 0 , e0 ), (y 1 , e1 ), . . . , (y K , eK )},

(1)

where y t is the solution generated at turn t and et is the corresponding execution feedback. We retain trajectories where the initial solution fails but is successfully repaired within three turns: Pass(y 0 , T ) = 0,

Pass(y K , T ) = 1.

(2)

Such trajectories capture the transition from an incorrect initial interpretation to a correct solution, and therefore expose the information that the original instruction failed to elicit. Abductive Reasoning via Masked Distillation. A successful repair trajectory contains a debugging observation: the hidden constraint, edge case, or invalid assumption that explains why the initial solution failed and how the later solution repaired it. The goal of distillation is to convert this retrospective observation into a proactive instruction

that can be provided before code generation. In this sense, the tip serves as a compact substitute for the foresight normally obtained only after costly iterative debugging. To prevent solution leakage, we apply masked distillation. Implementation-level code blocks and solution-specific details are masked before the teacher extracts the observation. Formally, the teacher first infers the debugging observation from the problem and the masked trajectory: oτ = Obsteacher (x, Mask(τ )).

(3)

The observation is then verbalized as an auxiliary tip: c ∼ qteacher (· | x, oτ ). (4) The generated tip is required to describe high-level constraints, edge cases, or likely failure modes rather than prescribe the final implementation. This process distills the retrospective knowledge revealed by multi-turn repair into a single-turn guidance signal available at the initial generation stage. Iterative Feedback Verification. The distilled tip is retained only if it is causally useful for the student model. We append the generated tip c to the original instruction x and run the student model again. If the generated code still fails, the execution feedback is returned to the teacher model to refine the tip, with at most two refinement rounds. A pair (x, c) is added to the final training set if and only if

the verified tip guides the student model to pass all unit tests: Pass(ytip , T ) = 1. (5) This filtering process yields 2,126 high-quality problem-tip pairs for supervised initialization. 2.3

non-differentiable reward based on execution results. The optimization objective J (ϕ) maximizes the expected execution-based reward with KL regularization: J (ϕ) =Ec∼πϕ (·|x) [R(c)]

Proposer Optimization

We optimize the Proposer in two stages. Supervised Fine-Tuning initializes the policy with verified debugging-derived tips, and Reinforcement Learning further aligns the policy with downstream code correctness. Supervised Fine-Tuning (SFT). We initialize the Proposer policy πϕ using the distilled dataset Dtrain . This stage teaches the Proposer the form and patterns of useful tips. However, it does not directly optimize downstream correctness, motivating the subsequent RL stage. Latent-Variable Interpretation. The tip c mediates the relationship between the problem x and a correct solution y ∗ . SFT supplies a reference distribution over natural-language tips, and RL searches for tips with higher downstream utility. Taking the frozen SFT πref (c|x) as a prior P model ∗ ∗ gives P (y |x) = c P (y |x, c)πref (c|x). Using the Proposer policy πϕ (c|x) as a variational distribution yields the following Evidence Lower Bound: log P (y ∗ |x) = log

πϕ (c|x)

P (y ∗ |x, c)πref (c|x) πϕ (c|x)

πϕ (c|x) log

P (y ∗ |x, c)πref (c|x) πϕ (c|x)

X c

≥

X c

= Ec∼πϕ [log P (y ∗ |x, c)] − DKL (πϕ (·|x)||πref (·|x)) . {z } {z } | | Reconstruction Objective

Regularization Constraint

Here P (y ∗ |x, c) denotes the likelihood of the fixed Code LLM generating a correct solution given the tip. The implemented objective uses execution pass rates, the piecewise marginal-utility reward below, and a length penalty as operational approximations to the reconstruction term. Reinforcement Learning Objective. Since P (y ∗ |x, c) is not directly accessible in code generation, we replace it with an execution-based reward. We define the instruction proposer πϕ (c|x) as the policy and the combination of the frozen Code LLM and the deterministic execution sandbox as the environment. The environment receives a tip c, generates code, and returns a

− βDKL (πϕ (·|x)||πref (·|x)). We optimize this objective with Group Relative Policy Optimization (GRPO) (Shao et al., 2024). For each input x, we sample a group of outputs {c1 , . . . , cG } and estimate the baseline using the group average. Marginal-Utility Reward. A naive reward based only on the pass rate of the tip-guided solution is noisy: on simple tasks where the base model already succeeds, the policy can receive high reward even for redundant or empty tips. We therefore define the reward by the marginal utility of the tip relative to base generation. Let Pbase denote the pass rate of the code generated with the original problem description and Pcurr denote the pass rate with the generated tip c. We define the Utility Gain ∆ to explicitly measure the improvement brought by the tip: ( 1.0 if Pbase < 1 and Pcurr = 1 ∆= Pcurr − Pbase otherwise This reward approximates the marginal contribution of the proposed tip before RM-based selection. A positive value indicates that the tip improves the tip-guided branch relative to the base branch, while a negative value indicates regression. Subtracting the base performance filters out redundant tips on already solved tasks and makes the reward depend on the relative contribution of the instruction rather than absolute task difficulty. This centering also reduces reward variance, encouraging policy updates to focus on whether the tip changes the outcome. Finally, to further encourage conciseness, we incorporate a length penalty using a ReLU style threshold. The total reward R(c) is defined as R(c) = ∆ − λ · max(0, L(c) − Lth ), where L(c) is the token length of the tip and Lth = 2000 is the soft threshold. 2.4

Reward-Guided Reranking Inference

At inference time, T IP C ODER follows a generateand-rerank protocol. The Proposer generates a

problem-specific tip c for every input problem x. We do not require the Proposer to decide whether the base model would fail on the original prompt, since problem difficulty and failure modes are model-dependent. Instead, T IP C ODER constructs two branches: the base branch from the original instruction and the tip-guided branch from the augmented instruction. • Base branch: the target Code LLM generates ybase from the original instruction x. • Tip-guided branch: the same target Code LLM generates ytip from the augmented instruction x ⊕ c. T IP C ODER then applies a Reward Model R as a post-hoc selector: ( ytip , R(x, ytip ) > R(x, ybase ), ŷ = (6) ybase , otherwise. The deployed objective is therefore to improve the pass rate of the selected output, max Ec∼πϕ (·|x) Eybase ,ytip ⊮ Pass(ŷ, T ) . (7) ϕ

This protocol completes the explorationselection decomposition of T IP C ODER: the Proposer exposes an alternative instructionconditioned candidate, and the Reward Model selects between the base and tip-guided solutions for each instance. This post-hoc selection reduces regressions from unnecessary or noisy tips without requiring the Proposer to predict whether the base model will fail. T IP C ODER is black-box with respect to the target Code LLM. It operates only through promptlevel augmentation and candidate scoring, requiring no access to model parameters, gradients, or execution feedback during inference. Because it explores the instruction dimension rather than replacing solution-level sampling, it remains compatible with test-time scaling methods such as Best-of-N .

Experiments

3.1

Setup

Datasets and Metrics. We evaluate on three code-generation benchmarks: HumanEval+ (Chen et al., 2021), MBPP+ (Austin et al., 2021), and BigCodeBench-Instruct (Zhuo et al., 2025). For HumanEval+ and MBPP+, we use the EvalPlus

implementation (Liu et al., 2023), which provides more rigorous test suites than the original benchmarks. HumanEval and MBPP primarily cover algorithmic and entry-level Python programming tasks, while BigCodeBench-Instruct evaluates more realistic programming scenarios involving library usage and complex function calls. Since T IP C ODER targets instruction refinement, we evaluate on the instruction-following split of BigCodeBench rather than the completion split. We use Pass@1 as the primary metric. Evaluation tests are used only for reporting final correctness and are not available to T IP C ODER during inference. Proposer Training. The Proposer is initialized from Qwen3-4B-Instruct-2507. We use a general instruction model rather than a code-specialized model because tip generation requires identifying constraints, edge cases, and failure modes in natural language. During RL, the Proposer interacts with a frozen code-generation environment instantiated by Qwen2.5-7B-Coder-Instruct. This environment is used only for training-time rollouts and reward computation; it is not the sole target model used in evaluation. For RL training, we use the hard subset of AceCoder-87K and exclude all samples used in the SFT stage. We further downsample extreme cases where the base pass rate is either 0 or 1, so that the policy receives informative marginal-utility signals rather than learning from tasks that are already solved or consistently unsolved by the rollout environment. This yields 13,376 RL training samples. The evaluation benchmarks are held out from Proposer training. We implement training with OpenRLHF and vLLM. The rollout group size is set to 8. The environment Code LLM uses temperature 0 to provide deterministic execution feedback during RL, while the Proposer samples tips with temperature 1.0 during training rollouts and uses greedy decoding during evaluation. Full hyperparameters are provided in Appendix C. Target Code LLMs. We evaluate T IP C ODER on four open-source Code LLM backbones: Qwen2.5-Coder-7B-Instruct, DeepSeek-Coder-7B-Instruct-v1.5, Qwen3-Coder-30B-A3B-Instruct, and DeepSeek-Coder-V2-Lite-Instruct. Unless otherwise specified, the same trained Proposer is applied to all target backbones without targetspecific fine-tuning. This setting tests whether

the learned instruction proposer transfers across different Code LLMs. Baselines. We compare against the following baselines. Base directly generates code from the original instruction. Best-of-N samples multiple candidate solutions from the original instruction. In practice, we set N = 2. Self-Hint is a zeroshot instruction-proposing baseline. It uses the same tip-generation prompt as T IP C ODER, but replaces the trained Proposer with an untrained instruction model to generate the auxiliary tip; the target Code LLM then generates a tip-guided candidate from the resulting augmented instruction. To compare with generic prompt or context optimization, we include OPRO (Yang et al., 2024), which treats prompt design as an optimization problem and iteratively proposes improved prompts; GEPA (Agrawal et al., 2026), which evolves prompts through reflective feedback; and CTRL-32B (Xie et al., 2025), a critique-based model that produces contextual guidance for code revision. Reward Model. For all baselines and T IP C ODER, the candidate set consists of the base solution and the corresponding augmented solution. We use the same AceCodeRM-32B (Zeng et al., 2025) to select the best candidate, without access to execution feedback or hidden tests at inference time. This keeps the selector fixed across methods, so performance differences reflect the generated candidate set rather than the reward model. 3.2

Main Results

Table 1 reports the main results across four target Code LLMs. All methods requiring candidate selection use the same AceCodeRM-32B Reward Model. Prompt- and context-level optimization generally improves over base generation, but the gains vary across backbones and benchmarks. OPRO and GEPA are competitive on Qwen2.5-Coder-7B and Qwen3-Coder-30B-A3B, but are less stable on DeepSeek-Coder-7B. Bestof-2 improves HumanEval+ and MBPP+ in many cases, yet consistently degrades BigCodeBench, suggesting that merely sampling from the original instruction is insufficient for more realistic programming tasks. Table 2 compares a range of Bestof-N budgets, and Appendix A extends the MBPP+ scaling curves to all four target backbones. T IP C ODER-RL achieves the best average performance on all four backbones, reaching 61.17 on DeepSeek-Coder-7B,

65.27 on DeepSeek-Coder-V2-Lite, 68.64 on Qwen2.5-Coder-7B, and 71.10 on Qwen3-Coder-30B-A3B. Compared with the strongest non-T IP C ODER baseline on each backbone, this yields average improvements of 1.81, 0.92, 0.99, and 0.47 points, respectively. These results indicate that instance-level instruction proposing provides a more stable scaling direction than generic prompt optimization. Self-Hint uses the same tip-generation prompt without training the Proposer, serving as a zeroshot instruction-refinement baseline. The trained T IP C ODER variants consistently improve over SelfHint in average performance, showing that the gains come not merely from adding auxiliary instructions, but from learning problem-specific tips aligned with downstream code correctness. 3.3

Disentangling Exploration from Selection

Figure 2 decomposes test-time performance into two factors: candidate exploration and post-hoc selection. For each method, the figure explicitly reports the base performance, raw adoption of the additional branch, reward-model-selected performance under different Reward Models, and the oracle upper/lower bounds over the candidate set. The oracle upper bound measures the best achievable performance if an ideal selector could choose between the base and additional candidates, while the lower bound measures the worst possible selection. Thus, the gap between base performance and the oracle upper bound reflects the exploration potential introduced by the method, whereas the gap between reward-model selection and the oracle upper bound reflects how much of this potential remains unrealized by the selector. The results show that instruction-guided generation can expose useful candidates beyond the original prompt. In many settings, T IP C ODERRL raises the oracle upper bound, indicating that learned tips create complementary solution paths rather than merely perturbing the prompt. However, this potential is not automatically realized: raw adoption can select harmful candidates, and weaker selectors may fail to recover the improved branch. Stronger code-oriented Reward Models, such as AceCodeRM-32B, generally convert more of the oracle potential into final pass-rate gains than weaker or less aligned selectors. Therefore, the Reward Model is not the source of the exploration benefit; it determines how much of the candidatespace improvement opened by the Proposer can be

Table 1: Main results on code-generation benchmarks across diverse backbones. Results are reported as mean±std over three independent runs. All methods that require candidate selection use the same AceCodeRM-32B Reward Model. Avg. denotes the unweighted average of the three benchmark means. Model

Method Base Best-of-2 OPRO GEPA CTRL Self-Hint T IP C ODER-SFT T IP C ODER-RL

HumanEval+ 62.40±1.86 69.92±1.86 65.85±0.00 65.85±0.00 67.89±0.93 69.92±1.86 71.34±0.00 72.36±1.76

MBPP+ 61.55±2.12 66.93±2.38 68.25±0.00 68.25±0.00 68.96±0.15 69.40±0.31 71.08±0.81 71.16±1.15

BigCodeBench 37.43±0.05 32.54±0.00 39.80±0.05 39.94±0.05 38.39±0.48 38.77±0.61 39.44±0.05 40.00±0.05

Avg. 53.79 56.46 57.97 58.01 58.41 59.36 60.62 61.17

DeepSeek-Coder-V2-Lite

Base Best-of-2 OPRO GEPA CTRL Self-Hint T IP C ODER-SFT T IP C ODER-RL

74.39±1.61 78.25±0.93 76.22±1.61 77.03±0.35 78.66±0.00 80.49±1.61 78.66±0.61 80.89±1.04

68.78±1.40 71.34±1.00 72.13±0.61 72.49±0.00 72.93±0.40 71.60±0.15 72.13±1.10 73.46±0.31

38.19±0.33 34.21±0.00 40.88±0.09 40.61±0.69 39.04±0.55 40.96±1.14 39.82±0.23 41.46±0.43

60.45 61.27 63.08 63.38 63.54 64.35 63.54 65.27

Qwen2.5-Coder-7B

Base Best-of-2 OPRO GEPA CTRL Self-Hint T IP C ODER-SFT T IP C ODER-RL

78.86±1.27 85.77±1.27 85.57±0.35 85.57±0.35 85.57±0.35 86.38±0.35 84.96±0.35 88.21±0.35

64.46±1.00 70.37±0.92 73.81±0.00 73.81±0.00 73.54±0.46 73.54±0.46 73.19±1.07 73.81±0.92

41.64±0.05 36.32±0.00 43.57±0.05 42.87±0.05 42.08±0.27 41.49±0.00 42.34±0.05 43.89±0.56

61.65 64.15 67.65 67.42 67.06 67.14 66.83 68.64

Qwen3-Coder-30B-A3B

Base Best-of-2 OPRO GEPA CTRL Self-Hint T IP C ODER-SFT T IP C ODER-RL

86.99±0.93 87.60±0.70 89.02±0.35 89.02±0.35 87.80±0.00 88.62±1.27 89.02±0.70 89.23±0.93

73.72±0.40 74.60±1.15 74.07±0.46 74.34±0.00 73.46±0.31 73.90±0.40 74.96±0.15 75.31±0.33

48.39±0.05 43.16±0.00 48.71±0.20 48.54±0.18 48.22±0.28 46.29±0.37 47.98±0.30 48.77±0.09

69.70 68.45 70.60 70.63 69.83 69.60 70.65 71.10

DeepSeek-Coder-7B

realized at inference time. 3.4

Cost-Normalized Best-of-N Scaling

We compare Best-of-N under a controlled testtime budget, holding the target Code LLM, original prompt, and AceCodeRM-32B selector fixed while varying N ∈ {1, 2, 4, 8, 16}. Table 2 reports Pass@1, token counts, and wall-clock latency for Qwen2.5-Coder-7B on all three benchmarks. On HumanEval+, T IP C ODER-RL reaches 88.21 Pass@1 with 2,812 tokens and 0.99 seconds, compared with 84.15, 4,610 tokens, and 1.94 seconds for Best-of-8. On MBPP+, it reaches 73.81 with 1,210 tokens, while Best-of-8 reaches 73.54 with 2,468 tokens; Best-of-16 improves to 74.60 using 4,814 tokens. On BigCodeBench-full, T IP C ODERRL reaches 43.89 with 2,546 tokens, exceeding Best-of-4 at 42.54 with 3,017 tokens. Best-of-8

reaches 44.56 with 5,828 tokens. Overall, T IP C ODER-RL exceeds Best-of-8 on HumanEval+ and MBPP+ with fewer tokens and falls between Bestof-4 and Best-of-8 on BigCodeBench-full. 3.5

Systematic Analysis of Generated Tips

We analyze the tips selected by AceCodeRM-32B across three seeds, four target Code LLMs, and HumanEval+/MBPP+, totaling 6,504 evaluated instances. The RM selects the guided branch 2,187 times: 310 selections repair a base failure, 117 corrupt a correct base solution, and 1,760 leave execution correctness unchanged. The selected tips therefore yield a net gain of 193 corrected instances, equivalent to 2.97 micro-averaged Pass@1 points. Semantic/constraint clarification is the strongest source of repairs, accounting for 43.2% of all positive changes and a net gain of 109 instances.

Lower-Upper

HumanEval+ Pass@1 (%)

Qwen2.5-Coder-7B

90.0

87.5

85.0

75 PRO

A GEP

L CTR

er Cod

80 75 70

O OPR

A GEP

L CTR

oder

Bo2

TipC

55 50

CTR

der

o TipC

Bo2

O OPR

A GEP

CTR

der

A GEP

CTR

der

o TipC

O OPR

A GEP

CTR

der

o TipC

OPR

A GEP

CTR

o TipC

Bo2

OPR

A GEP

CTR

o TipC

Bo2

OPR

A GEP

CTR

o TipC

65 60 55 O

OPR

A GEP

CTR

der

o TipC

der

35 30

20 Bo2

der

Bo2

40 OPR

TipC

Bo2

oder

L CTR

Bo2

o TipC

A GEP

PRO

A GEP

65 O

OPR

Bo2

DeepSeek-Coder-V2-Lite

65 55

InternLM RM

Bo2

Tip

AceCodeRM-32B

DeepSeek-Coder-7B

80.0 Bo2

MBPP+ Pass@1 (%)

AceCodeRM-7B

82.5

BigCodeBench Pass@1 (%)

Raw

Qwen3-Coder-30B-A3B

92.5

Base

Bo2

OPR

A GEP

CTR

der

o TipC

der

Figure 2: Disentangling candidate exploration from post-hoc selection. For each target model and benchmark, we fix the candidate set produced by each method and vary only the selection rule. The figure explicitly reports base performance, raw adoption of the additional branch, reward-model-selected performance under different Reward Models, and oracle upper/lower bounds over the candidate set. The shaded region denotes the oracle lower-to-upper range, where the upper bound measures the best achievable performance with an ideal selector and the lower bound measures the risk of selecting the worse candidate. A higher oracle upper bound indicates that a method exposes additional correct candidates, while the gap between RM-selected performance and the oracle upper bound reflects the remaining selection bottleneck.

Boundary cases, index/order/output alignment, and algorithmic guidance are also net-positive. Type/parsing/API guidance is balanced, whereas numerical/formula/precision guidance is net-negative. For example, the tip for HumanEval/26 clarifies duplicate-removal semantics and repairs a failure, while the tip for HumanEval/7 introduces an unsupported case-insensitive-matching assumption and breaks a correct solution. These examples also show that the selector is useful but imperfect.

Related Work

Iterative Refinement with Code LLMs Iterative refinement constitutes a post-hoc scaling strategy where models systematically correct outputs based on feedback. Originating in general reasoning domains approaches like Self-Refine (Madaan et al., 2023) and Reflexion (Shinn et al., 2023) prompted models to critique their own generations using intrinsic knowledge. In the context of code generation this methodology has evolved to leverage objective execution signals such as compiler errors to enhance debugging capabilities (Zheng et al., 2024). Contemporary research further internalizes this refinement through training objectives

exemplified by SCoRe (Kumar et al., 2025) which optimizes the entire generation trajectory via reinforcement learning and ReflectionCoder (Ren et al., 2025) which distills knowledge from executionbased reflection sequences. CTRL (Xie et al., 2025) trains LLMs to critique without human supervision, enabling them to supervise stronger models and achieve test-time scaling through iterative critiquerevisions. However, these methods necessitate deployed execution environments and explicit feedback loops introducing significant workflow complexity. T IP C ODER addresses this by distilling the foresight from iterative repair into a lightweight proactive tip achieving refinement benefits in a single inference pass without runtime execution. Prompt Optimization and Scaling Prompt optimization functions as inference-time scaling by refining input queries to elicit superior responses. General domain research treats instruction generation as a search problem where models iteratively propose and score candidate prompts (Zhou et al., 2023). Recent frameworks like DIPPER (Hu et al., 2025) further demonstrate that enforcing prompt diversity effectively unlocks varied reason-

Table 2: Cost-normalized comparison on Qwen2.5-Coder-7B. For Best-of-N , tokens include one shared coder prefill, N code completions, and N RM inputs (no RM for N = 1); T IP C ODER includes one Proposer call, two code branches, and two RM inputs. Wall-clock latency is measured on a fixed 100-problem subset of each benchmark using one NVIDIA A800, excluding model loading. For N > 1, latency is code generation plus RM scoring; T IP C ODER latency is max(base code, proposer + guided code) + RM. Method Best-of-1 Best-of-2+RM Best-of-4+RM T IP C ODER-RL+RM Best-of-8+RM Best-of-16+RM

Pass@1 78.86 85.77 83.54 88.21 84.15 84.15

HumanEval+ Tokens Wall (s) 482.65 0.36 1,322.29 0.77 2,408.62 1.15 2,812.13 0.99 4,610.02 1.94 8,980.98 3.41

Pass@1 64.46 70.37 73.07 73.81 73.54 74.60

MBPP+ Tokens 250.47 712.30 1,294.26 1,210.38 2,468.28 4,813.69

Wall (s) 0.22 0.47 0.72 0.73 1.20 2.11

BigCodeBench-full Pass@1 Tokens Wall (s) 41.64 511.76 0.49 36.32 1,619.30 0.94 42.54 3,017.06 1.44 43.89 2,545.53 1.15 44.56 5,828.02 2.48 46.05 11,448.37 4.37

Table 3: Outcome-conditioned analysis of generated-tip categories. Positive/negative denote repairing a base failure/corrupting a correct base solution. Selected-tip category Semantic/constraint clarification Boundary/edge-case handling Index/order/output alignment Algorithmic strategy/invariant Type/parsing/API guidance Numerical/formula/precision Total

Positive 134 56 56 32 18 14 310

Negative 25 17 21 15 18 21 117

ing paths. These advancements establish that allocating compute to explore the instruction space yields predictable gains comparable to increasing model size (Snell et al., 2025). Despite this success, systematic instruction-level scaling remains underexplored in code generation where methods typically rely on stochastic solution sampling. T IP C ODER addresses this gap by introducing Test-time Instruction Proposing to code generation. We learn a proposer policy to efficiently search for reasoning tips and convert inference compute into correctness gains through instruction-space optimization.

Conclusion

We presented T IP C ODER, a black-box test-time framework that explores the instruction space as a complement to solution-space sampling. By training a Proposer via SFT and RL to generate auxiliary instructions, T IP C ODER exposes useful alternative candidates without requiring target-model parameters, gradients, or runtime execution feedback. Across the evaluated Python code-generation benchmarks, the same trained Proposer transfers to four target backbones and is competitive with larger sampling budgets under RM-based selection. Future work includes training with multiple rollout environments, improving post-hoc selection, and evaluating multilingual and repository-level code

Net +109 +39 +35 +17 0 -7 +193

Positive rate among changed 84.3% 76.7% 72.7% 68.1% 50.0% 40.0% 72.6%

generation.

Limitations T IP C ODER has three main limitations. First, it introduces additional inference cost by generating an auxiliary tip and a tip-guided code candidate before reward-model selection. Second, its realized gains depend on the quality of the post-hoc Reward Model: learned tips may expose better candidates, but an imperfect selector may fail to choose them. Third, although the Proposer transfers across multiple target Code LLMs in our experiments, it is trained with a specific rollout environment and debugging-data construction pipeline, so its guidance may still reflect the error patterns of the training environment. Extending T IP C ODER to broader model families, programming languages, and repository-level tasks remains future work.

References Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. 2026. GEPA: Reflective prompt evolution can outperform reinforcement learning. In The Fourteenth International Conference on Learning Representations.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. DeepSeek-Coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. Wenyang Hu, Gregory Kang Ruey Lau, Liu Diwen, Chen Jizhuo, See Kiong Ng, and Bryan Kian Hsiang Low. 2025. Dipper: Diversity in prompts for producing large language model ensembles in reasoning tasks. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35558–35572. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. Qwen2.5-Coder technical report. arXiv preprint arXiv:2409.12186. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. 2025. Training language models to self-correct via reinforcement learning. In The Thirteenth International Conference on Learning Representations. Maya Larbi, Amal Akli, Mike Papadakis, Rihab Bouyousfi, Maxime Cordy, Federica Sarro, and Yves Le Traon. 2025. When prompts go wrong: Evaluating code model robustness to ambiguous, contradictory, and incomplete task descriptions. arXiv preprint arXiv:2507.20439.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558– 21572. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20275–20321. Houxing Ren, Mingjie Zhan, Zhongyuan Wu, Aojun Zhou, Junting Pan, and Hongsheng Li. 2025. ReflectionCoder: Learning from reflection sequence for enhanced one-off code generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9999–10020. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652. Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations. Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong. 2025. Teaching language models to critique via reinforcement learning. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 68559–68577. PMLR. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. In The Twelfth International Conference on Learning Representations.

Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. 2025. ACECODER: Acing coder RL via automated test-case synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12023–12040, Vienna, Austria. Association for Computational Linguistics. Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. OpenCodeInterpreter: Integrating code generation with execution and refinement. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12834–12859. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 14 others. 2025. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations.

MBPP+ Pass@1 (%)

Best-of-N Scaling Across Target Backbones

76 74 72 70 68 66 64 62 60

Best-of-N + existing RM rerank

TIPCoder-RL + AceCodeRM-32B

4 Number of sampled solutions (N)

Qwen2.5-Coder-7B Qwen3-Coder-30B-A3B

DeepSeek-Coder-7B DeepSeek-Coder-V2-Lite

Figure 3: Comparison between solution-space scaling and instruction-guided scaling on MBPP+. Solid curves show Best-of-N sampling with RM-based reranking for N ∈ {1, 2, 4, 8, 16}. Dashed horizontal lines show T IP C ODER-RL with AceCodeRM-32B.

Figure 3 extends Table 2 by comparing T IP C ODER-RL with Best-of-N sampling on MBPP+ across all four target Code LLM backbones. T IP C ODER-RL remains competitive with the Best-of4-to-8 range across the evaluated backbones.

Discussion about Post-hoc Verification Strategy

The post-hoc verification strategy is favored over pre-hoc refusal to maintain proposer diversity and adaptability. Training a refusal mechanism often leads to reward hacking, resulting in conservative policies that suppress high-value guidance. Furthermore, since problem difficulty is model-dependent, the proposer cannot accurately predict failure without access to the generator’s internal state. By utilizing a specialized RM to filter candidates, T IP C ODER implements a dynamic, model-adaptive scaling strategy that maximizes the available potential range without parameter intervention.

Hyperparameter Settings

To ensure the reproducibility of our results, we detail the hyperparameter configurations used in our experiments. Our training pipeline consists of two stages: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). In the SFT stage, we fine-tuned the Qwen3-4B model using the Adam optimizer with state offloading to optimize memory efficiency. We employed a maximum sequence length of 4096 to accommodate long code contexts and set the learning rate to 3e-5. For the RL stage, we utilized the OpenRLHF framework to implement the GRPO algorithm. To maintain stability during the alignment process, we used a lower learning rate of 1e-6 for the actor model and initialized the KL divergence coefficient at 1e-3. We employed group normalization for advantage estimation and applied reward normalization to stabilize the training scale. Additionally, a length penalty of 1e-5 was applied to the RL reward function to discourage verbosity. We conducted all experiments on a computational node equipped with 4 NVIDIA A800 GPUs. To efficiently compute the KL divergence constraint during the GRPO phase we allocated one dedicated GPU for the reference model inference while the remaining three GPUs were utilized for optimizing the policy model. Consequently the global batch sizes were specifically configured to be divisible by 3 to ensure balanced data parallelism across the active training devices. Table 4 presents the comprehensive list of hyperparameters.

Hyperparameter

Value Common Settings Base Model Qwen3-4B-Instruct-2507 Precision BF16 Zero Stage 2 Gradient Checkpointing True Stage 1: Supervised Fine-Tuning Optimizer Adam (Offload) Learning Rate 3 × 10−5 Max Sequence Length 4096 Micro Batch Size 4 Max Epochs 1 Logging Steps 1 Stage 2: Reinforcement Learning (GRPO) Framework OpenRLHF Actor Learning Rate 1 × 10−6 Initial KL Coefficient (β) 1 × 10−3 Advantage Estimator GRPO Reward Normalization True Reward Length Penalty 1 × 10−5 Batch Configuration Train Batch Size 96 Rollout Batch Size 192 Micro Train Batch Size 8 Micro Rollout Batch Size 8 Samples per Prompt (N ) 8 Generation Specs Prompt Max Length 4096 Generate Max Length 4096 Training Loop Max Epochs 1 Num Episodes 1

Table 4: Comprehensive hyperparameter settings for SFT and RL stages.

Case Study

Figure 4 presents a representative example where the original problem instruction is ambiguous about whether the input lists have equal lengths. The base model assumes equal lengths and iterates over the first list, which can cause an index error when the other lists are shorter. T IP C ODER generates tips that explicitly warn about variable-length inputs and index alignment. Conditioned on these tips, the generator uses the minimum list length and handles the edge case correctly. This example illustrates the main role of the Proposer: it does not provide the solution code, but surfaces problem-specific constraints and pitfalls that steer the generator away from a likely failure mode.

Prompt Templates

We present the comprehensive prompt specifications utilized throughout the T IP C ODER framework. The templates are categorized into offline data construction protocols and online inference

workflows. For data construction we design two distinct mechanisms where the first employs direct contrast between incorrect and correct solutions to distill static tips while the second leverages multiturn debugging trajectories to extract executionbased reasoning logic. The inference process follows a structured three-stage pipeline involving a Proposer for tip generation and a Generator for code synthesis followed by an Evaluator for quality scoring. Visualizations of these templates are provided in Figures 5, 6, 7 to illustrate the input-output format and system instructions.

Use of AI Assistants

We used AI assistants to improve writing clarity and automate parts of experimental data collection. We take full responsibility for the content of this paper.

C ASE S TUDY: H ANDLING I MPLICIT C ONSTRAINTS (MBPP/142) [User Instruction] Write a function to count number items that are identical in the same position of three given lists. Note: The prompt is ambiguous regarding whether the three lists have equal lengths.

G ENERATED T IPS (T IP C ODER ): 1. Index Alignment: Ensure that you compare elements at the same index across all lists. If the lists are of unequal length, you must handle cases where one list ends before another. 2. Variable Lengths: Always check if the lists are empty or have different lengths before attempting to iterate. Avoid accessing indices that do not exist. 3. Edge Cases: Handle cases where the lists are empty or have different lengths gracefully. T IP C ODER (S UCCESS ) BASELINE (FAIL ) Logic: Iterates by List 1 def count_samepair ( list1 , list2 , list3 ) : count = 0 # DANGER : Implicitly assumes equal lengths # Will crash if list2 / list3 are shorter for i in range ( len ( list1 ) ) : if list1 [ i ] == list2 [ i ] == list3 [ i ]: count += 1 return count

Analysis: The model assumes fixed lengths, leading to potential IndexError.

Logic: Robust Length Handling def count_samepair ( list1 , list2 , list3 ) : # Handle edge cases if not list1 or not list2 or not list3 : return 0 # SAFE : Use minimum length min_len = min ( len ( list1 ) , len ( list2 ) , len ( list3 ) ) count = 0 for i in range ( min_len ) : if list1 [ i ] == list2 [ i ] == list3 [ i ]: count += 1 return count

Analysis: Guided by the tip, the model uses min() to prevent crashes.

Figure 4: Comparison of code generation with and without T IP C ODER. The baseline model fails to account for variable list lengths due to prompt ambiguity, while T IP C ODER proactively generates warnings about "Variable Lengths", leading to a robust solution.

Data Construction B: Trajectory-based Distillation Data Construction A: Direct Contrast Distillation [System] You are given a coding problem and a solution that failed some test cases. Task: Please fix the code to pass all test cases. [User] <Problem Instruction> Problem: Implement a function valid_parentheses(s) that takes a string of parentheses... <Your Previous Solution> Code: def valid_parentheses ( s ): count = 0 for char in s : if char == ’( ’: count += 1 elif char == ’) ’: count -= 1 return count == 0 # Bug : No negative check

<Execution Feedback> Failed Test Cases: 1. assert valid_parentheses(")(()") == False (Expected: False, Got: True) 2. assert valid_parentheses("())(") == False (Expected: False, Got: True) [Assistant] <Analysis & Fix> The previous solution fails to handle cases where a closing parenthesis appears before an opening one... To fix this, we must ensure count never drops below zero. Fixed Code: def valid_parentheses ( s ): count = 0 for char in s : if char == ’( ’: count += 1 elif char == ’) ’: count -= 1 if count < 0: return False # Fixed return count == 0

Figure 5: The prompt template used to distill tips by directly comparing an incorrect solution with a correct one. This simulates the teacher identifying the gap between the student’s attempt and the ground truth.

[System Directive] You are analyzing a programming debugging conversation where a human and an AI assistant are working together to fix code issues. The conversation has been masked to hide specific code implementations and test cases. Task: Extract and summarize key programming tips. Focus on: Problem Analysis, Missing Requirements, Best Practices, and Common Pitfalls. Constraint: Do not provide solutions or code snippets. Be concise. [User Input: Masked Debugging Conversation] <Step 1> User: Implement a function to find the longest substring without repeating characters. Assistant: Here is the solution: [CODE_BLOCK_MASKED] System: Execution failed. [TEST_CASES_MASKED] <Step 2> User: The previous code failed on empty string input. Please fix it. Assistant: I missed the edge case checks. Here is the fixed version: [CODE_BLOCK_MASKED] System: Success! All tests passed. [Assistant Response: Extracted Tips] Based on the debugging process, here are the key tips for this problem: • Edge Case Handling: Always validate if the input string is empty or None at the beginning to avoid runtime errors. • Sliding Window Strategy: When maintaining the window of unique characters, ensure the left pointer is updated correctly to optimize efficiency. • State Management: Use a dictionary to store the most recent index of each character for O(1) lookups.

Figure 6: Trajectory-based Distillation. The template employs a masking strategy to abstract away specific implementation details from multi-turn debugging logs, forcing the model to synthesize high-level optimization tips based on the execution feedback loop.

Inference Workflow <Step 1: Proposer (Tip Generation)> [Input] System: You are an expert programming mentor. Analyze the following coding instruction and provide specific tips to help developers implement this correctly. Do not provide solution or any solution-related ideas. Your only task is to provide tips to clarify the instruction and identify potential pitfalls, common mistakes, and key considerations during solving the problem. Your tips should be problem-specific, not general programming principles. Do not conclude debug/test/verify tips. User: {PROBLEM_DESCRIPTION} [Output] Assistant (Proposer): {GENERATED_TIP} (The model predicts a tip based on the problem description.) <Step 2: Generator (Code Synthesis)> [Input] User: {GENERATED_TIP} {PROBLEM_DESCRIPTION} (The tip is prepended to the user query as a hint.) [Output] Assistant (Generator): def solution ( nums ): # Generated code ... return result

<Step 3: Evaluator (Reward Model Scoring)> [Input Format] The Reward Model accepts the interaction history as a structured chat template: [ {"role": "user", "content": PROBLEM}, {"role": "assistant", "content": GENERATED_CODE} ] [Process] tokenizer.apply_chat_template(chats, return_tensors="pt") → AceCodeRM(input_tokens) [Output] Scalar Reward: 23.0688 (A higher score indicates higher confidence in correctness, used for reranking or RL feedback.)

Figure 7: The complete inference pipeline. In Step 1, the Proposer predicts a tip. In Step 2, the Generator synthesizes code using the tip. Finally, in Step 3, the AceCodeRM evaluates the quality of the (Problem, Code) pair to assigning a scalar reward.

TIPCODER: Reinforcement Learning Boosted Test-time Instruction Proposer for Code Generation

Analyze & explore related research in Conceptio

Related documents