Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

arXiv:2611.04320v1 [cs.AI] 3 Nov 2026

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study WENTAO WANG, Peking University, China ZEYU SUN, Institute of Software, Chinese Academy of Sciences, China RUWEI PAN, Peking University, China YUEYI XIAO, Peking University, China QINGYUAN LIANG, Peking University, China LU ZHANG, Peking University, China Planning has been identified as an important factor for improved code generation with Large Language Models (LLMs). However, the interaction between planning capability and coding capability remains largely underexplored, especially when the two capabilities are from different LLMs. In this paper, we conduct the first comprehensive empirical study of planning-coding interaction in LLM-based code generation, where planning and coding are assigned to different LLMs (i.e., planning-coding separation). Specifically, we explore different planner-coder combinations to investigate the effectiveness for coders to utilize planning information from various planners and the impact of further training of coders on planner-coder combinations. Our results demonstrate that almost all planner-coder combinations are as competitive as using the same LLM for planning and coding, and the combination of a stronger planner and a stronger coder typically achieves improved effectiveness. Furthermore, using a large model as the planner paired with a much smaller model as the coder can sometimes be as competitive as using the large model as both the planner and the coder, and further training of the coder can typically boost various planner-coder combinations. These findings provide empirical evidence that planning and coding can be separated to a considerable extent in LLM-based code generation, which may offer a more flexible way to construct code-generation systems. CCS Concepts: • Software and its engineering → Automatic programming. Additional Key Words and Phrases: Code Generation, Large Language Models, Planning-Coding Separation

1

Introduction

Large Language Models (LLMs) have made substantial progress in code generation [8, 15, 25]. Their capabilities have been well demonstrated across programming tasks ranging from functionlevel [24, 28, 32, 54] to repository-level [3, 29, 50, 52]. Recent studies increasingly use explicit planning to further improve code generation performance [7, 9, 43]. By formulating a solution plan before implementation, an LLM can externalize key algorithmic decisions and use them to guide subsequent code generation [21]. Existing planning-based approaches are commonly based on a close coupling between planning and coding. Prompt-based approaches typically use the same LLM to first generate a plan and then implement it [16, 23]. Multi-agent approaches assign planning and coding to different roles, but often instantiate these roles with the same underlying LLM [12, 20]. Training-based approaches also rely on a strong connection between planning and coding: Some approaches distill planning and reasoning capabilities of larger models into smaller models [35], while others explicitly train coders to better follow and implement plans provided by planners [10]. In these settings, planning and coding are usually developed and evaluated within one particular model or a tightly coupled planner-coder configuration, indicating that the coordination between the planner and the coder should be carefully dealt with. Recently, researchers have conducted further investigations into cross-model planner-coder configurations, i.e., using plans produced by one LLM to assist coding Authors’ Contact Information: Wentao Wang, wwt@stu.pku.edu.cn, Peking University, Beijing, China; Zeyu Sun, Institute of Software, Chinese Academy of Sciences, Beijing, China; Ruwei Pan, Peking University, Beijing, China; Yueyi Xiao, Peking University, Beijing, China; Qingyuan Liang, Peking University, Beijing, China; Lu Zhang, Peking University, Beijing, China.

2

Wang et al.

by another [18, 47]. These investigations are often motivated by practical objectives, such as reducing inference cost. Although these studies investigate some less tightly coupled planner-coder configurations, they typically focus on specific planner-coder configurations designed for specific objectives rather than examining planning and coding as underlying capabilities. As LLMs continue to exhibit increasingly diverse capabilities, a fundamental question emerges: Should the ability of an LLM-based code generation system be understood as a unified capability of a single model, or as the result of multiple capabilities that can be separately characterized and leveraged? In this context, planning and coding provide an important case for examining whether different stages of code generation exhibit similar or different capability patterns. Understanding this relationship is thus important for revealing how planning and coding interact in LLM-based code generation and how these capabilities can be better studied and improved. To help researchers have a deeper understanding of the relationship between models’ planning capability and coding capability, we conduct a comprehensive empirical study on planning-coding interaction in LLM-based code generation. In our study, we separate planning from coding and assign the two stages to independently configurable LLMs, which we refer to as planning-coding separation. Given a programming problem, a planner first generates a solution plan, and a coder then generates code based on the problem and the plan. This formulation makes it possible to vary planners and coders independently and examine how planning information from different planners is utilized by different coders. Specifically, we investigate planning-coding interaction from three perspectives. First, we systematically vary the planner and coder across different LLMs to examine how the resulting performance changes under different pairings and whether the relative effectiveness remains consistent across different settings of the two stages. Second, we study the impact of model scales by varying the sizes of the planner and coder independently, allowing us to compare how additional model capacity contributes to overall effectiveness. Third, we further train coders with standard instruction-tuning data for code generation and evaluate the trained coders with different planners to examine how coder-side improvement changes the overall effectiveness. Our results reveal three main findings. (1) Planning and coding show substantial compatibility across LLMs: cross-model plannercoder configurations are generally competitive with same-model configurations, and stronger planners and stronger coders generally lead to better combined performance. When comparing each cross-model pair with the corresponding same-model pair, 27 of 44 pairs (61.4%) achieve equal or better performance. Moreover, the relative effectiveness of planners remains consistent across different coders in 9 out of 10 model-group comparisons, suggesting that effective planner-coder combinations are not restricted to the use of the same LLM. (2) Planning and coding exhibit different requirements for model scales. Although increasing the coder scale generally improves plan-assisted generation, a strong planner can enable substantially smaller coders to achieve competitive performance. For example, when GPT-5.6-Sol is used as the planner, Qwen3.5-27B and Qwen3.8-27B coders achieve 68.00 and 71.05 Pass@1 on LiveCodeBench, matching or even exceeding the performance of using GPT-5.6-Sol for both planning and coding (66.86 Pass@1). With further scale reduction, Qwen3.5-27B (planner) paired with Qwen3-4B (coder) still achieves 37.71 Pass@1, significantly outperforming direct generation with Qwen3-32B (27.43 Pass@1). These results highlight the potential of the Large-Planner-Small-Coder (LPSC) pattern, where a strong planner can be combined with a substantially smaller coder. (3) Further training of the coder generally improves its performance with different planners, suggesting that planner-agnostic improvements on the coder side are beneficial across different planner-coder combinations. When using the same planning inputs before and after standard code instruction tuning, the training improves performance in 19 of 20 plan-assisted code generation

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

3

settings, and the training effect follows the same direction as direct generation in 22 of 24 planassisted code generation settings across the two benchmarks. These results indicate that coder-side improvements can transfer across different planners even without planner-specific adaptation. The main contributions of this paper are as follows: • To the best of our knowledge, we are the first to systematically study planning-coding separation across LLMs. That is, rather than focusing on a fixed relationship between the two stages, we independently vary the models responsible for planning and coding to study how their interaction affects code generation. • Our study of planning-coding separation adopts diverse model families and scales, and further examines the effect of coder training with different planners. This provides a controlled characterization of how planner and coder choices jointly shape generation performance. • Our study reveals that planning and coding can be separated to a substantial extent: Planning information is effectively transmitted across LLMs, planning-coding separation can effectively operate with substantially different model scales, and planner-agnostic improvements of a coder remain effective across multiple planners. These findings provide a foundation for further investigation of more flexible planner-coder configurations in LLM-based code generation. 2 2.1

Preliminaries Solution Information

Planning-based code generation commonly introduces intermediate solution information before code implementation. Such information typically describes the solution to be implemented and provides guidance for subsequent code generation. Although it may contain some reasoning-related content, it differs from the reasoning process through which a solution is derived. A reasoning process may include exploration, alternative attempts, self-reflection, and/or revision [36], whereas the solution information provides representations of resulting solutions. Existing approaches represent such solution information in various forms, including concise solution steps [16], structured solution descriptions [23], natural language solution descriptions [39] and algorithmic editorials [9]. The provided information may range from high-level solution strategies and major procedures to more structured and implementation-oriented guidance. Prior work explores hierarchical decomposition of solutions [31, 48], program-structure-aware representations [23], and detailed implementation decisions such as algorithmic choices, data structures, and key operations [9]. Other approaches further emphasize constraints, edge cases, and solution validation during planning [20, 21]. Despite these differences in form and level of detail, such solution information shares a common role: bridging the problem description and final implementation by specifying how a problem should be solved while leaving the concrete code realization to the subsequent coding stage. Among these forms, solution steps have been widely adopted as a practical form of intermediate solution information in code generation [12, 17, 53]. Following this formulation, we refer to this form of solution information as solution plans and use them as the intermediate representation transmitted from planners to coders. Specifically, the plans in our study follow step-based planning [16] and structured solution representation [23], where a plan is represented as ordered steps for the final implementation, supplemented with task requirements, constraints, edge cases, output requirements, data structures, program states, control flow information, and key operations. To focus on the resulting solution rather than its derivation or concrete realization, we exclude exploratory reasoning, alternative solutions, and code.

4

2.2

Wang et al.

Planning-Coding Separation

Most existing code LLMs are based on next-token prediction, where generated texts can be directly used as contexts for subsequent generation. Therefore, planning and coding are typically performed by the same LLM or closely related models, as these models share similar internal mechanisms and model configurations. In our study, we consider a planning-coding separation setting, where planning and coding are assigned to potentially different LLMs. This setting can be formulated as: 𝑝 ∼ 𝑀𝜃 𝑃 (𝑝 |𝑥),

𝑐 ∼ 𝑀𝜃𝐶 (𝑐 |𝑥, 𝑝),

where 𝑥 denotes the programming problem, 𝑝 denotes the solution plan, 𝑐 denotes the generated code, and 𝑀𝜃 𝑃 and 𝑀𝜃𝐶 denote the planner and coder with parameters 𝜃 𝑃 and 𝜃𝐶 , respectively. Therefore, if we set 𝜃 𝑃 = 𝜃𝐶 , we use the same LLM for both planning and coding. In contrast, assigning the two stages to different LLMs corresponds to 𝜃 𝑃 ≠ 𝜃𝐶 . 3

Research Questions (RQs)

Our study aims to investigate the following research questions: RQ1: How do different combinations of planners and coders perform? This RQ aims to investigate to what extent planning and coding can be decoupled. To achieve this purpose, we systematically vary the LLM used for each stage and compare the effectiveness of same-model and cross-model pairings. This also allows us to examine how performance changes when either the planner or coder changes, and whether models exhibit consistent relative effectiveness across the two stages. RQ2: Do planners and coders have different requirements for model scales? This RQ aims to investigate to what extent planning-coding separation can lead to a reduced need for large LLMs as either the planner or the coder for code generation. To achieve this purpose, we systematically vary planner and coder sizes to examine how the model scale of each stage impacts overall effectiveness and whether effective code generation requires comparable model capacity for planning and coding. RQ3: Can coder-side training produce improvements generalizable across different planners? This RQ aims to investigate to what extent further training of coders can lead to improved effectiveness of various planner-coder combinations. To achieve this purpose, we further train coders using standard instruction-tuning data for code generation and evaluate the effectiveness of using coders before and after training paired with various planners. 4 4.1

Experimental Setup Benchmarks

We evaluate on two recent public code-generation benchmarks, LiveCodeBench and Code2Bench. We select them because their tasks are self-contained and sufficiently challenging to require both solution formulation and code implementation, making them suitable for controlled comparisons across different planner-coder settings. Use of these recent task collections also helps reduce potential concerns about contamination from model training data. Moreover, the two benchmarks provide complementary task formats, covering complete program generation and function-level implementation under explicit constraints. LiveCodeBench [14]. Specifically, we use LiveCodeBench v6, which contains 175 Python problems released between January and April 2025. Each problem provides a natural-language specification and requires generating a complete program that reads from standard input and produces the required output. The generated programs are executed against the benchmark test cases and evaluated for functional correctness.

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

5

Code2Bench [51]. For Code2Bench, we use the CODE2BENCH-2509 release and select its Strongly Constrained Python subset, which contains 217 function-level tasks constructed from code committed between May and September 2025. Each task provides a predefined function signature together with explicit behavioral constraints that the generated implementation must satisfy. Code2Bench evaluates functional correctness through property-based testing, which exercises the generated implementation on diverse inputs and compares its behavior with the corresponding ground-truth implementation. 4.2

Models

We evaluate LLMs spanning different model families, parameter scales, and model generations. Qwen3, Ministral-3, and Gemma-3 provide multiple model sizes within the same family, enabling controlled comparisons across both families and scales. We also include more recent and stronger models from Qwen, DeepSeek, and GPT to broaden the range of evaluated LLMs. In addition, we use Qwen2.5-Coder models to provide a code-focused, non-reasoning setting for controlled experiments of coder training. Qwen3 family [44]. Qwen3 is an open-weight LLM family developed by Alibaba. A distinguishing feature of Qwen3 is its hybrid thinking mechanism, which supports both thinking and non-thinking modes. We use four dense variants, Qwen3-4B, Qwen3-8B, Qwen3-14B, and Qwen332B. Their broad range of parameter sizes provides a natural setting for examining model-scale impacts within the same family. Ministral-3 family [27]. Ministral-3 is a family of compact open-weight models developed by Mistral AI, designed to provide strong general-purpose performance at relatively small model sizes. We use Ministral-3-3B-Instruct-2512, Ministral-3-8B-Instruct-2512, and Ministral-3-14B-Instruct2512. These sizes closely overlap with those covered by Qwen3 and Gemma-3, providing comparable models from a different family. Gemma-3 family [38]. Gemma-3 is Google’s open model family covering a range of model sizes for general-purpose language tasks. We use the instruction-tuned Gemma-3-4B-it, Gemma-312B-it, and Gemma-3-27B-it. The three variants provide another independent multi-scale family and complement the scale ranges covered by Qwen3 and Ministral-3. Qwen3.5 [33] and Qwen3.8 [34]. Qwen3.5 and Qwen3.8 are two later generations of Qwen models, distinct from the Qwen3 family above. We use Qwen3.5-27B and Qwen3.8-27B, which are both dense models at the 27B scale. Qwen3.8 builds on the architectural foundation of Qwen3.5 and represents a newer and more capable model. These two recent Qwen models are at the same nominal parameter scale. DeepSeek family [41, 42]. We use DeepSeek-V4-Pro and DeepSeek-V4.1-Flash, two large-scale MoE models from DeepSeek. DeepSeek-V4-Pro contains 1.6T total parameters with 49B activated per token, whereas the newer DeepSeek-V4.1-Flash contains 552B total parameters and adopts an asymmetric architecture that activates 8B parameters for input and 16B for output. Despite its substantially smaller active parameter footprint, DeepSeek reports that V4.1-Flash outperforms V4-Pro on its evaluated benchmarks. GPT-5.6-Sol [30]. GPT-5.6-Sol is the flagship model in OpenAI’s GPT-5.6 family, designed for complex professional workloads. We include it as a strong proprietary model, complementing the open-weight models in our evaluation. Qwen2.5-Coder family [11]. Qwen2.5-Coder is a code-specific model family built on Qwen2.5 and further trained on large-scale code-related data. We use Qwen2.5-Coder-1.5B-Instruct and Qwen2.5-Coder-3B-Instruct. Unlike reasoning-mode models such as Qwen3, Qwen2.5-Coder does not expose a dedicated thinking mode, making it suitable for studying coder-side improvement

6

Wang et al.

Text is not SVG - cannot display

Fig. 1. Planning prompt.

without introducing an additional reasoning mechanism. Its relatively small model sizes also make controlled fine-tuning practical. 4.3

Experimental Settings

Model serving. GPT-5.6-Sol and the DeepSeek models are accessed through their official APIs, while the remaining open-weight models are deployed locally using vLLM. Across serving backends, we use the same prompt templates and generation protocol whenever applicable. Plan generation. Prompt. We use the same planning prompt template for all models and both benchmarks to avoid model-specific prompt adaptation. The complete prompt is shown in Figure 1. Sampling. We use sampling for plan generation rather than greedy generation, preserving the normal stochastic generation behavior of each planner. For each planner and each task, we issue three independent requests to reduce the influence of any individual sampled plan while keeping the overall evaluation cost manageable. We set the maximum output length to 2,048 tokens and otherwise retain the default sampling parameters of each serving endpoint to avoid model-specific parameter tuning. Plan retention. All three plans are retained and used as generated, without revision, filtering, or selection based on subsequent code generation performance. For each planner and each task, the same three plans are reused across all coders, providing fixed planning inputs for coder comparisons. Code generation. Input. For plan-assisted generation, the coder receives the original task specification, one generated plan, and the benchmark-specific output instruction. Each of the three plans is used once to generate a candidate solution. Direct generation uses the same task specification and output instruction without any plan. Generation. We use greedy generation with temperature 0, top-𝑝 1, and a maximum output length of 8192 tokens. This deterministic setting reduces sampling variation in code generation and provides a stable basis for comparing different coders under the same planning input. Thinking mode. For models that provide an optional thinking mode, we disable it during both planning and coding. This reduces the influence of model-specific reasoning modes and makes the evaluation more focused on the explicitly generated plan and the coder’s use of that plan.

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

7

Output processing. The model is instructed to return the generated code in a code block. For evaluation, we extract the last code block from the model output as the candidate solution. No additional repair or regeneration is applied. Coder training. Training data. We use OpenCodeInstruct [1], a large-scale collection of standard instruction-tuning data for code generation. Its broad coverage of programming tasks provides diverse supervision for improving coding capability. We randomly sample one million examples for Supervised Fine-Tuning (SFT). Training setting. We perform full-parameter supervised fine-tuning for one epoch with a learning rate of 10−5 , a cosine learning-rate schedule, a warmup ratio of 0.1, and a batch size of 64. 4.4

Evaluation

We use execution-based Pass@1 as the primary evaluation metric because both benchmarks provide executable tests for functional correctness and each code-generation request produces a single solution. For a benchmark with 𝑁 tasks, the Pass@1 of one evaluation run is computed as 𝑁

Pass@1 =

1 ∑︁ I[Eval(𝑐𝑖 ) = 1] , 𝑁 𝑖=1

where 𝑐𝑖 denotes the generated solution for task 𝑖, and Eval(𝑐𝑖 ) = 1 if it passes the corresponding benchmark evaluation. Direct generation contains one evaluation run and therefore reports this Pass@1 directly. For plan-assisted generation, the three independently generated plans define three separate evaluation runs, each producing one Pass@1 value. We report the arithmetic mean of these three values: 3

Pass@1 =

1 ∑︁ Pass@1𝑟 . 3 𝑟 =1

Averaging at the run level reduces the influence of any single sampled plan and avoids favoring particular plan samples, leading to a more stable comparison across different model settings. 5 5.1

Experimental Designs and Results RQ1: Cross-Model Pairing and Relative Effectiveness

Design. We organize the evaluated LLMs into five groups for cross-model comparison. Four groups contain models with comparable parameter sizes: 3B–4B (Gemma-3-4B-it, Ministral-33B-Instruct-2512, and Qwen3-4B), 8B (Ministral-3-8B-Instruct-2512 and Qwen3-8B), 12B–14B (Gemma-3-12B-it, Ministral-3-14B-Instruct-2512, and Qwen3-14B), and 27B–32B (Gemma-3-27B-it and Qwen3-32B). The fifth group consists of three advanced models: DeepSeek-V4-Pro, DeepSeekV4.1-Flash, and GPT-5.6-Sol. Within each group, every model is used as both planner and coder and paired with every model in the other role, forming a complete matrix of model combinations. Results. Tables 1 and 2 compare same-model and cross-model planner-coder configurations across different LLMs, showing the following observations. Observation 1: Using the same LLM for planning and coding does not provide a systematic advantage. Across the two benchmarks, 27 of the 44 cross-model pairs (61.4%) perform no worse than the corresponding same-model pair for the same coder, including 14 of 22 pairs on LiveCodeBench and 13 of 22 on Code2Bench. Moreover, the best-performing configuration is cross-model in seven of the ten model-group settings across the two benchmarks. These patterns are observed across different model scales. For example, on LiveCodeBench, using Qwen3-14B as the planner improves Gemma-3-12B-it and Ministral-3-14B-Instruct-2512 over their same-model results

8

Wang et al.

Table 1. Planner–coder pairing results on LiveCodeBench (Pass@1). Parentheses show the relative change from the same-model pair for each coder. Bold marks the best plan-assisted result for each coder; gray marks the highest-scoring pair in each model group. Coder

Direct

3B–4B Qwen3-4B Gemma-3-4B-it Ministral-3-3B-Instruct

24.57 18.86 23.43

Planner Qwen3-4B

Gemma-3-4B-it

24.95 19.62 (+14.4%) 26.48 (+36.3%)

23.43 (-6.1%) 17.14 24.19 (+24.5%)

Qwen3-8B

8B Qwen3-8B Ministral-3-8B-Instruct

Qwen3-14B

12B–14B Qwen3-14B Gemma-3-12B-it Ministral-3-14B-Instruct

28.00 25.71 32.57

31.62 30.10

27.43 30.29

65.14 53.71 65.71

56.57 53.90 64.95

(+6.4%) (-2.8%)

(-13.7%) (+3.3%)

(-10.1%)

Ministral-3-14B-Instruct

(-7.5%)

28.76 25.33 29.52

(+5.8%)

(-5.0%) (+0.8%)

Gemma-3-27B-it 31.24 27.62

(+9.0%)

DeepSeek-V4.1-Flash

Advanced Models DeepSeek-V4.1-Flash DeepSeek-V4-Pro GPT-5.6-Sol

28.00 25.14 31.24

Qwen3-32B

27B–32B Qwen3-32B Gemma-3-27B-it

23.62 25.90 Gemma-3-12B-it

30.29 28.57 (+13.6%) 32.19 (+9.0%)

21.52 17.71 19.43

Ministral-3-8B-Instruct

26.29 28.95 (+11.8%)

24.57 29.14

Ministral-3-3B-Instruct

DeepSeek-V4-Pro 56.57 50.67 60.00

(0.0%) (-10.3%)

(-1.2%)

GPT-5.6-Sol 72.95 (+29.0%) 72.19 (+42.5%) 66.86

by 13.6% and 9.0%, respectively. Among the advanced models, using GPT-5.6-Sol as the planner improves DeepSeek-V4.1-Flash by 29.0% and DeepSeek-V4-Pro by 42.5%. Overall, the planning information seems to be a good intermediate representation to decouple planners and coders. Observation 2: The relative effectiveness of planners is largely consistent across different coders. In 9 of the 10 model-group settings across the two benchmarks, the same planner achieves the highest Pass@1 for every coder within the group. This pattern appears across different model scales and on both benchmarks. In the 3B–4B, 8B, and 12B–14B groups, Qwen3 consistently achieves the best planning results for all coders on both LiveCodeBench and Code2Bench. Among the advanced models, GPT-5.6-Sol similarly achieves the best results for all three coders on both benchmarks. The only exception occurs in the 27B–32B group on Code2Bench, where the preferred planner differs between the two coders. However, this difference is marginal for the Qwen3-32B coder, with Gemma-3-27B-it exceeding Qwen3-32B by only 0.31 Pass@1 (30.88 vs. 30.57), while Qwen3-32B retains a clear advantage for the Gemma-3-27B-it coder (29.95 vs. 27.04). Overall, the relative effectiveness of planners remains largely stable when paired with different coders. Observation 3: Strong planner-side and coder-side performance usually leads to the best configuration. At the model-group level, 7 of the 10 comparisons have a best-performing configuration that combines a planner consistently performing best across coders with a coder performing best or tied-best under most planners. The planner-side pattern appears in 9 of the 10 comparisons, as shown in Observation 2, while the corresponding coder-side pattern appears in 8 of the 10 comparisons. Their overlap in 7 comparisons indicates that the strongest configuration is

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

9

Table 2. Planner–coder pairing results on Code2Bench (Pass@1). Parentheses show the relative change from the same-model pair for each coder. Bold marks the best plan-assisted result for each coder; gray marks the highest-scoring pair in each model group. Coder

Direct Qwen3-4B

3B–4B Qwen3-4B Gemma-3-4B-it Ministral-3-3B-Instruct

Planner

24.88 21.20 24.88

22.58 22.43 (+23.7%) 23.96 (+30.0%)

Qwen3-14B 32.26 28.57 30.41

32.41 29.80 (+18.3%) 32.26 (+14.1%)

Qwen3-32B Gemma-3-27B-it

29.49 30.88

DeepSeek-V4.1-Flash DeepSeek-V4-Pro GPT-5.6-Sol

36.41 36.41 36.87

35.48 35.94 36.10

(-0.8%) (-12.3%)

(-21.1%) (-8.5%)

Ministral-3-8B-Instruct 23.20 23.35 Gemma-3-12B-it 29.19 25.19 29.65

(-14.2%)

Ministral-3-14B-Instruct

(-10.0%)

27.96 26.11 28.26

(+4.9%)

(-13.7%) (+3.7%)

Gemma-3-27B-it 30.88 27.04

30.57 29.95 (+10.8%) DeepSeek-V4.1-Flash

Advanced Models

17.82 16.59 18.43

(+8.3%)

Qwen3-32B

27B–32B

Ministral-3-3B-Instruct

(-17.7%)

27.04 27.96 (+19.7%)

24.88 28.57

12B–14B Qwen3-14B Gemma-3-12B-it Ministral-3-14B-Instruct

18.59 18.13 19.97

Qwen3-8B

8B Qwen3-8B Ministral-3-8B-Instruct

Gemma-3-4B-it

DeepSeek-V4-Pro 36.56 36.25 36.56

(+3.0%) (-11.2%)

(+1.0%)

GPT-5.6-Sol 40.25 (+13.4%) 40.25 (+11.0%) 41.17

usually explained by the effectiveness of its two components rather than by an isolated planner– coder match. The 8B group provides a clear example on both benchmarks. Qwen3-8B achieves the best planning results for both coders, while Ministral-3-8B-Instruct-2512 achieves the best coding results under both planners. Their combination therefore produces the best 8B result on both LiveCodeBench (28.95) and Code2Bench (27.96). Note that this principle is irrespective of whether the same or different LLMs are used in the two stages. On LiveCodeBench, Qwen3-32B is strongest on both the planning and coding sides, and using Qwen3-32B for both stages gives the best 27B–32B result at 31.62. Among the seven comparisons following this pattern, five are from a cross-model configuration and two are from a same-model configuration. Direct performance provides an additional reference for coding effectiveness. In 6 of the 10 model-group comparisons, the coder in the best configuration also has the highest or tied-highest direct Pass@1. For example, Ministral-3-8B-Instruct-2512 has the highest Direct performance in the 8B group and also remains the stronger coder under planning on both benchmarks. However, this correspondence is not universal, as shown by the 27B–32B group on LiveCodeBench, where Gemma-3-27B-it has higher Direct performance but Qwen3-32B performs better as the coder under either planner. Overall, our results suggest that strong configurations can be obtained by considering plannerside and coder-side effectiveness separately, rather than relying on a specific relationship between model identities.

10

Wang et al.

Finding 1. Cross-model planner–coder configurations are generally competitive with samemodel configurations, and pairing stronger planners with stronger coders tends to yield better combined performance. This finding supports combining an effective planner with a strong coder across model boundaries, rather than restricting both stages to the same LLM. 5.2

RQ2: Scale Requirements of Planning and Coding

Design. We examine model scaling from both the planner and coder sides. For planner scaling, we vary model sizes within the Qwen3, Gemma-3, and Ministral-3 families with fixed small and larger coders, and further include Qwen3.5-27B and three advanced models to broaden the planning settings. For coder scaling, Qwen3.5-27B and the three advanced models are paired with different coder sizes from the same three families, together with Qwen3.5-27B and Qwen3.8-27B. By varying one side while fixing the other, we examine the scaling behavior of each stage and whether planning and coding require comparable model scale. Results. Figures 2 and 3 show how performance changes with planner scale for small and larger coders, while Tables 3 and 4 show how performance changes with coder scale under different planners. These results lead to the following observations.

Text is not SVG - cannot display

Fig. 2. Planner performance across model scales with three small coders. Q3, Gem, and Min abbreviate Qwen3, Gemma-3-it, and Ministral-3-Instruct-2512; DS Flash/Pro abbreviate DeepSeek-V4.1-Flash/V4-Pro, and GPT abbreviates GPT-5.6-Sol.

Observation 4: Increasing planner scale generally improves plan-assisted performance. With small coders fixed, plan-assisted performance generally increases with planner scale within the same model family, particularly for Gemma-3 and Ministral-3. For example, with Qwen3-4B

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

11

Text is not SVG - cannot display

Fig. 3. Planner performance across model scales with three larger coders. Q3, Gem, and Min abbreviate Qwen3, Gemma-3-it, and Ministral-3-Instruct-2512; DS Flash/Pro abbreviate DeepSeek-V4.1-Flash/V4-Pro, and GPT abbreviates GPT-5.6-Sol. Table 3. Coder performance across model scales with four planners on LiveCodeBench (Pass@1). Parentheses show the relative change from Direct for each coder. Bold marks the best planner for each coder; gray marks the best coder for each planner within each group.

Coder

Planner

Direct Qwen3.5-27B

DeepSeek-V4.1-Flash

DeepSeek-V4-Pro

GPT-5.6-Sol

(+73.6%) (+76.0%) (+67.3%) (+53.5%)

50.10 (+103.9%) 51.24 (+108.5%) 55.24 (+97.3%) 53.52 (+95.1%)

27.24 (+44.4%) 38.48 (+49.6%) 42.10 (+39.0%)

28.00 (+48.5%) 38.10 (+48.1%) 43.05 (+42.1%)

26.10 (+38.4%) 44.76 (+74.1%) 52.00 (+71.7%)

33.14 (+41.5%) 38.29 (+31.4%) 38.48 (+18.1%)

34.67 (+48.0%) 40.76 (+39.9%) 44.57 (+36.8%)

35.43 (+51.2%) 42.67 (+46.4%) 42.86 (+31.6%)

40.19 (+71.5%) 52.38 (+79.7%) 56.57 (+73.7%)

48.19 50.86

55.81 (+19.1%) 59.43 (+2.0%)

52.00 (+11.0%) 55.24 (-5.2%)

68.00 (+45.1%) 71.05 (+21.9%)

Qwen3-4B Qwen3-8B Qwen3-14B Qwen3-32B

24.57 24.57 28.00 27.43

37.71 39.05 40.57 40.19

(+53.5%) (+58.9%) (+44.9%) (+46.5%)

Gemma-3-4B-it Gemma-3-12B-it Gemma-3-27B-it

18.86 25.71 30.29

25.14 (+33.3%) 36.38 (+41.5%) 38.67 (+27.7%)

Ministral-3-3B-Instruct Ministral-3-8B-Instruct Ministral-3-14B-Instruct

23.43 29.14 32.57

Qwen3.5-27B Qwen3.8-27B

46.86 58.29

(+2.8%) (-12.7%)

40.95 42.29 44.38 44.57

(+66.7%) (+72.1%) (+58.5%) (+62.5%)

42.67 43.24 46.86 42.10

12

Wang et al.

Table 4. Coder performance across model scales with four planners on Code2Bench (Pass@1). Parentheses show the relative change from Direct for each coder. Bold marks the best planner for each coder; gray marks the best coder for each planner within each group.

Coder

Planner

Direct Qwen3.5-27B

DeepSeek-V4.1-Flash

DeepSeek-V4-Pro

GPT-5.6-Sol

Qwen3-4B Qwen3-8B Qwen3-14B Qwen3-32B

24.88 24.88 32.26 29.49

30.57 (+22.8%) 31.80 (+27.8%) 33.64 (+4.3%) 33.18 (+12.5%)

32.10 (+29.0%) 32.41 (+30.2%) 33.95 (+5.2%) 34.10 (+15.6%)

32.87 (+32.1%) 34.25 (+37.7%) 35.02 (+8.6%) 35.48 (+20.3%)

34.56 34.87 37.94 37.94

Gemma-3-4B-it Gemma-3-12B-it Gemma-3-27B-it

21.20 28.57 30.88

27.50 (+29.7%) 31.34 (+9.7%) 31.80 (+3.0%)

28.42 (+34.1%) 33.79 (+18.3%) 33.18 (+7.5%)

30.41 (+43.5%) 34.72 (+21.5%) 34.41 (+11.4%)

29.80 (+40.6%) 36.41 (+27.4%) 38.25 (+23.9%)

Ministral-3-3B-Instruct Ministral-3-8B-Instruct Ministral-3-14B-Instruct

24.88 28.57 30.41

27.80 (+11.7%) 32.57 (+14.0%) 31.95 (+5.1%)

31.95 (+28.4%) 33.95 (+18.8%) 35.18 (+15.7%)

33.79 (+35.8%) 35.18 (+23.1%) 34.87 (+14.6%)

32.87 (+32.1%) 35.79 (+25.3%) 39.78 (+30.8%)

Qwen3.5-27B Qwen3.8-27B

34.10 34.56

34.56 35.02

34.56 34.10

35.94 36.25

38.10 (+11.7%) 38.40 (+11.1%)

(+1.4%) (+1.3%)

(+1.4%) (-1.3%)

(+5.4%) (+4.9%)

(+38.9%) (+40.1%) (+17.6%) (+28.6%)

as the coder, scaling the Gemma-3 planner from 4B to 27B improves Pass@1 from 23.43 to 29.71 on LiveCodeBench and from 18.59 to 25.04 on Code2Bench. Qwen3 shows a less pronounced scaling trend, as its smaller variants already provide relatively effective plans, consistent with the observation in RQ1. Qwen3.5-27B and the advanced models achieve further improvements, showing that better planner-side performance can provide additional gains. Observation 5: Strong planners consistently benefit larger coders. When coder scale is relatively large, plans from smaller models provide limited and less consistent gains over direct generation. This may be because larger coders can already obtain much of the required solution information directly from the task, reducing the additional value of plans from smaller models. In contrast, strong planners such as Qwen3.5-27B and the advanced models consistently improve all three larger coders on both benchmarks. For example, with Qwen3-32B as the coder, several smaller planners remain close to its direct performance of 27.43 and 29.49 on LiveCodeBench and Code2Bench, whereas GPT-5.6-Sol increases Pass@1 to 53.52 and 37.94. This contrast suggests that scaling up the coder does not eliminate the advantage of retaining a large model for planning. Observation 6: Increasing coder scale generally improves plan-assisted performance. When the planner is fixed, plan-assisted performance generally increases with the coder scale, particularly for the Gemma-3 and Ministral-3 families. For example, with GPT-5.6-Sol as the planner on LiveCodeBench, scaling Gemma-3 from 4B to 27B improves Pass@1 by 25.90 points, while scaling Ministral-3 from 3B to 14B improves it by 16.38 points. Qwen3 follows the same trend up to 14B, but its 32B variant performs slightly below 14B (53.52 vs. 55.24). This reversal is also observed in direct generation on both benchmarks, suggesting that it may be due to specific characteristics of the two models. One possible explanation is the difference in post-training: Qwen3 models up to 14B use Strong-to-Weak Distillation, whereas Qwen3-32B follows a separate four-stage post-training process [44]. Overall, the results show a clear general benefit from increasing coder scale, with Qwen3-32B as a model-specific exception.

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

13

Observation 7: Planning and coding do not require comparable model scale. A strong planner can retain nearly the same effectiveness when paired with a substantially smaller coder. With GPT-5.6-Sol as the planner, the two 27B coders already achieve performance comparable to the advanced coders on both benchmarks. On LiveCodeBench, Qwen3.5-27B and Qwen3.8-27B achieve 68.00 and 71.05, respectively, compared with 66.86–72.95 for the three advanced coders in Table 1. On Code2Bench, they achieve 38.10 and 38.40, compared with 40.25–41.17 for the advanced coders in Table 2. Therefore, using an advanced model for coding provides only limited additional benefit over a 27B coder when the same strong planner is used. In some settings, the coding model can be reduced further while remaining competitive. On Code2Bench, GPT-5.6-Sol paired with Ministral-3-14B-Instruct achieves 39.78, only 0.47 percentage points of Pass@1 below either DeepSeek advanced coder and 1.39 percentage points below GPT5.6-Sol itself as the coder. More aggressive scale reduction can still preserve substantial gains from planning: Qwen3.5-27B paired with Qwen3-4B achieves 37.71 on LiveCodeBench and 30.57 on Code2Bench, outperforming the direct performance of the eight-times-larger Qwen3-32B on both benchmarks (27.43 and 29.49). These results demonstrate that effective planning can substantially reduce the model capacity required on the coder side. These results reveal asymmetric scaling behaviors between planning and coding. Increasing planner scale generally continues to improve performance across coder scales, whereas the gains from increasing coder scale become more limited under a strong planner. As a result, the coder does not need to scale to the same extent as the planner. Under a strong planner, substantially smaller coders can approach the performance of much stronger coders and, in some settings, remain competitive with using a large model for both planning and coding. This asymmetry motivates Large-Planner-Small-Coder (LPSC), in which a large model is retained for planning while the coding stage uses a substantially smaller model. Finding 2. Planning and coding do not require comparable model scales. A large planner paired with a substantially smaller coder can sometimes remain competitive with using the large model for both stages, supporting LPSC as a practical configuration that retains a large model for planning while substantially reducing the model scale used for coding. 5.3

RQ3: Transferability of Coder Improvement

Design. We further train three coders: Qwen2.5-Coder-1.5B-Instruct, Qwen2.5-Coder-3B-Instruct, and Gemma-3-4B-it. For each coder, we compare its original and fine-tuned versions under both direct and plan-assisted generation. For plan-assisted generation, we reuse the same plans from Qwen3.5-27B and advanced models as in RQ2. For each planner, the original and fine-tuned coders are evaluated using exactly the same plans. Results. Tables 5 and 6 present comparison results of coder performance before and after SFT across different planners. The results reveal the following observations. Observation 8: Coder-side improvements consistently transfer across different planners. SFT improves direct generation in five of the six coder–benchmark settings. For these five settings, the same positive effect appears in 19 of the 20 corresponding plan-assisted settings, with the remaining one unchanged. For example, on LiveCodeBench, Qwen2.5-Coder-3B improves from 14.29 to 20.00 under direct generation, while its performance also increases under all four planners, with relative gains ranging from 4.6% to 10.5%. The pattern is even more consistent on Code2Bench, where all 12 plan-assisted settings benefit from SFT. Importantly, SFT uses only standard code instruction data, and the planning inputs are held fixed before and after training. The results

14

Wang et al.

Table 5. Base and SFT coder performance on LiveCodeBench (Pass@1). Relative Change is computed from Base to SFT. Bold marks the higher Base/SFT score for each condition.

Coder

Planner

Setting Direct

Qwen3.5-27B DeepSeek-V4.1-Flash DeepSeek-V4-Pro GPT-5.6-Sol Base SFT

10.86 13.71

23.05 24.19

24.95 24.95

25.90 27.43

28.00 28.95

Relative Change (%)

+26.3

+5.0

+0.0

+5.9

+3.4

Base SFT

14.29 20.00

26.67 29.14

29.14 32.19

32.95 34.48

34.48 37.33

Relative Change (%)

+40.0

+9.3

+10.5

+4.6

+8.3

Base SFT

18.86 13.71

25.14 25.52

27.24 24.19

28.00 26.29

26.10 25.71

Relative Change (%)

-27.3

+1.5

-11.2

-6.1

-1.5

Qwen2.5-Coder-1.5B

Qwen2.5-Coder-3B

Gemma-3-4B-it

Table 6. Base and SFT coder performance on Code2Bench (Pass@1). Relative Change (%) is computed from Base to SFT. Bold marks the higher Base/SFT score for each condition.

Coder

Planner

Setting Direct

Qwen3.5-27B DeepSeek-V4.1-Flash DeepSeek-V4-Pro GPT-5.6-Sol Base SFT

15.21 21.20

25.19 27.65

25.65 29.19

25.96 31.18

23.35 30.26

Relative Change (%)

+39.4

+9.8

+13.8

+20.1

+29.6

Base SFT

20.28 25.81

28.11 29.49

27.96 31.49

29.95 31.64

30.57 34.10

Relative Change (%)

+27.3

+4.9

+12.6

+5.6

+11.6

Base SFT

21.20 25.81

27.50 28.88

28.42 29.65

30.41 32.10

29.80 31.80

Relative Change (%)

+21.7

+5.0

+4.3

+5.6

+6.7

Qwen2.5-Coder-1.5B

Qwen2.5-Coder-3B

Gemma-3-4B-it

therefore show that coder-side improvements learned independently of planning can be retained across plans generated by different LLMs without planner-specific adaptation. Observation 9: The effect of coder training can vary across benchmarks, but is largely preserved across planners. Gemma-3-4B-it provides a clear contrast. On LiveCodeBench, SFT reduces its direct performance from 18.86 to 13.71. Its performance also decreases under three of the four planners, by 11.2%, 6.1%, and 1.5%, while the remaining planner yields only a marginal 1.5% increase. In contrast, on Code2Bench, the same SFT improves direct performance from 21.20 to 25.81, and performance improves under all four planners, by 4.3%–6.7%. More generally, among all 24 plan-assisted Base–SFT comparisons, 22 change in the same direction as the corresponding direct result, one remains unchanged, and only one changes in the opposite direction. The direction of the training effect is therefore more closely associated with the coder–benchmark setting than

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

15

with a particular planner source, indicating that changing the planner rarely reverses the effect of coder-side training. The consistency of this transfer across the evaluated coder scales and model families points to a broader opportunity for planner-agnostic coder optimization. Although our training experiments are limited to relatively small coders due to resource constraints, this result is particularly relevant to LPSC: training effort can be concentrated on improving the small coder while retaining an already capable large planner. More effective coder-side training may therefore provide a direct path toward further strengthening LPSC and narrowing the gap to configurations that use large models for both planning and coding. Finding 3. Further training of the coder generally improves its performance across different planners, with the resulting gains largely preserved across plan sources. This supports planneragnostic coder-side optimization, allowing training to focus on the coder itself without being tailored to a particular planner and providing a direct path for further strengthening LPSC through targeted improvement of the small coder.

6

Case Study

To further understand the behaviors observed in our evaluation, we analyze a representative crossmodel case involving Qwen3-8B and Ministral-3-8B-Instruct. As shown in Figure 4, both models fail when directly generating code for this problem. Furthermore, only the configuration using Qwen3-8B as the planner and Ministral-3-8B-Instruct as the coder produces a correct solution; all other planner-coder pairs fail to generate a correct solution. A model with failed direct generation can still produce effective solution plans. Although Qwen3-8B fails to generate correct code directly, the solution plan produced by Qwen3-8B provides the key strategy used by the successful planner-coder pair. The plan proposes sorting the array and using combinatorial counting to compute how each element contributes when it serves as the minimum or maximum of subsequences with at most 𝑘 elements. This strategy avoids enumerating all possible subsequences and provides useful guidance for implementation. This observation suggests that direct code generation performance does not fully reflect a model’s ability to provide useful solution plans. The generated plan alone is insufficient for successful code generation. The different outcomes of Qwen3-8B and Ministral-3-8B-Instruct as coders further explain the success of the cross-model pair. When Qwen3-8B consumes its own plan, the generated code remains incorrect. In contrast, Ministral-3-8B-Instruct turns the same plan into a correct implementation by precomputing factorials and modular inverses, calculating the contribution counts for each valid subsequence length, and accumulating the weighted contributions modulo 109 + 7. This comparison indicates that identifying a solution strategy and realizing it in executable code may not necessarily be achieved by the same model. Direct code generation performance does not fully characterize planner behavior. Ministral-3-8B-Instruct demonstrates stronger direct code generation performance than Qwen3-8B in our evaluation. However, assigning it as the planner does not produce the same outcome in this case. Its generated solution plan follows a dynamic programming-based approach with additional state definitions and optimization procedures, but this approach does not produce a correct solution. In contrast, the plan produced by Qwen3-8B is successfully used to generate a correct solution. These results suggest that direct code generation performance alone does not fully characterize a model’s behavior when assigned as a planner.

16

Wang et al. Problem

Constraints 1 ≤ nums.length ≤ 105

You are given an integer array nums and a positive integer k. Return the sum of the maximum and minimum elements of all subsequences of nums with at most k elements. Since the answer may be very large, return it modulo 109 + 7.

0 ≤ nums[i] ≤ 109 1 ≤ k ≤ min(70, nums.length)

Plan Generated By Ministral-3-8B-Instruct (Ministral) 1. Sort the input array nums in ascending order to simplify subsequence generation and min/max identification. 2. Use dynamic programming to precompute powers of MOD (10^9+7) up to the maximum possible subsequences count for efficiency. 3. Initialize a DP table down[i][j] where i represents the first j elements of the sorted array and stores the sum of min*max for all subsequences of length up to k ending at nums[i]. 4. For each element in nums, calculate contributions to subsequences of lengths 1..k by leveraging: - For subsequences of length l: the sum of (current_min) * (current_max) where current_min is nums[i], current_max is the l-th largest in the first i elements (precomputed or derived from prefixes/postfixes). - Use a sliding window or prefix sums to efficiently retrieve 1..k-th order statistics to compute current_max for each possible subsequence length. 5. Sum up all down[i][l] values for all i and l <= k, multiplying each by its DP weight (number of subsequences) during initialization and subtract overlaps/duplicate calculations. 6. Optimize by exploiting the sorted property through binary search or reverse sorted traversal for order statistics. 7. Apply MOD at each intermediate step to prevent overflow and return the final result modulo 10^9+7.

Direct Generation 1

Qwen Direct

Ministral Direct

2. Precompute prefix sums of the sorted array. 3. For each subsequence of length from 1 to k, calculate the sum of minimum and maximum elements for all possible subsequences of that length. 4. Use combinatorics to determine the number of subsequences of each length and their contribution to the total sum. 5. Accumulate the total sum considering the contribution of each element as minimum and maximum in various subsequences. 6. Return the result modulo 10^9 + 7.

Code With Ministral’s Plan Failed

def minMaxSums(self, nums: List[int], k: int) -> int: MOD = 10**9 + 7 nums.sort() n = len(nums) total = 0 prefix = [0] * (n + 1) for i in range(n): prefix[i + 1] = prefix[i] + nums[i] for i in range(n): min_count = 0 for t in range(k): if i + t + 1 <= n: min_count += comb(n - i - 1, t) total += nums[i] * min_count max_count = 0 for t in range(k): if t <= i: max_count += comb(i, t) total += nums[i] * max_count return total % MOD

2

Plan Generated By Qwen3-8B (Qwen) 1. Sort the input array nums.

3

Ministral (Planner) → Qwen (Coder)

Code With Qwen’s Plan Failed

def minMaxSums(self, nums: List[int], k: int) -> int: nums.sort() n = len(nums) total = 0 pow2 = [1] * (n + 1) for i in range(1, n + 1): pow2[i] = (pow2[i - 1] * 2) % MOD prefix = [0] * (n + 1) for i in range(n): prefix[i + 1] = prefix[i] + nums[i] for i in range(n): min_count = pow2[min(k - 1, n - i - 1)] min_sum = nums[i] * min_count total = (total + min_sum) % MOD max_count = pow2[min(k - 1, i)] max_sum = nums[i] * max_count total = (total + max_sum) % MOD return total

Failed

def minMaxSums(self, nums: list, k: int) -> int: n = len(nums) if n == 0: return 0 min_dp = [[0] * (k + 1) for _ in range(n)] max_dp = [[0] * (k + 1) for _ in range(n)] for i in range(n): min_dp[i][1] = 1 max_dp[i][1] = 1 for l in range(2, k + 1): for i in range(n): total = 0 for j in range(i): if nums[j] >= nums[i]: total += min_dp[j][l - 1] total %= MOD min_dp[i][l] = total total = 0 for j in range(i): if nums[j] <= nums[i]: total += max_dp[j][l - 1] total %= MOD max_dp[i][l] = total total_sum = 0 for l in range(1, k + 1): for i in range(n): total_sum += nums[i] * (min_dp[i][l] + max_dp[i][l]) total_sum %= MOD return total_sum

4

Ministral (Planner) → Ministral (Coder)

Failed

def minMaxSums(self, nums: list[int], k: int) -> int: nums.sort() n = len(nums) total = 0 max_n = 10**5 max_k = 70 comb_table = [[0] * (max_k + 1) for _ in range(max_n + 1)] comb_table[0][0] = 1 for i in range(1, max_n + 1): comb_table[i][0] = 1 for j in range(1, min(i, max_k) + 1): comb_table[i][j] = (comb_table[i-1][j-1] + comb_table[i-1][j]) % MOD for l in range(1, k + 1): for i in range(l - 1, n): if i - l + 1 < 0: continue cnt = comb_table[i][l - 1] current_sum = (nums[i] * nums[i - l + 1]) % MOD total = (total + cnt * current_sum) % MOD return total

5

Qwen (Planner) → Qwen (Coder)

Failed

def minMaxSums(self, nums: List[int], k: int) -> int: nums.sort() n = len(nums) prefix = [0] * (n + 1) for i in range(n): prefix[i + 1] = prefix[i] + nums[i] max_n = n comb = [[0] * (max_n + 1) for _ in range(max_n + 1)] for i in range(max_n + 1): comb[i][0] = 1 comb[i][i] = 1 for j in range(1, i): comb[i][j] = (comb[i - 1][j - 1] + comb[i - 1][j]) % MOD total = 0 for length in range(1, k + 1): for i in range(n - length + 1): min_count = comb[n - i - 1][length - 1] min_sum = nums[i] * min_count max_sum = 0 for j in range(i + length - 1, n): max_sum += nums[j] * comb[j - i - 1][length - 2] max_sum %= MOD total = (total + min_sum + max_sum) % MOD for i in range(length - 1, n): max_count = comb[i][length - 1] max_sum = nums[i] * max_count min_sum = 0 for j in range(0, i - length + 1 + 1): min_sum += nums[j] * comb[i - j - 1][length - 2] min_sum %= MOD total = (total + min_sum + max_sum) % MOD return total

6

Qwen (Planner) → Ministral (Coder)

Passed

def minMaxSums(self, nums: List[int], k: int) -> int: nums.sort() n = len(nums) max_fact = n fact = [1] * (max_fact + 1) inv_fact = [1] * (max_fact + 1) for i in range(1, max_fact + 1): fact[i] = fact[i-1] * i % MOD inv_fact[max_fact] = pow(fact[max_fact], MOD-2, MOD) for i in range(max_fact - 1, -1, -1): inv_fact[i] = inv_fact[i + 1] * (i + 1) % MOD def comb(a, b): if a < 0 or b < 0 or a < b: return 0 return fact[a] * inv_fact[b] % MOD * inv_fact[a - b] % MOD total = 0 for i in range(n): num = nums[i] min_contrib = 0 for l in range(1, k + 1): if l - 1 <= i: min_contrib = (min_contrib + comb(i, l - 1)) % MOD max_contrib = 0 for l in range(1, k + 1): if l - 1 <= n - i - 1: max_contrib = (max_contrib + comb(n - i - 1, l - 1)) % MOD total = (total + num * (min_contrib + max_contrib)) % MOD return total

Fig. 4. Plans and generated code from Qwen3-8B and Ministral-3-8B-Instruct for LiveCodeBench problem 3714, comparing direct generation with four planner-coder pairings.

This case illustrates why different planner-coder combinations can exhibit different behaviors. A model that fails to generate the final code may still provide useful solution plans, while another model may be better at realizing those plans. 7 7.1

Threats to Validity Construct Validity

A potential threat is whether benchmark-based evaluation provides a reliable measurement of planner-coder interaction. Our evaluation uses functional correctness as the primary criterion, which is determined by the testing mechanism of each benchmark. LiveCodeBench evaluates generated programs against predefined test cases, where finite test coverage may miss some valid input scenarios. Code2Bench adopts property-based testing to generate diverse inputs and evaluates generated functions on the resulting test suites. However, the test generation process may introduce randomness in our evaluation setup, potentially causing the generated test inputs and correctness judgments to vary across runs. To ensure a consistent evaluation, all planner-coder configurations are evaluated on the same benchmark tasks following the original evaluation pipelines of each benchmark, including their test

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

17

data, generators, and runners without modification. In addition, evaluating on two complementary benchmarks reduces the dependence on any single testing methodology. Nevertheless, finite test coverage and potential randomness in automated testing remain inherent limitations of benchmarkbased code generation evaluation. 7.2

Internal Validity

Generation variability may affect the measured performance of a planner-coder pair. This variability can arise in both stages. Plan generation uses sampling, so the same planner can produce different plans for the same problem, which may lead to different coding outcomes. For code generation, we use greedy decoding to reduce randomness, but complete determinism is still difficult to guarantee. For API-based models, some serving details are outside our control, while local inference with vLLM may also show small run-to-run differences. We reduce the influence of these factors through a consistent evaluation protocol. For each planner and problem, we generate three independent plans, retain all of them, and average the corresponding coding results rather than selecting a favorable plan. The same plans are reused when comparing different coders, keeping the planning input fixed across coder comparisons. We also use greedy decoding throughout code generation to reduce additional variation. 7.3

External Validity

Our evaluation is based on two Python code-generation benchmarks and models from several families and scales. The observed patterns may not directly extend to other programming languages or other software-engineering tasks, such as repository-level development, debugging, or program repair. We also study a specific form of planner-coder interaction, where a natural-language solution plan is generated once and then passed to the coder. Workflows with different plan representations, execution feedback, plan revision, or repeated interaction may lead to different results. Our coder-side training experiments cover three relatively small coders using standard code instruction tuning. The observed transfer of coder improvements across planners is therefore established within this training setting, and may differ for larger coders or substantially different training approaches. 8 8.1

Related Work Planning as Reasoning

Planning has been explored as an explicit reasoning stage in code generation, where models first formulate intermediate solution information and then use it to guide code generation. Inspired by Chain-of-Thought (CoT) reasoning, Self-Planning [16] introduces explicit solution steps before implementation, while other studies explore different forms of planning information, including structured solution representations [23], hierarchical decomposition [48] and modular problem decomposition [31]. Beyond generating a single plan, later work also explores multiple candidate solutions to improve planning. PlanSearch [39] searches over diverse natural-language plans, while other approaches explore alternative solution paths [7], identify consensus across candidate plans [40], or organize candidate solutions through tree-structured exploration [2, 22]. Empirical studies [17, 49] further examine the effectiveness of intermediate solution information, showing that its quality does not always align with the correctness of generated code and that different forms of intermediate information can lead to different generation outcomes.

18

Wang et al.

This line of work mainly investigates how to generate effective plans and how planning information affects code generation. Our work shifts the focus from the quality of individual plans to the capabilities of models that generate and utilize these plans. 8.2

Planning as Interface

Plans can also serve as explicit interfaces between solution formulation and subsequent stages of code generation. Multi-agent approaches such as MapCoder [12] separate planning from implementation and pass the generated plan to a coding stage. Other systems organize planning, coding, execution, and verification into different roles or components [5, 26, 43]. Later work extends this basic plan-then-code process with iterative interaction. Some approaches verify or revise plans before implementation [21], while others use simulation [13] or iterative planning-coding collaboration [4] to refine the solution. Other systems adapt the use of planning according to generation outcomes [20, 53]. Recent studies further broaden how such interfaces are used. A study [9] uses natural-language editorials to separate problem solving from implementation in code generation. Other work explores cross-model collaboration for practical objectives: COPE [18] uses plans to coordinate small and large LLMs for efficient inference, while PaT [47] introduces planning after an unsuccessful initial generation to reduce unnecessary planning cost. These studies demonstrate the usefulness of plans as interfaces for organizing code generation workflows. However, such interfaces are typically embedded in specific systems or designed for particular objectives. Our work also treats plans as interfaces. However, our work does not use specific plans for a fixed workflow, but studies how various plans can be used across various planner-coder configurations. 8.3

Planning in Model Training

Planning-related information has also been incorporated into the training of code generation models. Inspired by reasoning distillation [35], code generation studies introduce solution plans or intermediate reasoning information as training supervision to improve code generation models [37]. COTTON [46] trains lightweight models to generate CoTs for code generation, while MapCoderLite [19] distills knowledge from multi-stage planning-and-coding workflows into smaller models. In addition to using intermediate information as direct supervision, recent studies further optimize models based on the solution process itself. ReCode [6] introduces process-level signals during training, while other approaches [45] use structured intermediate objectives to better connect solution formulation with generated code. Another related direction focuses on improving how coders utilize externally provided plans. PADA-Coder [10] explicitly trains coders to better follow generated plans. Existing approaches aim to tie model training to planning information, either by using plans as training signals or by optimizing models to utilize planning-related information. Our work differs by decoupling coder improvement from planning supervision and examining whether planner-agnostic coder-side improvements can generalize across different planners. 9

Conclusion

In this paper, we investigate whether planning and coding in LLM-based code generation should be considered as a unified capability or two separable capabilities. Our study shows that these capabilities can exhibit different strengths across models and can be independently utilized to improve code generation. This separation provides new opportunities for constructing code generation systems by combining complementary models and improving individual components without requiring joint optimization of the entire pipeline.

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

19

References [1] Wasi Uddin Ahmad, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Vahid Noroozi, Somshubra Majumdar, and Boris Ginsburg. 2025. OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs. arXiv:2504.04030. Retrieved from https://arxiv.org/abs/2504.04030. [2] Yingqian Cui, Zhenwei Dai, Pengfei He, Bing He, Hui Liu, Zhan Shi, Xianfeng Tang, Jingying Zeng, Suhang Wang, Yue Xing, Jiliang Tang, and Benoit Dumoulin. 2026. A Reward-Guided Dual-Phase Framework for Adaptive Inference-Time Reasoning. In Findings of the Association for Computational Linguistics: ACL 2026. 10506–10531. doi:10.18653/v1/2026. findings-acl.511 [3] Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion. In Advances in Neural Information Processing Systems, Vol. 36. 46701–46723. doi:10.52202/075280-2023 [4] Duy Tung Doan, Quang Huy Phung, Dzung Nguyen, and Khac-Hoai Nam Bui. 2026. CollabCoder: Plan-Code Co-Evolution via Collaborative Decision-Making for Efficient Code Generation. In Findings of the Association for Computational Linguistics: ACL 2026. 27964–27985. doi:10.18653/v1/2026.findings-acl.1393 [5] Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2024. Self-Collaboration Code Generation via ChatGPT. ACM Transactions on Software Engineering and Methodology 33, 7 (2024), 38 pages. doi:10.1145/3672459 [6] Lishui Fan, Yu Zhang, Mouxiang Chen, and Zhongxin Liu. 2026. ReCode: Reinforcing Code Generation with ReasoningProcess Rewards. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 43896–43914. doi:10.18653/v1/2026.acl-long.2031 [7] Shuzheng Gao, Chaozheng Wang, Cuiyun Gao, and Michael R. Lyu. 2026. SEER: Enhancing Chain-of-Thought Code Generation Through Self-Exploring Deep Reasoning. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26). 233–245. doi:10.1145/3744916.3773197 [8] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yifan Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming—The Rise of Code Intelligence. arXiv:2401.14196. Retrieved from https://arxiv.org/abs/2401.14196. [9] Sama Hadhoud, Alaa Elsetohy, Frederikus Hudi, Jan Christian Blaise Cruz, Steven Halim, and Alham Fikri Aji. 2026. Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming. In Findings of the Association for Computational Linguistics: ACL 2026. 35708–35747. doi:10.18653/v1/ 2026.findings-acl.1778 [10] Yihong Huang, Ke Qin, Rongzheng Wang, Muquan Li, Jiakai Li, Xiurui Xie, and Shuang Liang. 2026. PADA-Coder: Improving Plan-Following Code Generation via Perturbation-Verified Attention Distillation and Dynamic Alignment. In Proceedings of the 43rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 306). 47478–47505. [11] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186. Retrieved from https://arxiv.org/abs/2409.12186. [12] Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 4912–4944. doi:10.18653/v1/2024.acl-long.269 [13] Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2025. CodeSim: Multi-Agent Code Generation and Problem Solving Through Simulation-Driven Planning and Debugging. In Findings of the Association for Computational Linguistics: NAACL 2025. 5128–5154. doi:10.18653/v1/2025.findings-naacl.285 [14] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In International Conference on Learning Representations. 58791–58831. [15] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A Survey on Large Language Models for Code Generation. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 72 pages. doi:10.1145/3747588 [16] Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024. Self-Planning Code Generation with Large Language Models. ACM Transactions on Software Engineering and Methodology 33, 7 (2024), 30 pages. doi:10.1145/3672456 [17] Naizhu Jin, Zhong Li, Guang Yang, Tian Zhang, and Qingkai Zeng. 2027. Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis. Empirical Software Engineering 32 (2027). doi:10.1007/s10664-026-10947-8 [18] Byeongchan Lee, Jonghoon Lee, Dongyoung Kim, Jaehyung Kim, Kyungjoon Park, Dongjun Lee, and Jinwoo Shin. 2026. Efficient LLM Collaboration via Planning. Transactions on Machine Learning Research (2026). [19] Woongkyu Lee, Junhee Cho, and Jungwook Choi. 2026. MapCoder-Lite: Distilling Multi-Agent Coding into a Single Small LLM. In Findings of the Association for Computational Linguistics: EACL 2026. 6569–6596. doi:10.18653/v1/2026.

20

Wang et al.

findings-eacl.346 [20] Woojin Lee and Jin-Xia Huang. 2026. SolidCoder: Bridging the Mental-Reality Gap in LLM Code Generation Through Concrete Execution. In Findings of the Association for Computational Linguistics: ACL 2026. Association for Computational Linguistics, 7294–7316. doi:10.18653/v1/2026.findings-acl.361 [21] Chao Lei, Yanchuan Chang, Nir Lipovetzky, and Krista A. Ehinger. 2025. Planning-Driven Programming: A Large Language Model Programming Workflow. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 12647–12684. doi:10.18653/v1/2025.acl-long.621 [22] Jierui Li, Hung Le, Yingbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. 2025. CodeTree: Agent-Guided Tree Search for Code Generation with Large Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 3711–3726. doi:10.18653/v1/2025.naacl-long.189 [23] Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2025. Structured Chain-of-Thought Prompting for Code Generation. ACM Transactions on Software Engineering and Methodology 34, 2 (2025), 23 pages. doi:10.1145/3690635 [24] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-Level Code Generation with AlphaCode. Science 378, 6624 (2022), 1092–1097. doi:10.1126/science.abq1158 [25] Qingyuan Liang, Zhao Zhang, Chen Liu, Zeyu Sun, Wenjie Zhang, Yizhou Chen, Zixiao Zhao, Qi Luo, Wentao Wang, Yanjie Jiang, Yingfei Xiong, and Lu Zhang. 2025. Condor: A Code Discriminator Integrating General Semantics With Code Details. IEEE Transactions on Software Engineering 51, 12 (2025), 3524–3539. doi:10.1109/TSE.2025.3620145 [26] Feng Lin, Dong Jae Kim, and Tse-Hsun Chen. 2025. SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE). 1527–1539. doi:10.1109/ICSE55347.2025.00140 [27] Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. 2026. Ministral 3. arXiv:2601.08584. Retrieved from https://arxiv.org/abs/2601.08584. [28] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems, Vol. 36. 21558–21572. doi:10.52202/075280-0943 [29] Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. RepoBench: Benchmarking Repository-Level Code AutoCompletion Systems. In International Conference on Learning Representations. 47832–47850. [30] OpenAI. 2026. GPT-5.6 Sol. Retrieved October 1, 2026 from https://developers.openai.com/api/docs/models/gpt-5.6-sol [31] Ruwei Pan and Hongyu Zhang. 2026. MoT: Modularization-of-Thought Prompting for Effective Code Generation. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). [32] Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. COFFE: A Code Efficiency Benchmark for Code Generation. Proceedings of the ACM on Software Engineering 2, FSE (2025), 24 pages. doi:10.1145/3715727 [33] Qwen. 2026. Qwen3.5-27B. Retrieved October 1, 2026 from https://huggingface.co/Qwen/Qwen3.5-27B [34] Qwen. 2026. Qwen3.8-27B. Retrieved October 1, 2026 from https://huggingface.co/Qwen/Qwen3.8-27B [35] Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling Reasoning Capabilities into Smaller Language Models. In Findings of the Association for Computational Linguistics: ACL 2023. 7059–7073. doi:10.18653/v1/ 2023.findings-acl.441 [36] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, Hanjie Chen, and Xia Hu. 2025. Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models. Transactions on Machine Learning Research (2025). [37] Zhihong Sun, Chen Lyu, Bolun Li, Yao Wan, Hongyu Zhang, Ge Li, and Zhi Jin. 2024. Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 5878–5895. [38] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 Technical Report. arXiv:2503.19786. Retrieved from https://arxiv.org/abs/2503.19786. [39] Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. 2025. Planning in Natural Language Improves LLM Search for Code Generation. In International Conference on Learning Representations. [40] Chao Wen, Jie Liu, and Liang Du. 2026. Consensus Planning Boosts LLM Code Generation. Information and Software Technology 192 (2026), 108030. doi:10.1016/j.infsof.2026.108030 [41] Anyi Xu, B Li, Bangcai Lin, Bing Xue, BingCheng Xian, Bingzheng Xu, Bochao Wu, Bowei Zhang, Boyi Deng, CC Yu, et al. 2026. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. arXiv:2609.19969. Retrieved from https://arxiv.org/abs/2609.19969.

Planning-Coding Separation in LLM-Based Code Generation: An Empirical Study

21

[42] Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348. Retrieved from https://arxiv.org/abs/2606.19348. [43] ZhenChun Xu, Yi Cai, Dajun Zheng, li Yuan, Mengchen Zhao, Qixiang Wang, and Jiexin Wang. 2026. MavenCoder: Competitive Code Generation via Model Adaptive Planning Strategies and Multi-Perspective Verification Enhancement. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 30415–30439. doi:10.18653/v1/2026.acl-long.1403 [44] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 Technical Report. arXiv:2505.09388. Retrieved from https://arxiv.org/abs/2505.09388. [45] Chengran Yang, Zhensu Sun, Hong Jin Kang, Jieke Shi, and David Lo. 2026. Think Like Human Developers: Harnessing Community Knowledge for Structured Code Reasoning. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE ’26). 580–591. doi:10.1145/3744916.3764550 [46] Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Terry Yue Zhuo, and Taolue Chen. 2024. Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models. IEEE Transactions on Software Engineering 50, 9 (2024), 2437–2457. doi:10.1109/TSE.2024.3440503 [47] Youngsik Yoon, Sungjae Lee, Seockbean Song, Siwei Wang, Wei Chen, and Jungseul Ok. 2026. PaT: Planning-after-Trial for Efficient Test-Time Code Generation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 36738–36755. doi:10.18653/v1/2026.acl-long.1703 [48] Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D. Goodman, and Nick Haber. 2023. Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions. In Advances in Neural Information Processing Systems, Vol. 36. 31466–31523. doi:10.52202/075280-1367 [49] Binquan Zhang, Li Zhang, Zhiwen Luo, Yuxin Du, Fang Liu, Song Wang, and Lin Shi. 2026. Are They All Good? Evaluating the Quality of CoTs in LLM-Based Code Generation. IEEE Transactions on Software Engineering 52, 6 (2026), 1928–1943. doi:10.1109/TSE.2026.3676295 [50] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2471–2484. doi:10.18653/v1/2023.emnlp-main.151 [51] Zhe Zhang, Runlin Liu, Aishan Liu, Xingyu Liu, Xiang Gao, and Hailong Sun. 2026. Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction. In International Conference on Learning Representations. 153348–153391. [52] Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. 2025. HumanEvo: An Evolution-Aware Benchmark for More Realistic Evaluation of Repository-Level Code Generation. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE). 1372–1384. doi:10.1109/ICSE55347.2025. 00228 [53] Yueheng Zhu, Chao Liu, Xuan He, Xiaoxue Ren, Zhongxin Liu, Ruwei Pan, and Hongyu Zhang. 2026. AdaCoder: An Adaptive Planning and Multi-Agent Framework for Function-Level Code Generation. IEEE Transactions on Software Engineering 52, 2 (2026), 631–650. doi:10.1109/TSE.2025.3642621 [54] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In International Conference on Learning Representations. 66602–66656.

Related documents

• Software Engineering LLMs: Why We Need Them and How Software Engineering Helps Build Them#1456674
• CodeArena: A Collective Evaluation Platform for LLM Code Generation#1490348
• Pseudo2CodeQA: A Benchmark for LLM-Based Structured Algorithmic Reasoning in Code Generation#441873
• CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation#429448
• Exploring the Potential of Program Flowcharts on Code Generation Using Multimodal LLMs#361521