arXiv CS open access

Asymmetric Capacity Allocation in Self-Refinement Pipelines

2026 · arxiv_cs

arXiv CS · Papers · 2026

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Asymmetric Capacity Allocation in Self-Refinement Pipelines Zhuoyi Yang1 Ian G. Harris1 Salar Hashemitaheri1 Cassie Huang2 Yuangang Li1 Hyunwoo Oh1 Paul Dourish1 Tony Givargis1 Mohsen Imani1 Li Zhang2 1

University of California, Irvine 2 Drexel University {zhuoyy1, iharris, salarh, yuanganl, hyunwooo, givargis, mohseni}@uci.edu jpd@ics.uci.edu {Cassie.Huang, Harry.Zhang}@drexel.edu Abstract

et al., 2024; Zhou et al., 2025; Chen et al., 2023b). More broadly, similar refinement loops are now embedded within many agentic workflows, where agents repeatedly evaluate intermediate outputs before taking subsequent actions (Zhang et al., 2026; Wu et al., 2023; Yao et al., 2023). Existing work typically selects model sizes heuristically or uniformly scales the entire pipeline, without examining whether all stages benefit equally from increased model capacity (Madaan et al., 2023; Shinn et al., 2023; Gou et al., 2024). Such design likely leads to a waste of resources. The decomposition of generator-critic-refiner is also consistent with classical cognitive models that distinguish production, evaluation, and revision as separate cognitive processes (Flower and Hayes, 1980; Hayes, 1996; Kellogg, 1996). The three stages perform fundamentally different functions but model capacities in the pipeline have not been treated as a subject of systematic investigation. This raises a fundamental question: How should model capacity be allocated across the stages of a self-refinement pipeline? We address this question through a controlled stage-wise model size analysis on the selfrefinement pipeline (Figure 1). We independently vary the size of the generator, critic, and refiner while holding the other two stages fixed (Figure 2). This experimental design isolates the contribution of each stage and enables a direct comparison of their sensitivity to model sizes across 5 benchmarks and 2 model families (Qwen3 (Yang et al., 2025) and Gemma 3 (Team et al., 2025)). We draw two main conclusions. First, self-refinement performance is highly sensitive to the capacities of the generator and refiner. Scaling either stage generally improves overall performance, whereas an undersized refiner can even degrade performance below that of the initial generation. Second, pipeline performance is comparatively insensitive to critic size. Although larger critics provide only marginal gains,

arXiv:2608.21345v1 [cs.LG] 21 Aug 2026

Self-refinement, typically structured as generation, critique, and revision, is a widely adopted paradigm for improving LLM generation and serves as a core mechanism in many LLM agents. While the three stages involve different cognitive demands, most existing approaches conveniently treat the model size as an implementation detail rather than a subject of study, which may lead to a waste of resources. Little work has systematically examined how model size affects each stage or whether effective selfrefinement requires equally capable models for generation, critique, and revision. We present the first stage-wise model size study of the selfrefinement pipeline on 5 benchmarks from different domains using 6 model sizes of Qwen3 and 4 model sizes of Gemma 3. We conclude that larger generators and refiners generally improve the pipeline, whereas an undersized refiner can even harm performance. Second, performance is highly insensitive to the size of the critic, although including even a small critic consistently outperforms omitting critique altogether. Our findings demonstrate that model capacity should not be allocated uniformly across self-refinement pipelines. Instead, different stages exhibit distinct size scaling characteristics, providing practical guidance for designing more computationally efficient multi-stage language model systems.

Introduction

Large language models increasingly rely on iterative reasoning rather than one-shot generation. A common paradigm is self-refinement (Madaan et al., 2023), in which a model first produces a solution, critiques its own output, and then revises the solution based on the generated feedback. This generator–critic–refiner pipeline has become a fundamental component of modern LLM systems, improving performance on reasoning, planning, coding, and generation tasks (Shinn et al., 2023; Gou 1

Input x

1. Generator (G)

2. Critic (C)

3. Refiner (R)

Generate initial

Critique the

Revise the

Evaluation

solution

Example: Goal: Schedule the maximum number of meetings. Input: Ronald: Chinatown 9:30am-11:30am >= 75 mins ...

solution

solution Feedback f

Initial solution y0

Final solution yr

Ronald: The meeting with Ronald lasts Ronald: 9:30am-10:30am only 60 minutes, but the required 9:30am-10:45am (60 mins)... duration is 75 minutes. ... (75 mins) ...

Figure 1: Overview of our three-stage self-refinement pipeline with a simplified example. Given an input x, the generator produces an initial solution y0 , the critic provides natural-language feedback f on x and y0 , and the refiner revises the solution based on x, y0 , and f .

even the smallest critic consistently outperforms a pipeline without explicit critique. Our findings show that the three stages exhibit markedly different scaling characteristics, indicating that uniform model allocation is suboptimal. These results provide practical guidance for designing computationally efficient self-refinement pipelines.

treating the model assigned to each stage as fixed. A separate line of research investigates how model capacity and computational budget influence language model performance. Classical scaling law studies characterize how performance improves with increasing model size, training data, and compute, providing principles for efficient model development (Kaplan et al., 2020; Hoffmann et al., 2022). More recently, researchers have shifted attention from training-time scaling to inference-time scaling, showing that allocating additional computation during reasoning can substantially improve performance without increasing model parameters. Representative approaches include deliberate reasoning through longer inference trajectories (OpenAI et al., 2026; Guo et al., 2025), self-guided reasoning and internal thought generation (Zelikman et al., 2024), and test-time compute scaling strategies that trade additional inference computation for higher accuracy (Muennighoff et al., 2025; Tarunokusumo and Cunha, 2025).

Related Works

Self-refinement has become a widely adopted paradigm for improving large language models at inference time. Early work such as SELF-REFINE demonstrated that a single language model can iteratively generate feedback and refine its own outputs without additional training (Madaan et al., 2023). Since then, numerous approaches have extended this paradigm by incorporating stronger critics, external verification, execution feedback, retrieval, and tool use to improve reasoning, planning, coding, and text generation (Shinn et al., 2023; Zhou et al., 2025; Chen et al., 2024, 2023b). More recently, the generate–critique–refine loop has become one of the core reasoning primitives underlying many contemporary agentic systems. (Yao et al., 2023; Kim et al., 2024; Wu et al., 2023). Recent surveys likewise identify reflection and selfcorrection as central cognitive components of modern agent architectures (Zhang et al., 2026; Huang et al., 2024; Zhang et al., 2025). Despite the diversity of refinement strategies, existing work primarily focuses on designing more effective feedback mechanisms or refinement algorithms to maximize downstream task performance, while typically

Complementary work studies how inference computation should be allocated, including LLM routing (Ong et al., 2025), model cascading (Chen et al., 2023a), adaptive model selection (Ding et al., 2025), and mixture-of-experts routing (Fedus et al., 2022; Lepikhin et al., 2020), which dynamically select models or experts according to input difficulty or computational budget. While these methods determine how much computation to spend on each query or which model should answer it, they generally treat inference as a single prediction stage. In contrast, we study how model capacity should be 2

distributed across the interacting stages of a multistage self-refinement pipeline. Although self-refinement pipelines and model size scaling have each been studied extensively, their intersection remains largely unexplored. To the best of our knowledge, no prior work has systematically investigated how model capacity should be allocated across the generator, critic, and refiner in a self-refinement pipeline. We address this gap by conducting the first systematic stage-wise model size analysis of self-refinement pipelines.

Generator (G)

Refiner (R)

Generator (G)

Critic (C)

Refiner (R)

Generator (G)

Critic (C)

Varying

Varying Refiner (R) Varying Refiner (R)

Generator (G)

Figure 2: Stage-wise model size analysis. The three panels illustrate the generator, critic, and refiner sweeps. The highlighted stage is the stage under evaluation, whose model size is varied while the other two stages are held fixed. We also compare against a no-critique baseline.

Method

This section describes our experimental framework for analyzing model capacity in self-refinement pipelines. We first formalize the generator–critic– refiner pipeline, then present our stage-wise model size analysis protocol. 3.1

Critic (C)

3.2

Stage-wise Model Size Analysis Protocol

Our objective is to quantify the contribution of model size at each stage of the self-refinement pipeline. To isolate the effect of an individual stage, we vary exactly one stage while keeping the remaining two stages fixed to the same models. This controlled design ensures that any performance differences can be attributed solely to the stage being varied (Figure 2). Specifically, we perform three stage-wise model size analysis experiments:

Self-Refinement Pipeline

Our self-refinement framework consists of three sequential stages: a generator, a critic, and a refiner, following the SELF-REFINE paradigm proposed by Madaan et al. (2023). Given an input x, the generator first produces an initial solution y0 = G(x), where G denotes the generator. The critic then analyzes the generated solution and produces natural-language critique

• Generator sweep: vary the generator size while fixing the critic and refiner at the same size, P (Gi , C, R),

f = C(x, y0 ), where C denotes the critic. f includes potential errors and actionable critique. Finally, the refiner generates the final solution by applying the critique to the original solution,

where Gi denotes the i-th generator model. • Critic sweep: vary the critic size while fixing the generator and refiner at the same size, P (G, Ci , R),

yr = R(x, y0 , f ),

where Ci denotes the i-th critic.

where R denotes the refinement model. We denote a complete self-refinement pipeline by

• Refiner sweep: vary the refiner size while fixing the generator and critic at the same size, P (G, C, Ri ),

P (G, C, R),

where Ri denotes the i-th refiner.

where G, C, and R specify the models used for the generator, critic, and refiner. For example,

We repeat this procedure for every benchmark and every model family. For example, under the Qwen3-32B configuration, the generator sweep evaluates P(0.6B, 32B, 32B), P(1.7B, 32B, 32B), P(4B, 32B, 32B), P(8B, 32B, 32B), P(14B, 32B, 32B), and P(32B, 32B, 32B).

P (32B, 0.6B, 32B) represents a pipeline using a 32B generator, a 0.6B critic, and a 32B refiner. 3

3.3

No-Critique Baseline

maintaining consistent architectures and training paradigms, making them suitable for studying the effect of model capacity across different stages of self-refinement. For the Qwen3 family, we consider models with 0.6B, 1.7B, 4B, 8B, 14B, and 32B parameters. To examine whether our findings depend on the maximum available model capacity, we conduct three complete stage-wise scaling analyses using Qwen332B, Qwen3-14B, and Qwen3-8B as the largest models in the pipeline for the fixed stages. Unless otherwise specified, the main paper reports results from the Qwen3-32B configuration. The Qwen314B and Qwen3-8B experiments are presented in Appendix D and Appendix E as robustness studies. To further validate the generality of our observations, we repeat the analysis on the Gemma 3 family, using Gemma 3-1B-IT, Gemma 3-4BIT, Gemma 3-12B-IT, and Gemma 3-27B-IT. The corresponding results are reported in Appendix C and exhibit the same qualitative stage-wise scaling trends observed for Qwen3.

To determine whether explicit critique provides benefits beyond an additional refinement pass, we compare the standard self-refinement pipeline against a matched no-critique baseline. The nocritique pipeline is denoted by P (G, ∅, R), where the critic is removed and the refiner directly revises the generator output without receiving feedback. To isolate the effect of introducing critique, we fix the critic to the smallest model in the Qwen family (Qwen3-0.6B) and compare it against the corresponding no-critique pipeline. For the generator sweep, the critique gain is defined as ∆gen critique (Gi ) = P (Gi , 0.6B, R) − P (Gi , ∅, R), Similarly, for the refiner sweep,

4.2

∆ref critique (Ri ) = P (G, 0.6B, Ri ) − P (G, ∅, Ri ),

We evaluate stage-wise model size analysis on subsets of five diverse benchmarks spanning planning, summarization, logical reasoning, code optimization, and story generation. These tasks cover a broad range of reasoning and generation capabilities, allowing us to assess whether our findings generalize across heterogeneous application domains rather than being specific to a single task. Each benchmark is evaluated using its distinct taskspecific metric. We put the subset details in Appendix B.

where only the refiner varies while the generator remains fixed. For each benchmark, we report the average critique gain across the generator and refiner sweeps. By fixing the critic to Qwen3-0.6B, this comparison measures whether even a lightweight critic provides benefits beyond an additional refinement pass.

Experimental Setup

We conduct stage-wise model size analysis across five benchmarks spanning planning, summarization, logical reasoning, code optimization, and open-ended story-writing. Experiments are conducted with 2 model families (Qwen3 and Gemma 3) and 5 configurations (Qwen3-32B, Qwen3-14B, Qwen3-8B, Gemma 3-27B, and the no-critique baseline). For each benchmark, we independently vary the model assigned to one pipeline stage while holding the other two stages fixed, following the protocol described in Section 3.2. 4.1

Benchmarks and Evaluation

Meeting Planning (Zheng et al., 2024). In this task, the model must generate a feasible itinerary that satisfies a set of temporal and travel constraints. We evaluate planning performance using planning accuracy, defined as the percentage of generated itineraries that satisfy all constraints. An itinerary is considered correct only if it is fully feasible; any violation of the specified constraints results in a score of zero for that instance. CNN/DailyMail (Hermann et al., 2015; See et al., 2017). In this task, the model must generate a concise summary that captures the key information from the original news article. We evaluate performance using coverage rate, defined as the percentage of reference highlight sentences whose information is covered by the generated summary.

Models

We evaluate our stage-wise scaling analysis using two families of open-weight large language models: Qwen3 and Gemma 3. These model families provide a diverse range of parameter scales while 4

Task Goal: Schedule the maximum number of meetings. Input: Ronald: Chinatown 9:30am11:30am >= 75 mins ... Goal: Summarize the article. Article:"A severe storm disrupted rail services..."

Initial Generation

Goal: Optimize code while preserving correctness.

Refined Solution

Ronald: 9:30am-10:30am (60 mins)...

The meeting with Ronald lasts only 60 minutes, but the required duration is 75 minutes.

Ronald: 9:30am-10:45am (75 mins) ...

A storm affacted transportation in England. ...

The summary omits that rail routes remained closed while crews cleared fallen trees. ...

A storm disrupted rail services across northern England, leaving several routes closed as crews removed fallen trees. ...

House1

House 2

Name

Arnold

Eric

Smoke

Pall Mall

Prince

Pet

Cat

Dog

Goal: Complete the puzzle. Puzzle: House 1 | House 2 ...

Critique

House1

Eric must be assigned Pall Mall. And the Prince smoker owns a cat.

for i in range(2, n //2): ...

Only check up to √n.This will reduce the number of divisibility checks.

A dusty photograph lay on the desk.

The sentence didn't cover any of the keywords, and it doesn't connect the sentence smoothly.

House 2

Name

Arnold

Eric

Smoke

Prince

Pall Mall

Pet

Cat

Dog

for i in range(2, int(n**0.5) + 1): ...

Input : for i in range(2, n): ... Goal: Write a sentence using the given keywords that fits the story. Input: Maya found an unopened envelope beneath the old wooden door...

Relieved, Maya picked up the letter from the doorway, finally understanding why it had been hidden there.

Keywords: relieved, doorway, letter

Figure 3: Representative examples from the five benchmarks. The first column briefly illustrates the objective and input for each task, followed by the initial generation, the generated critique, and the final refined solution, demonstrating how the self-refinement pipeline operates across diverse domains.

ZebraLogic (Lin et al., 2025). In this task, the model must infer a complete logic grid puzzle that satisfies a set of logical constraints. We report cell accuracy, which measures the percentage of correctly assigned attribute values across all cells in the completed solution grid.

Benchmark Meeting Planning CNN/DailyMail ZebraLogic PIE CollaboSentGen

PIE (Shypula et al., 2024). In this task, the model must optimize a program to improve its efficiency while preserving its original functionality. We evaluate performance using %OPT, defined as the percentage of generated programs that are both functionally correct and successfully optimized.

Model Qwen Gemma Qwen Gemma Qwen Gemma Qwen Gemma Qwen Gemma

Generator ∆ Std. 16.00 6.15 24.00 8.76 10.00 3.95 25.10 9.62 8.65 3.07 27.60 10.71 28.00 10.43 37.50 14.79 2.79 0.96 2.46 1.038

Critic ∆ Std. 9.00 3.10 8.00 2.92 8.00 3.10 6.30 2.24 3.89 1.46 5.80 2.31 4.00 1.63 5.00 1.92 0.56 0.20 0.63 0.26

Refiner ∆ Std. 32.00 11.20 34.00 12.64 23.40 8.99 16.80 6.39 13.39 4.71 17.50 6.71 50.00 20.60 17.00 6.50 5.61 2.03 4.36 1.92

Table 1: Performance variability across model sizes when varying one stage while fixing the other two at the largest model. Values are reported in each benchmark’s evaluation metric, as specified in Section 4.2.

within a sweep, and the standard deviation (σ) of the performance scores. Larger values of ∆ and σ indicate that pipeline performance is more sensitive to model size increase.

CollaboSentGen. This is a benchmark derived from the ROCStories corpus (Mostafazadeh et al., 2016), where one of the middle three sentences is removed from a story and a set of keywords from the missing sentence is provided. In this task, the model must generate a sentence using a given set of keywords that coherently fits into an existing story. We evaluate performance using Overall-Story-Fit, where GPT-4 (OpenAI et al., 2024) serves as the automatic evaluator to assess how well the generated sentence integrates with the surrounding narrative.

4.3

Prompting Strategy

All three stages use task-specific few-shot instruction prompts for format and structure alignment (Brown et al., 2020). The complete prompts used for each benchmark are provided in Appendix G. Figure 3 illustrates the self-refinement pipeline on all five benchmarks before the quantitative analysis. These examples provide intuition for the different types of tasks considered in our evaluation before we analyze stage-wise scaling quantitatively.

Stage-wise Sensitivity Metrics. To quantify the sensitivity of each pipeline stage to model size, we summarize each stage-wise sweep using two statistics: the performance range (∆), defined as the difference between the highest and lowest scores 5

(a) Meeting Planning

(b) CNN/DailyMail

0.8

0.7 0.7

0.6

0.6 0.6

0.5

0.4

0.5

Task-specific evaluation metric

0.4 0.2

0.4

0.3 0.6B

1.7B

14B

32B

0.2

0.6B

(d) PIE

1.7B

14B

32B

0.3

0.6B

1.7B

14B

32B

(e) CollaboSentGen

0.7

0.6

0.5

0.4

0.3

0.2

4 0.6B

1.7B

14B

32B

Initial generation Generator Sweep Critic Sweep Refiner Sweep 0.6B

1.7B

14B

32B

Model size of the stage under evaluation

Figure 4: Stage-wise model-size sweeps across five benchmarks under the Qwen3-32B configuration. X-axis is the model sizes. Blue line, green line, and the red line represent Generator sweep, Critic sweep and the Refiner sweep. The grey dashed line reports the corresponding initial-generation performance without refinement. Steeper curves indicate greater sensitivity to the model size of that stage. Metrics are defined in Section 4.2.

Input

Start: The Castro (9:00 AM) Ronald: Chinatown (9:30–11:30, 75 min) Nancy: Nob Hill (11:15–12:00, 30 min)

Generated

09:30–10:30 Ronald 10:50–11:20 Nancy

0.6B Critic (Score 4) 32B Critic (Score 5)

the performance of the corresponding stage is less sensitive to the increased size of the model. A consistent trend across all benchmarks is that scaling the size of the generator and the refiner substantially enhances the performance, which is visualized by the steeper lines in Figure 4 and quantitatively summarized in Table 1. In contrast, the critic stage is relatively insensitive to model size, as larger critic models provide only marginal gains over smaller ones. Results for the Gemma 3-27B, Qwen3-14B, and Qwen3-8B configurations are provided in Appendix C, D, and E. They exhibit the same qualitative trends.

Detects: Ronald’s meeting is only 60 min (requires 75 min). Detects: Ronald’s meeting is only 60 min. Detects: Nancy is scheduled outside her availability window.

Table 2: Representative example illustrating performance of 0.6B critic and 32B critic.

5.1

Model Size Analysis of The Generator

Finding 1. Pipeline performance improves consistently with generator size scaling. Figure 4 shows that increasing the generator size consistently improves end-to-end pipeline performance across all five benchmarks. The same trend is reproduced in Figure 7, 8, and 9, using different model configurations. We quantify this trend using the range (∆) and standard deviation (Std.) of performance across model sizes for each pipeline stage. The resulting statistics are summarized in Table 1. Under

Results

Figure 4 shows the effect of increasing model size at each stage independently while fixing the other two stages at Qwen3-32B. Each data point represents one pipeline. The blue line represents the trend of Generator sweep, the green line represents that of the Critic sweep, and the red line represents that of the Refiner sweep (defined in Section 3.2). The gray dashed line represents generation without any refinement. Visually, a flatter line means 6

Score 1

Score 2

32B 0.6B

32B 0.6B 32B 0.6B

10 16

32B 0.6B

20 20 20

34 16

44 40

14 60

20 20

Score 4

40 20

Score 3

Score 5

Benchmark

82% 72% 96% 94% 10 70% 10 64% 12 76% 66% 32 96% 24 92%

36 50 40

40 60 80 Percentage of critiques

Meeting Planning CNN/DailyMail ZebraLogic PIE CollaboSentGen

Refiner Sweep

+9.67 +4.56 +7.97 +9.03 +3.71

+10.34 +5.21 +7.86 +11.06 +4.01

Table 3: Average critique gain with Qwen3-0.6B as the critic over the matched no-critique baseline across the generator and refiner sweeps.

ual evaluation. Specifically, we randomly sample 50 critiques from the 0.6B and 32B critique models respectively from all five benchmarks. Following prior work on rubric-based evaluation (Rao and Callison-Burch, 2026), we design a task-specific five-point rubric to assess critique quality: A score of 5 denotes a completely correct, comprehensive, and well-justified critique; 4 indicates a correct critique with some omissions; 3 denotes an imperfect but non-misleading critique; 2 indicates partially misleading feedback containing at least one incorrect recommendation; and 1 denotes fundamentally incorrect or misleading feedback. We denote scores of 5 and 4 as correct, 3 as benign, and 2 and 1 as misleading. Figure 5 shows that the 0.6B and 32B critics rarely produce critiques with substantially different quality. When the 32B critic generates a correct critique, the 0.6B critic typically also produces a nonmisleading critique (scores 3–5), although it may miss some errors. For example, on ZebraLogic, 70% of the critiques produced by the 32B critic fall under scores 3-5 and 64% of the critiques produced by 0.6B critic fall under scores 3-5. Conversely, when the 32B critic produces misleading feedback (scores 1–2), the 0.6B critique is also usually misleading. The primary benefit of the larger critic is therefore detecting additional errors, rather than fundamentally changing whether the feedback is correct or incorrect. Table 2 is a representative example. 32B model detects more errors than 0.6B model but they are both providing correct feedback. Although the 32B critic often provides more complete feedback, the corresponding improvement in refinement performance remains modest (Figure 4). This suggests that the refiner does not always translate the additional critique information into better revisions, providing evidence for the second explanation as well.

100

Figure 5: Distribution of manual critique-quality scores across five benchmarks. For each benchmark (top to bottom: Meeting Planning, CNN/DailyMail, ZebraLogic, PIE, and CollaboSentGen): Stacked segments show the percentage of critiques assigned scores 1–5, while the percentages on the right report the fraction of nonmisleading critiques (scores 3–5).

the Qwen3 family, the generator exhibits standard deviations ranging from 0.96 on CollaboSentGen to 10.43 percentage points on PIE, indicating that generator scaling substantially affects pipeline performance. 5.2

Generator Sweep

Model Size Analysis of The Critic

Finding 2.1 Pipeline performance is largely insensitive to critic size. This behavior is visually evident in Figure 4, where the critic curves are consistently flatter than those of the generator and refiner. We further quantify this trend by computing the range (∆) and standard deviation (Std.) of performance across model sizes for each stage (Table 1). Across all benchmarks and both model families, the critic consistently exhibits the lowest variability. For example, its standard deviation remains below 0.21 on CollaboSentGen and below 3.1 percentage points on all the other benchmarks, whereas generator and refiner show substantially larger variation. These results indicate that scaling the critic model size yields comparatively limited improvements in end-to-end performance. To better understand this phenomenon, we investigate two possible explanations: (1) Critique quality changes only slightly across models, or (2) Even when larger critics produce higher-quality critiques, the refinement model may not fully utilize the additional information. We first examine the quality of the critiques themselves through a man-

Finding 2.2 Even the smallest critic outperforms no-critique pipeline To determine whether the 7

House 1

House 2

Arnold

Eric

✔

tial performance. Under the Qwen3-32B configuration, 12 of the 30 evaluated refiner pipelines (5 benchmarks × 6 refiner sizes) perform worse than the corresponding initial generation. The complete pipelines of all degraded configurations are in Appendix F. To understand the underlying failure mode, we manually analyzed 50 degradation events from the most extreme case, P(32B,32B,0.6B), where the weakest refiner exhibits the greatest performance drop. In all 50 events, the 0.6B refiner produced a worse output than both the initial generation and the 32B refiner under the same initial solution and critique. Among the 50 degradation events, 41 occur despite receiving non-misleading critiques (human scores 3–5). In these cases, the 0.6B refiner unnecessarily modifies correct portions of the original solution while attempting to apply the critique, whereas the 32B refiner generally preserves correct content and performs only the required edits. The remaining 9 cases involve misleading critiques (scores 1–2). The 0.6B refiner follows the incorrect feedback, propagating critique errors into the final solution, while the 32B refiner ignores the misleading suggestions and preserves the original solution. Figure 6 presents a representative degradation example. Both refiners receive the same critique, yet they respond differently. The 0.6B refiner unnecessarily modifies previously correct assignments, introducing new errors beyond those identified in the critique. In contrast, the 32B refiner performs only the required edits and preserves the remaining solution. This example reflects the dominant failure mode observed during manual analysis.

✔ Prince✘ Pall Mall✘

0.6B refiner

House 2 smoker should be Prince.

32B refiner

✔ House 1

House 2

Eric ✘ Arnold ✘ Prince ✘ Pall Mall✘

House 1

House 2

Arnold Eric ✔ ✔ Pall Mall✔ Prince ✔

Figure 6: A degradation example from ZebraLogic. Despite a correct critique, the 0.6B refiner is introducing new errors while the 32B refiner is incorporating the critique.

improvements of self-refinement arise from the critique itself or simply from performing an additional inference step, we compare the standard pipeline P(G, 0.6B, R) with the smallest critic to a matched no-critique baseline P(G, ∅, R), in which the refiner directly revises the generator output without receiving critique. We observe that even incorporating the smallest explicit critic would yield improvements over the initial generation across all five benchmarks, as shown in Table 3. On average, using Qwen-0.6b critic improves performance by 9.67 and 10.34 percentage points on Meeting Planning benchmark for the generator and refiner sweeps. We observe similar gains on the remaining benchmarks. These results demonstrate that the benefit of self-refinement cannot be attributed solely to additional computation. Instead, explicit critique provides complementary guidance that enables the refiner to produce higher-quality outputs. 5.3

Model Size Analysis of The Refiner

Conclusion

We presented the first stage-wise model size analysis of self-refinement pipelines. Across 5 benchmarks, 2 model families, and 4 configurations, we showed that pipeline performance is highly sensitive to the capacities of the generator and refiner, but comparatively insensitive to the critic size. While larger critics provide only modest gains, even a lightweight critic consistently outperforms a no-critique pipeline. In the meantime, a weak refiner can hurt the initial performance. These findings suggest that model capacity should be allocated non-uniformly across self-refinement pipelines, providing practical guidance for designing more compute-efficient multi-stage language model systems and LLM agentic systems.

Finding 3.1 Pipeline performance is highly sensitive to refiner size. The steeper lines of the refiner in Figure 4 show that across all five benchmarks, the refiner consistently exhibits substantially greater variability than the generator and critic. This observation is quantified in Table 1. For example, under the Qwen3 family, the refiner reaches a standard deviation of 20.60 percentage points on PIE compared to only 1.63 percentage points for the critic, 11.20 percentage points versus 3.10 percentage points on Meeting Planning, and 8.99 percentage points versus 3.10 percentage points on CNN/DailyMail. Finding 3.2. Weak refiners can degrade ini8

Limitations

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. Critic: Large language models can selfcorrect with tool-interactive critiquing. Preprint, arXiv:2305.11738.

This work analyzes stage-wise model scaling in the canonical generate–critique–refine pipeline with a single refinement iteration. Our findings may not directly generalize to more complex agentic systems that incorporate additional components such as retrieval, planning, tool use, memory, or multiple refinement rounds. Furthermore, although we evaluate two model families (Qwen3 and Gemma 3) across five diverse benchmarks, extending the analysis to additional architectures, tasks, and modalities would further establish the generality of the observed scaling trends. Finally, our study focuses on model capacity allocation and does not investigate other forms of inference-time scaling, such as adaptive routing, increased decoding budgets, or iterative refinement beyond a single critique–refine cycle.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-r1. Nature, 645(8081):633–638. John R. Hayes. 1996. A new framework for understanding cognition and affect in writing. In The Science of Writing. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, and 3 others. 2022. Training compute-optimal large language models. Preprint, arXiv:2203.15556.

References Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. Preprint, arXiv:2005.14165.

Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of llm agents: A survey. Preprint, arXiv:2402.02716.

Lingjiao Chen, Matei Zaharia, and James Zou. 2023a. Frugalgpt: How to use large language models while reducing cost and improving performance. Preprint, arXiv:2305.05176.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.

Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2024. Iterative translation refinement with large language models. Preprint, arXiv:2306.03856.

Ronald T. Kellogg. 1996. A model of working memory in writing. Journal of Writing Research.

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023b. Teaching large language models to self-debug. Preprint, arXiv:2304.05128.

Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. 2024. An llm compiler for parallel function calling. Preprint, arXiv:2312.04511.

Dujian Ding, Ankur Mallick, Shaokun Zhang, Chi Wang, Daniel Madrigal, Mirian Del Carmen Hipolito Garcia, Menglin Xia, Laks V. S. Lakshmanan, Qingyun Wu, and Victor Rühle. 2025. Best-route: Adaptive llm routing with test-time optimal compute. Preprint, arXiv:2506.22716.

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. Preprint, arXiv:2006.16668.

William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Preprint, arXiv:2101.03961.

Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. 2025. Zebralogic: On the scaling limits of llms for logical reasoning. Preprint, arXiv:2502.01100.

Linda Flower and John R. Hayes. 1980. Identifying the organization of writing processes. In Cognitive Processes in Writing.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. Preprint, arXiv:2303.17651.

Ravindra Aribowo Tarunokusumo and Rafael Fernandes Cunha. 2025. Boosting accuracy and efficiency of budget forcing in llms via reinforcement learning for mathematical reasoning. Preprint, arXiv:2510.21398. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.

Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In NAACL-HLT.

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation. Preprint, arXiv:2308.08155.

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. Preprint, arXiv:2501.19393. Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2025. Routellm: Learning to route llms with preference data. Preprint, arXiv:2406.18665.

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.

OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, and 246 others. 2026. Openai o1 system card. Preprint, arXiv:2412.16720.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. Preprint, arXiv:2210.03629. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. 2024. Quiet-star: Language models can teach themselves to think before speaking. Preprint, arXiv:2403.09629.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Minxing Zhang, Yi Yang, Roy Xie, Bhuwan Dhingra, Shuyan Zhou, and Jian Pei. 2025. Generalizability of large language model-based agents: A comprehensive survey. Preprint, arXiv:2509.16330.

Delip Rao and Chris Callison-Burch. 2026. Autorubric: Unifying rubric-based llm evaluation. Preprint, arXiv:2603.00077.

Minxing Zhang, Yi Yang, Roy Xie, Bhuwan Dhingra, Shuyan Zhou, and Jian Pei. 2026. Generalizability of large language model-based agents: A comprehensive survey. ACM Comput. Surv., 58(10).

Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.

Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, and Denny Zhou. 2024. Natural plan: Benchmarking llms on natural language planning. Preprint, arXiv:2406.04520.

Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Preprint, arXiv:2303.11366.

Changzhi Zhou, Xinyu Zhang, Dandan Song, Xiancai Chen, Wanli Gu, Huipeng Ma, Yuhang Tian, Mengdi Zhang, and Linmei Hu. 2025. Refinecoder: Iterative improving of large language models via adaptive critique refinement for code generation. Preprint, arXiv:2502.09183.

Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. 2024. Learning performanceimproving code edits. Preprint, arXiv:2302.07867.

Implementation details

results in Figure 7. The Gemma results closely mirror those observed for Qwen3. Across all five benchmarks, pipeline performance is largely sensitive to both the size of the generator and the refiner. A weak refiner can underperform the initial generation. The performance is the least sensitive to critic size. These results demonstrate that our conclusions are not specific to a particular model family.

All experiments were conducted using Qwen3 and Gemma 3 models on NVIDIA H100 GPUs. We used task-specific few-shot prompts for the generator, critic, and refiner stages, with the complete prompt templates provided in the appendix. Generation was performed using greedy decoding with a temperature of 0.0 and a maximum output length of 700 new tokens. The same decoding configuration was used across model sizes and pipeline stages to ensure that observed performance differences could be attributed to model capacity rather than changes in inference settings.

Figure 8 compares the effect of increasing model size at each stage independently while fixing the other two stages at Qwen3-14B. The blue line represents the trend of Generator sweep, the green line represents that of the Critic sweep, and the red line represents that of the Refiner sweep (Defined in Section 3.2) The same qualitative observations reported in the main text remain consistent. In particular, generator and refiner scaling substantially affect pipeline performance, whereas critique scaling produces comparatively smaller gains.

Dataset Specifics

We evaluate on a subset of each benchmark to keep the experiments computationally tractable. Depending on the benchmark, we either randomly sample evaluation instances or restrict the evaluation to subsets of controlled difficulty, as described below.

Meeting Planning. We evaluate on 500 examples. Since even the largest model used in our experiments struggles on the most difficult instances (Zheng et al., 2024), we restrict the evaluation to problems involving at most five people.

Qwen3-8B Configuration

Figure 9 compares the effect of increasing model size at each stage independently while fixing the other two stages at Qwen3-8B. The blue line represents the trend of Generator sweep, the green line represents that of the Critic sweep, and the red line represents that of the Refiner sweep (Defined in Section 3.2) The same qualitative observations reported in the main text remain consistent. In particular, generator and refiner scaling substantially affect pipeline performance, whereas critique scaling produces comparatively smaller gains.

CNN/DailyMail (Hermann et al., 2015; See et al., 2017). We evaluate on the first 500 examples from version 1.0.0 of the official test split. ZebraLogic. We evaluate on 320 examples. Since even the largest model used in our experiments struggles on the most difficult puzzles (Lin et al., 2025), we restrict the evaluation to puzzles with at most 10 cells.

Degradation Pipelines

To complement the qualitative analysis presented in Section 5.3, Table 4 summarizes all benchmark– refiner-size combinations in which the final refined output performs worse than the corresponding initial generation. These degradation cases occur across multiple benchmarks and model families, indicating that the phenomenon is systematic rather than limited to a single task or model. This table reports the refiner sizes that lead to degradation under each experimental configuration.

PIE (Shypula et al., 2024). Because evaluating each generated program is computationally expensive, we randomly sample 500 examples from the benchmark. CollaboSentGen (Mostafazadeh et al., 2016). We evaluate on 500 randomly selected examples.

Qwen3-14B Configuration

Additional Results on the Gemma Family

To evaluate whether our observations generalize beyond Qwen3, we repeat the stage-wise scaling analysis using the Gemma family. Specifically, we evaluate Gemma 3 1B-IT, Gemma 3 4B-IT, Gemma 3 12B-IT, and Gemma 3 27B-IT. We plotted the 11

Prompts

Generator Prompts The generator receives the task description with examples (x) and produces an initial solution y0 . Critic Prompts The critic receives the task description with examples (x) and the initial solution y0 . It analyzes the generated solution y0 , identify errors, and provide actionable critique f . Refiner Prompts The refiner receives the task description with examples (x), the initial solution y0 , and the critique f , and is instructed to produce a revised final solution yr that incorporates the critique. The prompts used in our experiments are organized as follows: • Meeting Plan: Figures 10–12 • CNN/DailyMail: Figures 13–15 • ZebraLogic: Figures 16–18 • PIE: Figure 19 – 21 • CollaboSentGen: Figures 22–24

(a) Meeting Planning

1.0

(b) CNN/DailyMail

0.9 0.8

0.8

0.7

0.7 0.6

0.6

0.5

Task-specific evaluation metric

0.4

0.5

0.4

0.3

0.2 0.0

0.3

0.2 1B

12B

27B

0.1

(d) PIE

12B

27B

0.2

12B

27B

(e) CollaboSentGen 9

0.7

0.6 0.5

0.4

0.3

0.2

0.1

0.0

0.8

12B

27B

Initial generation Generator Sweep Critic Sweep Refiner Sweep 1B

12B

27B

Model size of the stage under evaluation

Figure 7: Stage-wise model-size sweeps across five benchmarks under the Gemma 3-27 configuration. X-axis is the model sizes. Blue line, green line, and the red line represent Generator sweep, Critic sweep and the Refiner sweep respectively. The grey dashed line reports the corresponding initial-generation performance without refinement. Steeper curves indicate greater sensitivity to the model size of that stage. Metrics are defined in Section 4.2

(a) Meeting Planning

(b) CNN/DailyMail

0.8

0.7 0.7

0.6

0.6 0.6

0.5

0.4

0.5

Task-specific evaluation metric

0.4 0.2

0.6B

1.7B

14B

32B

0.2

0.6B

(d) PIE

0.8

1.7B

14B

32B

0.3

0.6B

1.7B

14B

32B

(e) CollaboSentGen 9

0.7

0.6

0.5 0.4

0.3

0.2

0.1

0.4

0.3

0.6B

1.7B

14B

32B

Initial generation Generator Sweep Critic Sweep Refiner Sweep 0.6B

1.7B

14B

32B

Model size of the stage under evaluation

Figure 8: Stage-wise model-size sweeps across five benchmarks under the Qwen3-14B configuration. X-axis is the model sizes. Blue line, green line, and the red line represent Generator sweep, Critic sweep and the Refiner sweep respectively. The grey dashed line reports the corresponding initial-generation performance without refinement. Steeper curves indicate greater sensitivity to the model size of that stage. Metrics are defined in Section 4.2

(a) Meeting Planning

0.8

(b) CNN/DailyMail

0.9

0.8 0.6

0.4

Task-specific evaluation metric

0.7

0.7 0.6

0.6

0.5

0.4

0.2

0.4

0.3 0.0

0.6B

1.7B

14B

0.2

32B

(d) PIE

0.6B

1.7B

14B

32B

0.3

0.6B

1.7B

14B

32B

(e) CollaboSentGen

0.7 7

0.6

0.5

0.4 0.3

0.2

0.1

0.6B

1.7B

14B

32B

Initial generation Generator Sweep Critic Sweep Refiner Sweep 0.6B

1.7B

14B

32B

Model size of the stage under evaluation

Figure 9: Stage-wise model-size sweeps across five benchmarks under the Qwen3-8B configuration. X-axis is the model sizes. Blue line, green line, and the red line represent Generator sweep, Critic sweep and the Refiner sweep respectively. The grey dashed line reports the corresponding initial-generation performance without refinement. Steeper curves indicate greater sensitivity to the model size of that stage. Metrics are defined in Section 4.2

(a) Qwen3-32B Benchmark

0.6B

1.7B

Meeting Planning CNN/DailyMail ZebraLogic PIE CollaboSentGen

✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓

Total

(b) Qwen3-14B 4B

✓ ✓

14B

32B

✓

12 / 30

Benchmark

0.6B

1.7B

Meeting Planning CNN/DailyMail ZebraLogic PIE CollaboSentGen

✓ ✓ ✓ ✓ ✓

✓ ✓ ✓

✓

Total

Meeting Planning CNN/DailyMail ZebraLogic PIE CollaboSentGen Total

0.6B ✓ ✓ ✓ ✓

1.7B

✓ ✓ ✓

14B

32B

9 / 30

(d) Gemma3-27B 4B

14B

32B

✓ 8 / 30

Benchmark

Meeting Planning CNN/DailyMail ZebraLogic PIE CollaboSentGen

✓

✓ ✓ ✓

Total

12B

27B

8 / 20

Table 4: Refiner sizes that perform below the corresponding initial generation under different model configurations. A checkmark indicates that using the corresponding refiner size results in lower performance than the initial generation.

Given the constraints, generated a meeting plan that allows to meet with the maximum number of people. Constraints: ["Mission District", "9:00AM"], ["Emily", "Financial District", "8:00AM to 3:30PM", 30], ["Kevin", "The Castro", "7:30PM to 8:30PM", 15], ["Joseph", "Sunset District", "8:15AM to 3:00PM", 75], ["James", "Alamo Square", "8:15AM to 12:00PM", 75] Dist_matrix: "Mission District": {"Financial District": 17, "The Castro": 7, "Sunset District": 24, "Alamo Square": 11}, "Financial District": {"Mission District": 17, "The Castro": 23, "Sunset District": 31, "Alamo Square": 17}, "The Castro": {"Mission District": 7, "Financial District": 20, "Sunset District": 17, "Alamo Square": 8}, "Sunset District": {"Mission District": 24, "Financial District": 30, "The Castro": 17, "Alamo Square": 17}, "Alamo Square": {"Mission District": 10, "Financial District": 17, "The Castro": 8, "Sunset District": 16} ** Solution: You start at Mission District at 9:00AM. You travel to Alamo Square in 11 minutes and arrive at 9:11AM. You meet James for 75 minutes from 9:11AM to 10:26AM. You travel to The Castro in 8 minutes and arrive at 10:34AM. You wait until 7:30PM. You meet Kevin for 15 minutes from 7:30PM to 7:45PM. You travel to Financial District in 23 minutes and arrive at 8:08PM. You meet Emily for 30 minutes from 8:08PM to 8:38PM. You travel to Sunset District in 31 minutes and arrive at 9:09PM. You meet Joseph for 75 minutes from 9:09PM to 10:24PM.

Figure 10: Generator prompt for Meeting Plan

Given the constraints, generated a meeting plan that allows meetings with the maximum number of people. Constraints: ["Mission District", "9:00AM"], ["Emily", "Financial District", "8:00AM to 3:30PM", 30], ["Kevin", "The Castro", "7:30PM to 8:30PM", 15], ["Joseph", "Sunset District", "8:15AM to 3:00PM", 75], ["James", "Alamo Square", "8:15AM to 12:00PM", 75] Dist_matrix: "Mission District": {"Financial District": 17, "The Castro": 7, "Sunset District": 24, "Alamo Square": 11}, "Financial District": {"Mission District": 17, "The Castro": 23, "Sunset District": 31, "Alamo Square": 17}, "The Castro": {"Mission District": 7, "Financial District": 20, "Sunset District": 17, "Alamo Square": 8}, "Sunset District": {"Mission District": 24, "Financial District": 30, "The Castro": 17, "Alamo Square": 17}, "Alamo Square": {"Mission District": 10, "Financial District": 17, "The Castro": 8, "Sunset District": 16} Solution: You start at Mission District at 9:00AM. You travel to Alamo Square in 11 minutes and arrive at 9:11AM. You meet James for 75 minutes from 9:11AM to 10:26AM. You travel to The Castro in 8 minutes and arrive at 10:34AM. You wait until 7:30PM. You meet Kevin for 15 minutes from 7:30PM to 7:45PM. You travel to Financial District in 23 minutes and arrive at 8:08PM. You meet Emily for 30 minutes from 8:08PM to 8:38PM. You travel to Sunset District in 31 minutes and arrive at 9:09PM. You meet Joseph for 75 minutes from 9:09PM to 10:24PM. ** Feedback: The generated plan is incorrect because it schedules the meeting with Emily at 8:08PM to 8:38PM, which is outside her availability window of 8:00AM to 3:30PM. Additionally, the meeting with Joseph is scheduled from 9:09PM to 10:24PM, which is outside his availability window of 8:15AM to 3:00PM. Both these meetings are invalid due to timing conflicts.

Figure 11: Critic prompt for Meeting Plan

Correct the errors in the plan based on the constraints, initial plan, and the feedback. Constraints: ["Mission District", "9:00AM"], ["Emily", "Financial District", "8:00AM to 3:30PM", 30], ["Kevin", "The Castro", "7:30PM to 8:30PM", 15], ["Joseph", "Sunset District", "8:15AM to 3:00PM", 75], ["James", "Alamo Square", "8:15AM to 12:00PM", 75] Dist_matrix: "Mission District": {"Financial District": 17, "The Castro": 7, "Sunset District": 24, "Alamo Square": 11}, "Financial District": {"Mission District": 17, "The Castro": 23, "Sunset District": 31, "Alamo Square": 17}, "The Castro": {"Mission District": 7, "Financial District": 20, "Sunset District": 17, "Alamo Square": 8}, "Sunset District": {"Mission District": 24, "Financial District": 30, "The Castro": 17, "Alamo Square": 17}, "Alamo Square": {"Mission District": 10, "Financial District": 17, "The Castro": 8, "Sunset District": 16} Solution: You start at Mission District at 9:00AM. You travel to Alamo Square in 11 minutes and arrive at 9:11AM. You meet James for 75 minutes from 9:11AM to 10:26AM. You travel to The Castro in 8 minutes and arrive at 10:34AM. You wait until 7:30PM. You meet Kevin for 15 minutes from 7:30PM to 7:45PM. You travel to Financial District in 23 minutes and arrive at 8:08PM. You meet Emily for 30 minutes from 8:08PM to 8:38PM. You travel to Sunset District in 31 minutes and arrive at 9:09PM. You meet Joseph for 75 minutes from 9:09PM to 10:24PM. Feedback: The generated plan is incorrect because it schedules the meeting with Emily at 8:08PM to 8:38PM, which is outside her availability window of 8:00AM to 3:30PM. Additionally, the meeting with Joseph is scheduled from 9:09PM to 10:24PM, which is outside his availability window of 8:15AM to 3:00PM. Both these meetings are invalid due to timing conflicts. ** Refined solution: You start at Mission District at 9:00 AM. You travel to Alamo Square in 11 minutes and arrive at 9:11 AM. You meet James for 75 minutes from 9:11 AM to 10:26 AM. You travel to Sunset District in 16 minutes and arrive at 10:42 AM. You meet Joseph for 75 minutes from 10:42 AM to 11:57 AM. You travel to Financial District in 30 minutes and arrive at 12:27 PM. You meet Emily for 30 minutes from 12:27 PM to 12:57 PM. You travel to The Castro in 23 minutes and arrive at 1:20 PM. You wait until 7:30 PM. You meet Kevin for 15 minutes from 7:30 PM to 7:45 PM.

Figure 12: Refiner prompt for Meeting Plan

You are an expert news editor. Given the original article, summarize the highlights of it. Article: (CNN)Never mind cats having nine lives. A stray pooch in Washington State has used up at least three of her own after being hit by a car, apparently whacked on the head with a hammer in a misguided mercy killing and then buried in a field -- only to survive. That's according to Washington State University, where the dog -- a friendly white-and-black bully breed mix now named Theia -- has been receiving care at the Veterinary Teaching Hospital. Four days after her apparent death, the dog managed to stagger to a nearby farm, dirt-covered and emaciated, where she was found by a worker who took her to a vet for help. She was taken in by Moses Lake, Washington, resident Sara Mellado. "Considering everything that she's been through, she's incredibly gentle and loving," Mellado said, according to WSU News. "She's a true miracle dog and she deserves a good life." Theia is only one year old but the dog's brush with death did not leave her unscathed. She suffered a dislocated jaw, leg injuries and a caved-in sinus cavity -- and still requires surgery to help her breathe. The veterinary hospital's Good Samaritan Fund committee awarded some money to help pay for the dog's treatment, but Mellado has set up a fundraising page to help meet the remaining cost of the dog's care. … ** Highlights: A stray dog named Theia in Washington State survived a car accident and injuries, now in need of surgery. …

Figure 13: Generator prompt for cnn/DailyMail

Given the original article and the generated highlights, provide feedback on how the generated highlights can be improved. Article: (CNN)Never mind cats having nine lives. A stray pooch in Washington State has used up at least three of her own after being hit by a car, apparently whacked on the head with a hammer in a misguided mercy killing and then buried in a field -- only to survive. That's according to Washington State University, where the dog -- a friendly white-and-black bully breed mix now named Theia -- has been receiving care at the Veterinary Teaching Hospital. Four days after her apparent death, the dog managed to stagger to a nearby farm, dirt-covered and emaciated, where she was found by a worker who took her to a vet for help. She was taken in by Moses Lake, Washington, resident Sara Mellado. "Considering everything that she's been through, she's incredibly gentle and loving," Mellado said, according to WSU News. "She's a true miracle dog and she deserves a good life." Theia is only one year old but the dog's brush with death did not leave her unscathed. She suffered a dislocated jaw, leg injuries and a caved-in sinus cavity -- and still requires surgery to help her breathe. The veterinary hospital's Good Samaritan Fund committee awarded some money to help pay for the dog's treatment, but Mellado has set up a fundraising page to help meet the remaining cost of the dog's care. … Highlights: A stray dog named Theia in Washington State survived a car accident and injuries, now in need of surgery. … ** Feedback: The summary misses Sara Mellado’s statement that Theia is “a true miracle dog” and her commitment to finding Theia a loving home. …

Figure 14: Critic prompt for cnn/DailyMail

Given the original article, the generated highlights, and the feedback, revise the generated highlights. Article: (CNN)Never mind cats having nine lives. A stray pooch in Washington State has used up at least three of her own after being hit by a car, apparently whacked on the head with a hammer in a misguided mercy killing and then buried in a field -- only to survive. That's according to Washington State University, where the dog -- a friendly white-and-black bully breed mix now named Theia -- has been receiving care at the Veterinary Teaching Hospital. Four days after her apparent death, the dog managed to stagger to a nearby farm, dirt-covered and emaciated, where she was found by a worker who took her to a vet for help. She was taken in by Moses Lake, Washington, resident Sara Mellado. "Considering everything that she's been through, she's incredibly gentle and loving, " Mellado said, according to WSU News. "She's a true miracle dog and she deserves a good life. " Theia is only one year old but the dog's brush with death did not leave her unscathed. She suffered a dislocated jaw, leg injuries and a caved-in sinus cavity -- and still requires surgery to help her breathe. The veterinary hospital's Good Samaritan Fund committee awarded some money to help pay for the dog's treatment, but Mellado has set up a fundraising page to help meet the remaining cost of the dog's care. … Highlights: A stray dog named Theia in Washington State survived a car accident and injuries, now in need of surgery. … Feedback: The summary misses Sara Mellado’s statement that Theia is “a true miracle dog” and her commitment to finding Theia a loving home. … ** Refined highlights: A stray dog named Theia in Washington State survived a car accident and severe injuries, now needs surgery, and was called “a true miracle dog” by her rescuer Sara Mellado, who is raising funds to cover her medical care. …

Figure 15: Refiner prompt for cnn/DailyMail

You are an expert puzzle solver. Given the clues, generate a solution to the puzzle. Puzzle: There are 2 houses, numbered 1 to 2 from left to right, as seen from across the street. Each house is occupied by a different person. Each house has a unique attribute for each of the following characteristics: - Each person has a unique name: `Arnold`, `Eric` - Each person lives in a unique style of house: `colonial`, `victorian` - People use unique phone models: `iphone 13`, `samsung galaxy s21` Clues: 1. Eric is somewhere to the left of the person who uses an iPhone 13. 2. The person who uses a Samsung Galaxy S21 is the person residing in a Victorian house. ** Solution: “House 1": {"Name": "Eric", "House Style": "colonial", "Phone": "samsung galaxy s21"}, "House 2": {"Name": "Arnold", "House Style": "victorian", "Phone": "iphone 13"}

Figure 16: Generator prompt for ZebraLogic

Given the puzzle and the generated solution, provide feedback to fix the errors. Puzzle: There are 2 houses, numbered 1 to 2 from left to right, as seen from across the street. Each house is occupied by a different person. Each house has a unique attribute for each of the following characteristics: - Each person has a unique name: `Arnold`, `Eric` - Each person lives in a unique style of house: `colonial`, `victorian` - People use unique phone models: `iphone 13`, `samsung galaxy s21` Clues: 1. Eric is somewhere to the left of the person who uses an iPhone 13. 2. The person who uses a Samsung Galaxy S21 is the person residing in a Victorian house. Solution: “House 1": {"Name": "Eric", "House Style": "colonial", "Phone": "samsung galaxy s21"}, "House 2": {"Name": "Arnold", "House Style": "victorian", "Phone": "iphone 13"} ** Feedback: Clue 1 states that Eric is to the left of the person who uses an iPhone 13. Since Eric is in House 1, the iPhone 13 user must be in House 2. Clue 2 states that the Samsung Galaxy S21 user resides in a Victorian house. Since the Samsung Galaxy S21 is assigned to House 1, House 1 must be Victorian.

Figure 17: Critic prompt for ZebraLogic

Given the puzzle, the generated solution, and the feedback, revise the solution based on the feedback. Puzzle: There are 2 houses, numbered 1 to 2 from left to right, as seen from across the street. Each house is occupied by a different person. Each house has a unique attribute for each of the following characteristics: - Each person has a unique name: `Arnold`, `Eric` - Each person lives in a unique style of house: `colonial`, `victorian` - People use unique phone models: `iphone 13`, `samsung galaxy s21` Clues: 1. Eric is somewhere to the left of the person who uses an iPhone 13. 2. The person who uses a Samsung Galaxy S21 is the person residing in a Victorian house. Solution: “House 1": {"Name": "Eric", "House Style": "colonial", "Phone": "samsung galaxy s21"}, "House 2": {"Name": "Arnold", "House Style": "victorian", "Phone": "iphone 13"} Feedback: Clue 1 states that Eric is to the left of the person who uses an iPhone 13. Since Eric is in House 1, the iPhone 13 user must be in House 2. Clue 2 states that the Samsung Galaxy S21 user resides in a Victorian house. Since the Samsung Galaxy S21 is assigned to House 1, House 1 must be Victorian. ** Refined solution: "House 1": { "Name": "Eric", "House Style": "victorian", "Phone": "samsung galaxy s21" }, "House 2": { "Name": "Arnold", "House Style": "colonial", "Phone": "iphone 13" }

Figure 18: Refiner prompt for ZebraLogic

“

You are an expert code optimizer, given the initial slow code, generate a more efficient code. Initial code: def count_duplicates(values): duplicate_count = 0 for value in values: if values.count(value) > 1: duplicate_count += 1 return duplicate_count ** Optimized code: from collections import Counter def count_duplicates(values): frequencies = Counter(values) return sum(count for count in frequencies.values() if count > 1)

Figure 19: Generator prompt for PIE

Given the initial slow code and the optimized code, generate feedback on the optimized code to help make it more efficient. Initial code: def count_duplicates(values): duplicate_count = 0 for value in values: if values.count(value) > 1: duplicate_count += 1 return duplicate_count Optimized code: from collections import Counter def count_duplicates(values): frequencies = Counter(values) return sum(count for count in frequencies.values() if count > 1) ** Feedback: The implementation is inefficient because values.count(value) scans the full list during every loop iteration, producing quadratic time complexity. Build a frequency table in one pass, then sum the frequencies of values that occur more than once.

Figure 20: Critic prompt for PIE

Given the initial slow code, the optimized code, and the feedback, revise the original optimized code to make it more effective. Initial code: def count_duplicates(values): duplicate_count = 0 for value in values: if values.count(value) > 1: duplicate_count += 1 return duplicate_count Optimized code: from collections import Counter def count_duplicates(values): frequencies = Counter(values) return sum(count for count in frequencies.values() if count > 1) Feedback: The implementation is inefficient because values.count(value) scans the full list during every loop iteration, producing quadratic time complexity. Build a frequency table in one pass, then sum the frequencies of values that occur more than once. ** Refined code: def count_duplicates(values): frequencies = {} for value in values: frequencies[value] = frequencies.get(value, 0) + 1 duplicate_count = 0 for frequency in frequencies.values(): if frequency > 1: duplicate_count += frequency return duplicate_count

Figure 21: Refiner prompt for PIE

You are an expert story writer, given a partial story and some keywords, generated the missing sentence that fits into the story. Story: 1. Garrett wanted to be the president one day. 2. He studied politics and law at college. 3. He was at the top of his class. 4. <MISSING> 5. He learned that it wasn't what you know, but who you know. Keywords: [to, Garett] ** Generated sentence: Garrett realized that to succeed, he needed connections and influence.

Figure 22: Generator prompt for CollaboSentGen

You are an expert story writer, given a partial story, some keywords, and the generated sentence, give feedback on the sentence to make it fit better into the story. Story: 1. Garrett wanted to be the president one day. 2. He studied politics and law at college. 3. He was at the top of his class. 4. <MISSING> 5. He learned that it wasn't what you know, but who you know. Keywords: [to, Garett] Generated sentence: Garrett realized that to succeed, he needed connections and influence. ** Feedback: It gives a conclusion instead of describing the key event that should happen at this point. It does not explain what led to the realization in the next sentence, so the connection is incomplete.

Figure 23: Critic prompt for CollaboSentGen

You are an expert story writer, given a partial story, some keywords, the generated sentence, and the feedback, revise the generated sentence to make it fit better into the story. Story: 1. Garrett wanted to be the president one day. 2. He studied politics and law at college. 3. He was at the top of his class. 4. <MISSING> 5. He learned that it wasn't what you know, but who you know. Keywords: [to, Garett] Generated sentence: Garrett realized that to succeed, he needed connections and influence. Feedback: It gives a conclusion instead of describing the key event that should happen at this point. It does not explain what led to the realization in the next sentence, so the connection is incomplete. ** Refined sentence: Despite this, Garett could not manage to find work in politics.

Figure 24: Refiner prompt for CollaboSentGen

Asymmetric Capacity Allocation in Self-Refinement Pipelines

Analyze & explore related research in Conceptio

Related documents