arXiv CSopen access

Controllable Spoken Dialogue Generation: An LLM-Driven Grading System for K-12 Non-Native English Learners

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificial intelligence, reasoning, knowledge representation

Controllable Spoken Dialogue Generation: An LLM-Driven Grading System for K-12 Non-Native English Learners Haidong Yuan1 * , Haokun Zhao5 , Wanshi Xu1 , Songjun Cao2 , Qingyu Zhou4 , Long Ma2 , Hongjie Fan3 1

Peking University, 2 Tencent, 3 China University of Political Science and Law, 4 Independent Researcher, 5 Fudan University Correspondence to: Hongjie Fan, Songjun Cao

arXiv:2604.22542v1 [cs.CL] 24 Apr 2026

Abstract

TOPIC: Manage time effectively and balance study with leisure. Hi! How was your day today? Can you tell me about your after-school routine and what you usually do?

Large language models (LLMs) often fail to meet the pedagogical needs of K-12 English learners in non-native contexts due to a proficiency mismatch. To address this widespread challenge, we introduce a proficiency-aligned framework that adapts LLM outputs to learner abilities, using China’s national curriculum (CSE) as a representative case. Our framework enables precise control over lexical complexity through a four-tier grading system, supported by a comprehensive suite of new resources: graded vocabulary lists and a multi-turn dialogue corpus. Our core technical contribution is the DDPO algorithm,Diversity Driven Policy Optimization, a multi-turn GRPO-based approach designed to preserve dialogue diversity while holistically optimizing dialogue quality. This method significantly outperforms conventional approaches, achieving low outof-vocabulary rates and high diversity while enhancing conversational naturalness and pedagogical value. While grounded in the CSE, our framework is designed for flexibility and can be readily adapted to other educational standards. Our models, data, and code will all be open-sourced, providing a scalable platform for personalized English speaking practice that effectively addresses the unique challenges faced by K-12 learners in non-immersive environments.

Today was good. After school, I do homework, playsports, and read. These activities help me stay healthy,creative, and improve teamwork.

Four different levels of replies

That sounds like fun! Do you usually have a favorite thing you do in your free time?

Great job! Homework, sports, and reading are all good. Do you think these activities also help you grow as a person?

Great! Those are excellent activities! Teamwork is important. Do you have any special goals for balancing these activities better?

Great job! Balancing these activities is excellent for personal growth. Do you have any specific goals or challenges in managing your time effectively?

Figure 1: The figure shows examples of outputs at different proficiency levels for the same topic and input. Words in red indicate vocabulary specific to each level.

significant barrier to improving their listening and speaking skills. Recent advances in artificial intelligence have led to the widespread adoption of large language models (LLMs) in education, with particularly promising applications in language learning and oral practice (Hagos et al., 2024; Dong et al., 2024; Gao et al., 2023). Although numerous AI-powered English learning tools have emerged, most fail to fully address the specific needs of English education in these regions. Taking China as a representative case, these systems often lack fine-grained differentiation based on students’ cognitive development, vocabulary levels, and grammatical proficiency. This results in a mismatch between learning content and learners’ abilities, ultimately leading to suboptimal outcomes (Li, 2024). According to the theory of the “zone of proximal devel-

Introduction

For language learners, an ideal environment must provide ample opportunities for both input and output in the target language, as well as authentic interactive scenarios. However, primary and secondary school students in non-native Englishspeaking countries often lack authentic English communication environments, which severely limits their opportunities for oral practice and poses a * Email: [email protected]

L1 dialogue

Step 1

Step 2

I own a toy car.

Rule check

I have a toy car.

Expert check

teacher textbook

constrained decoding

graded vocabulary

examination

topic

grammar

ability

word count

CSE

prompt

student

raw dialogue

fixed dialogue

final dialogue

System prompt Difficulty Level: {Level} … Identity: spoken English teacher … Topic: {Topic} … Student’s Language Ability: {Ability} … Sentence Length Limit: {Word Count} Vocabulary Level: {Level} Allowed Grammar: {Grammar} …

Exc it will be exciting! What else would you like to see at the zoo?

It will be exciting! What else would you like to see at the zoo?

tiger lion z z z z z z z z

Tigers and lions.

Tiger and lion, strong and brave! What do you think they do in their shows?

Tigers and lions, strong and brave! What do you think they do in their shows?

constrained decoding

GPT fix error

Figure 2: Data generation pipeline: Step 1, vocabulary, grammar, and proficiency information are extracted from textbooks and the CSE framework. The vocabulary list constrains decoding in Step 2, while other information is provided via prompts in this same step. Step 2, Two constrained-decoding language models then generate teacher-student dialogues using only the specified vocabulary. An unconstrained model corrects language errors, and finally, rule-based and human reviews select high-quality dialogues.

opment" (Danish et al., 2017), the most effective learning occurs when students are exposed to language input that is just near their current level, which promotes both comprehension and progress. If the learning content is too complex, students may struggle with comprehension, undermining their motivation and learning efficacy; conversely, if the content is too simple, it lacks sufficient challenge and fails to stimulate further growth. So our system realizes dynamic adaptation of responses to students at different levels. For details, please refer to Figure 1. Furthermore, there are substantial differences in English language education frameworks across countries. For instance, while the Common European Framework of Reference for Languages (CEFR) is widely adopted and supported by extensive research and datasets (Amiruddin et al., 2025; Bacher et al., 2024), it differs considerably from the educational standards in China (Peng et al., 2022), making it challenging to directly apply CEFR-based proficiency levels to local contexts. In contrast, China’s Standards of English Language Ability (CSE) (Zhao et al., 2017; Wu and Roever, 2021) are more closely aligned with the specific learning progression and educational context of Chinese students. From a technical perspective, controlled text generation (CTG) methods—such as prompt engineering, constrained decoding, instruction tuning, and reinforcement learning (RL) (Zhang et al., 2023)—offer effective techniques for guiding and constraining LLM outputs. Among these, reinforcement learning from human feedback (RLHF) has become the mainstream approach for aligning models with human preferences. For example, Chat-

GPT employs the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) to enhance dialogue consistency and alignment with human values. More recently, models such as DeepSeekR1 (Guo et al., 2025) have utilized the PPO variant GRPO (Shao et al., 2024) to optimize mathematical reasoning tasks, driving further advances in algorithms such as DAPO (Yu et al., 2025) and GRPO-LEAD (Zhang and Zuo, 2025). However, although reinforcement learning has been widely applied to enhance reasoning abilities, its application in aligning model outputs with language proficiency standards in educational contexts has received comparatively less attention. To address these challenges, we propose an English proficiency grading and dialogue system tailored to the needs of K-12 learners in non-native contexts, using China’s educational standards as a representative case study. Our main contributions are as follows: • Diversity Driven Policy Optimization. (DDPO): We introduce DDPO, a multi-turn reinforcement learning algorithm designed to mitigate the entropy collapse phenomenon often observed in GRPO during dialogue tasks. This approach effectively balances dialogue quality, response diversity, and reinforcement learning optimization objectives. • Comprehensive Graded Resources: We construct a large-scale suite of pedagogical resources grounded in China’s national curriculum and the China Standards of English (CSE). This includes hierarchically graded vocabulary lists, and a high-quality multi-turn dialogue corpus tailored for varying proficiency 2

Dynamic Generation

H, Q0

Policy Model

Reference Model

O1-1

Q1-1

O1-2

O2-1

Q2-1

O2-2

OG-1

QG-1

OG-2

Quality Reward

Multi turn Diversity

Dynamic Weight

…

Multi sample Diversity

…

A1 Group Computation

… AG

Figure 3: Overview of the Diversity Driven Policy Optimization (DDPO) framework. Given a fixed dialogue history h, we sample G multi-turn trajectories. We utilize the G first-turn responses to compute the Multi sample diversity, and the sequential outputs within a single trajectory to compute the Multi-turn diversity. Combined with a task-specific quality reward, these components are aggregated via dynamic weighting to determine the final reward, followed by advantage estimation and parameter updates.

levels.

suitable for overall difficulty assessment but not for training LLMs to generate short, natural conversational responses. TSCC (Caines et al., 2020) provides teacher-student dialogues with CEFR-level annotations, but its difficulty distribution is uneven and the content is more knowledge-based than conversational. DailyDialog (Li et al., 2017) offers natural and diverse conversations but lacks proficiency annotations, limiting its use for level-based modeling. Sentence-level datasets like CEFR-SP provide finer granularity but mainly contain isolated sentences, missing features of authentic dialogues such as phrases, single-word responses, and contextual references. Therefore, there is a clear need for proficiencygraded dialogue corpora that span multiple difficulty levels and align with the CSE.

• First CSE-Aligned Open-Source System: We release the first open-source dialogue system fully aligned with the CSE framework. Extensive experiments demonstrate that our system significantly outperforms conventional prompting and supervised baselines, achieving superior proficiency control while maintaining high dialogue quality and instructional value.

Related Work

2.1

Language Proficiency Frameworks

The integration of language proficiency frameworks with LLMs has mainly focused on automated assessment and controlled generation—detecting proficiency levels in texts and guiding models to produce content matching specific abilities (Amiruddin et al., 2025; Lagutina et al., 2023; Redlich and Pattison, 2024). Internationally, the CEFR is the most widely used framework, classifying learners from A1 to C2. While research has explored the alignment between CEFR and CSE (Coniam et al., 2022; Zhu et al., 2023), LLM applications based on the CSE remain limited compared to those using CEFR. Most English proficiency-annotated datasets, such as EFCAMDAT (Geertzen et al., 2013), CLCFCE (Yannakoudakis et al., 2011), and CEFR-SP (Uchida et al., 2024), are based on long texts and the CEFR framework. These datasets, typically consisting of multi-hundred-word passages, are

2.2

Controlled Text Generation

CTG methods, such as prompt engineering, constrained decoding, instruction tuning, and reinforcement learning, are widely used to regulate large language model outputs. Prompt engineering constrains outputs via carefully designed prompts, and is often used for dialogue generation or task evaluation. For example, Lin et al. (2024) used GPT-4’s few-shot learning to generate fine-grained feedback on teacher errors, achieving accuracy surpassing human experts in some educational settings. Constrained decoding restricts the generation process to meet specific requirements. For instance, Tyen et al. (2022) applied CEFR-based vocabulary 3

room dialogue texts at this level. To address this, our dialogue classification system prioritizes vocabulary difficulty. We first compile vocabulary lists from mainstream textbooks and categorize them into L1–L4, treating basic word forms (e.g., tense, plural) at the same level as their root forms. We then extract dialogue topics and content from primary and secondary school exams and textbooks, ensuring that scenarios align with students’ communicative abilities at each level. Since most collected materials are not in a teacher-student dialogue format, we use constrained decoding with two Qwen2.5-32B to simulate such interactions, strictly limiting output to the allowed vocabulary. The resulting dialogues are in a question-andanswer format, with the teacher model guiding the student model based on the topic and proficiency level. Although constrained decoding ensures that all vocabulary remains within the specified range, it may introduce grammatical errors, lexical mistakes, and meaningless repetitions. To address this, deepseek-v3 is employed to rewrite these dialogues, correcting errors without altering the content or vocabulary. The rewritten content is then subjected to another round of out-of-vocabulary (OOV) detection to ensure compliance, resulting in the final spoken dialogue corpus. The final output consists of a graded vocabulary list and a graded dialogue corpus. All proper nouns, such as personal and place names, are removed from the vocabulary list to prevent unreasonable OOV judgments during the final screening. The specific construction process is illustrated in Figure 2.

10 5 0

$YJ7XUQVSHU7RSLF

$YJ:RUGVSHU6HQWHQFH

/ /

Figure 4: The train dataset covers dialogues from L1 to L4. (Left) Average turns per dialogue and words per turn for each grade; (Right) Topic distribution by grade.

grading, Qian et al. (2023) increased the use of textbook vocabulary, and Glandorf et al. (2025) encouraged target grammatical structures. However, such interventions may introduce errors or reduce output quality. Instruction tuning encodes constraints as natural language instructions, as in the INSTRUCTCTG framework (Zhou et al., 2023), enabling models to learn constraints during training without modifying decoding. Reinforcement learning has also been applied for dialogue control. For example, Du et al. (2024) used stepwise rewards to improve dialogue completion, while Cho et al. (2024) combined multiple reward functions to enhance response quality and controllability. These methods can model realtime learner feedback and dynamically optimize dialogue strategies.

Dataset Construction

To enable the model’s graded dialogue capabilities, a scientifically designed dataset classification is essential. Following Chinese English education standards, we divide students’ English proficiency into four levels (L1–L4), corresponding to Grades 1–4 in primary school, Grades 5–6 in primary school, junior high school, and senior high school, respectively. For each stage, we extract the relevant English proficiency requirements, focusing primarily on vocabulary. Accordingly, we construct a graded dialogue dataset based on level-specific vocabulary. 3.1

Dialogue Turns Dialogue Topics Words Avg. Turns per Topic

Train

Validation

Test

6420 786 191894 8.17

412 51 12123 8.08

427 41 13356 10.41

Table 1: Statistics of the dialogue datasets.

Notably, dialogues at the junior and senior high school levels tend to be longer, resulting in a relatively smaller number of topics compared to the primary school level. Detailed distributions of topics and turns are illustrated in Figure 4 and Table 1.

Graded Dialogue Dataset

According to the Chinese Ministry of Education, L1 students possess a vocabulary of only about 600 words, making it challenging to find suitable class-

Methodology

4.1

Entropy Collapse

In the iterative process of Group Relative Policy Optimization (GRPO), the policy model tends to 4

Human

Model GPT-5.1 Gemini-3.0 Doubao-1.6 HunYuan-turbos

Multi-turn Diversity (Cross-Time) For subsequent turns (k > 1), we address contextual repetition. We penalize the token overlap ratio of the current response ai,k with the current user input ui,k and the previous assistant output ai,k−1 : |ai,k ∩ ui,k | |ai,k ∩ ai,k−1 | Rmul (ai,k ) = − + |ai,k | |ai,k |

Judge Models

Score

GPT5.1

Gemini

DouBao

Hunyuan-T1

4.47(4) 4.63(1) 4.48(3) 4.61(2)

4.07(4) 4.37(1) 4.16(3) 4.35(2)

3.93(4) 4.37(1) 4.02(3) 4.26(2)

3.99(4) 4.21(2) 4.15(3) 4.36(1)

4.24(4) 4.37(2) 4.26(3) 4.40(1)

Table 2: Comparison of average scores and rankings (in parentheses) between human evaluation and various LLM-based judges. The rankings from judge models align closely with human evaluation.

where |S| denotes the unique token count and ∩ denotes token set intersection.

disproportionately reinforce specific token paths that exhibit marginal advantages during intra-group sampling. This causes the probability mass to rapidly concentrate on local optima, leading to a sharp decline in generation diversity—a phenomenon we term Entropy Collapse. In dialogue tasks, this results in severe homogenization (both semantic and syntactic) across multiple samples and multi-turn interactions, significantly degrading user experience. Examples of Entropy Collapse are presented in Appendix G. 4.2

Dynamic Reward Composition We incorporate task-specific quality rewards (e.g., correctness or reward model scores), denoted as Rqual . To effectively balance exploration and exploitation, we employ a dynamic weighting schedule based on the global training step S. The composite reward Ri,k for the k-th turn of the i-th sample is: Ri,k = λqual · Rqual (ai,k ) + λsgl · Rsgl (ai,1 ) + λmul · Rmul (ai,k )

Diversity Driven Policy Optimization

To address the entropy collapse, we propose Diversity Driven Policy Optimization (DDPO). DDPO integrates a user simulator for multi-turn trajectory generation and introduces a dynamic diversity mechanism to balance exploration and exploitation, show in Figure 3.

The coefficients λ are independent weights for each reward, intended to adjust the influence of each component during different training phases. Optimization Objective We adopt token-level averaging to eliminate length bias. The DDPO objective function is defined as:

Multi-turn Sampling Given an initial prompt x, we generate G independent dialogue sessions by interacting with a user simulator U using the policy πθ . The trajectory for the i-th group, denoted as τi , consists of a sequence of K turns:

" JDDPO (θ) = E

ai,k ∼ πθ (·|hi,k ),

ui,k+1 ∼ U(·|hi,k , ai,k )

where hi,k represents the dialogue history for group i at turn k. The terms ui,k and ai,k denote the user utterance and model response at turn k, respectively. Note that all groups are initialized with the same prompt, i.e., ui,1 = x.

ρi,k,t =

1 P j̸=i RougeL(ai,1 , aj,1 ), γ G−1

i=1 k=1 t=1

πθ (ai,k,t |hi,k , ai,k,<t ) πθold (ai,k,t |hi,k , ai,k,<t )

Here, Âi,k is the standardized advantage for the k-th turn, computed relative to the group statistics of that specific turn:

Single-turn Diversity (Cross-Sample) The first turn (k = 1) is prone to mode collapse. We calculate the negative average Rouge-L F1 score of response ai,1 against other samples in the same group. To prevent meaningless gibberish caused by excessive diversity maximization, we apply a clipping threshold γ: Rsgl (ai,1 ) = − max

K |ai,k |

PG PK where Z = i=1 k=1 |ai,k | is the total token count in the batch. The clipped surrogate loss is: clip Li,k,t = ρi,k,t Âi,k , clip(ρi,k,t , 1 − ϵ, 1 + ϵ)Âi,k

τi = {(ui,1 , ai,1 ), . . . , (ui,K , ai,K )} s.t.

q∼P (Q) {τi }G i=1 ∼πθold

1 XX X clip min Li,k,t Z

Âi,k =

Ri,k − µk σk + δ

1 PG where µk = = j=1 Rj,k and σk G G std({Rj,k }j=1 ) represent the mean and standard deviation of rewards for turn k across the group, and δ is a small constant for numerical stability.

Evaluation Metrics

Model

A spoken language practice system must satisfy three core objectives: (1) Comprehensibility, ensuring vocabulary remains within the learner’s proficiency level; (2) Diversity, avoiding monotonous or repetitive outputs; and (3) Educational Quality, ensuring the dialogue is error-free, informative, and capable of guiding the conversation effectively. Accordingly, we establish a three-dimensional evaluation framework covering vocabulary constraints, diversity, and dialogue quality. 5.1

Vocabulary

Rel. ↑

Task ↑

Rich. ↑

Gui. ↑

Avg. ↑

Qwen2.5-7B Series Base 41.69 + Prompt 45.43 + Decoding 9.36 + SFT 23.18 + DDPO 9.36

0.66 0.64 0.65 0.76 0.72

4.97 4.97 4.95 4.98 4.96

3.21 3.41 3.10 3.33 3.63

3.58 3.70 3.47 3.51 3.64

4.50 4.54 4.49 4.47 4.87

4.06 4.15 4.00 4.07 4.30

Llama3.1-8B Series Base 55.26 + SFT 19.90 + DDPO 6.79

0.72 0.74 0.72

4.98 4.99 4.94

3.58 3.60 3.80

3.90 3.48 3.79

4.48 4.51 4.84

4.23 4.14 4.34

dialogue remains pedagogically valuable, engaging, and logically coherent. We employ five criteria to evaluate dialogue quality: • Topic Relevance: The response must be closely aligned with the designated dialogue topic. • Task Completion: The model must fulfill specific constraints in the prompt, such as length, theme, or response format. • Semantic Richness: The response should be informative and varied, avoiding generic or hollow content.

Diversity Evaluation

Diversity is assessed through both cross-sample and cross-turn dimensions. Specifically, for a fixed dialogue history, we generate eight independent multi-turn trajectories. We compute the average Rouge-L score across the first turns of these samples (inter-sample diversity) and the Rouge-L score across the generated multi-turn teacher-student interactions (intra-session diversity). The final diversity metric is calculated as the 1:1 weighted sum of these two scores. 5.3

Quality (1-5)

Div. ↑

Table 3: Main results on the spoken practice evaluation set. Voc.: Vocabulary Violation Rate; Div.: Diversity Score; Rel.: Topic Relevance; Rich.: Semantic Richness; Gui.: Topic Guidance. The best results in each group are highlighted in bold.

Vocabulary evaluation relies on pre-constructed graded vocabulary lists. During the dialogue, the system instructs the model to adopt a specific persona and difficulty level based on the scenario, restricting usage to the assigned vocabulary tier. A vocabulary violation is recorded if the model generates tokens exceeding the learner’s proficiency level (e.g., using L4 words in an L3 session). However, proper nouns (e.g., names, locations) and out-ofvocabulary (OOV) terms previously introduced in the dialogue history are exempt. This exception ensures naturalness, as proper nouns may not be in the predefined lists, and reusing established OOV terms is essential for coherent interaction. 5.2

Objective Voc. ↓

• Topic Guidance: The output should effectively guide the student to think critically and express themselves. LLM-based Evaluation Human evaluation is susceptible to subjective bias and inter-rater variability, making standardization difficult. Furthermore, large-scale manual annotation is prohibitively expensive. However, recent work by Long et al. (2024) demonstrates that LLMs possess strong capabilities in natural language understanding and pattern recognition suitable for educational data assessment. To validate the feasibility of LLM-based evaluation, we conducted a preliminary study using a mainstream closed-source LLM. We compiled a test set of 100 samples and collected ratings from four annotators with graduate-level English proficiency. We then prompted the LLM to perform the same assessment. As shown in Table 2, the LLM’s

Dialogue Quality Evaluation

Unlike general-purpose dialogue systems, evaluation in spoken practice must align with specific educational objectives (Mendonça et al., 2024). Beyond contextual relevance, the model must actively encourage learner participation. We instruct the model to adopt a teacher persona employing a “Response + Expansion + Question” strategy: acknowledging the student’s input, expanding on the idea, and posing a follow-up question based on the current topic (Bowden et al., 2024). This ensures the 6

ranking alignment with human annotators is substantial, particularly in identifying lower-quality models where the rankings were identical. Based on these results, we selected GPT-5.1 as the judge model for our experiments.

Experiments

Non-fine-tuning Methods

Rel. ↑

Task ↑

Rich. ↑

Gui. ↑

Avg. ↑

Llama-3.1-8B GRPO + Rmodel + Rmul + Rsgl +Rmodel + Rmul

55.26 2.57 13.81 6.56 3.27 8.90

0.72 0.39 0.53 0.61 0.64 0.61

4.98 4.90 4.99 4.94 4.48 4.97

3.58 3.40 3.89 3.55 3.07 3.61

3.90 3.28 3.75 3.35 3.27 3.58

4.48 3.81 4.71 4.95 4.14 4.73

4.23 3.84 4.33 4.20 3.74 4.22

DDPO (Ours)

6.79

0.72

4.94

3.80

3.79

4.84

4.34

Analysis

7.1

Main Results Analysis

Table 3 presents the performance of different approaches across Qwen2.5-7B and Llama3.1-8B architectures. Limitations of Non-Fine-Tuning Methods Standard LLMs struggle to adhere to strict vocabulary constraints through prompting alone. As shown in the Qwen series, the Prompting method fails to effectively reduce the Vocabulary Violation rate (45.43%), performing similarly to the Base Model (41.69%). While Constrained Decoding successfully minimizes vocabulary violations (9.36%), it severely hampers generation quality. The rigid constraints can disrupt the model’s language modeling probability, leading to grammatical mistakes and unforeseen vocabulary errors, which accounts for the residual 9.36% violation rate. The average quality score drops to 4.00. indicating that rigid decoding constraints force the model into unnatural or error phrasing.

Supervised Fine-Tuning (SFT)

Supervised Fine-Tuning serves as a robust baseline for task-specific adaptation. We employ Qwen2.57B and Llama3.1-8B as backbone models to evaluate the generalizability of our approach across different architectures. To ensure sufficient adaptation, we construct a training set comprising 1,000 dialogue turns for each proficiency level. This process aims to align the models with the specific interactive requirements of spoken language practice. 6.3

Div. ↑

single-turn (Rsgl ) diversity rewards. The results of these ablation studies are presented in Table 4.

We first explore methods that do not require parameter updates. The first approach is Prompting, where the complete list of allowed vocabulary is explicitly included in the system prompt, relying on the model’s in-context learning capability to select appropriate words. The second approach is Constrained Decoding, which enforces strict output control. Following Tyen et al. (2022), we implement a dynamic constraint mechanism during inference. At each decoding step, the system maintains a set of valid next-tokens based on the current prefix and the allowed vocabulary, explicitly masking out OOV terms, special symbols, and tokens from other languages. 6.2

Quality (1-5)

Voc. ↓

Table 4: Ablation study results. DDPO strikes a favorable trade-off, ensuring both vocabulary adherence and diversity while delivering high-quality dialogue.

We investigate three distinct approaches for training spoken dialogue agents, ranging from inferencetime constraints to parameter-efficient tuning and reinforcement learning. 6.1

Objective

Method

Effectiveness of SFT Supervised Fine-Tuning (SFT) significantly improves upon the base model, achieving the highest Diversity scores (0.76 for Qwen, 0.74 for Llama). However, SFT alone is insufficient for strict pedagogical constraints. For instance, the Llama3.1-SFT model still retains a vocabulary violation rate of 19.90%, which is suboptimal for learners requiring precise difficulty matching.

Reinforcement Learning (RL)

In this task, the quality reward (Rqual ) is primarily designed as a rule-based vocabulary grading mechanism (detailed in Appendix A. Employing this reward in isolation constitutes our baseline method, referred to as GRPO. Additionally, we investigate the impact of incorporating a model-based quality scorer, which is trained on 1,700 pairs of human-annotated preference data. Regarding our proposed DDPO, we conduct ablation studies to isolate the specific contributions of its two core diversity components: the multi-turn (Rmul ) and

Superiority of DDPO Our proposed DDPO method achieves the best trade-off between constraints and quality. It matches the strict vocabulary control of Constrained Decoding (e.g., 9.36% on Qwen, 6.79% on Llama) while surpassing SFT in 7

The OOV of Llama 3.1-8B across different difficulty ranges

80% 70%

64.3%

62.4%

60%

significantly recovers diversity (0.61), proving essential for preventing repetitive loops. Interestingly, while the single-turn diversity reward (+Rsgl ) improves diversity (0.64), it negatively impacts dialogue quality, leading to a lower average score (3.74). We hypothesize that in its pursuit of diversity given a fixed dialogue history, the model may resort to incoherent word usage or random stacking, thereby degrading semantic quality. When both the model-based and multi-turn rewards are integrated, diversity does not increase further, but dialogue quality is notably restored.

Training Method Base Model SFT DDPO

52.9%

OOV (%)

50% 40%

40.2% 33.9%

30% 20%

18.6% 8.8%

10% 0%

15.2%

11.5% 5.8%

7.1%

5.5%

Figure 5: Proportion of dialogues with OOV. Compared with the base model and SFT method, the DDPO method significantly reduces the OOV at each level.

The Balance of DDPO The full DDPO configuration effectively integrates these conflicting objectives. By combining vocabulary constraints with both diversity rewards and the model reward, DDPO maintains a low violation rate (6.79%) and high diversity (0.72) while achieving the best average quality score (4.34). This demonstrates that the DDPO method effectively optimizes target rewards while successfully preventing entropy collapse.

overall dialogue quality (Average 4.30 and 4.34, respectively). Meanwhile, DDPO maintains relatively good diversity. Comparison of Different Training Methods As shown in Figure 5, the OOV ratio of Llama-3.18B is extremely high at every level. Moreover, as the difficulty level increases, the proportion of OOV gradually rises, with the highest reaching 64.3% at L4. After SFT training, the OOV ratios of all levels—except Level 3 (L3)—drop sharply, yet they still remain above 10%. Following DDPO training, the OOV ratios of all levels are reduced to below 10%, and the optimization effect on L3 is particularly significant. 7.2

Conclusion

In this work, we address the critical need for proficiency-aligned oral English practice for K-12 students in non-native contexts. Using China’s educational standards as a representative framework, we constructed comprehensive graded vocabulary, and dialogue datasets to facilitate this task. Our primary contribution is the proposal of DDPO (Diversity Driven Policy Optimization), a reinforcement learning framework designed to balance strict pedagogical constraints with dialogue diversity. Our experiments demonstrate that while standard methods like Constrained Decoding or SFT struggle to simultaneously satisfy vocabulary limitations and generation diversity, DDPO effectively aligns Large Language Models with these conflicting objectives. It achieves precise proficiency control comparable to hard constraints while maintaining the naturalness and semantic richness of open-ended generation. This approach proves robust across different model architectures (Qwen and Llama), highlighting its potential as a generalizable solution for educational dialogue systems. In future work, we plan to extend our framework to other languages and curricula, and explore the integration of multimodal capabilities, such as speech recognition and prosody analysis, to provide a more immersive language learning experience.

Ablation Study

We conducted an ablation study on the Llama3.18B model to investigate the specific contribution of each reward component within the DDPO framework. The detailed results are presented in Table 4. Impact of Hard Constraints (GRPO Baseline) The baseline GRPO model, optimized solely with vocabulary constraints, achieves the lowest vocabulary violation rate (2.57%). However, this comes at a significant cost to diversity (0.39) and semantic richness (3.28), resulting in the lowest average quality score (3.95). This confirms that optimizing for constraints alone leads to repetitive and safe but unengaging responses; specifically, the GRPO algorithm, when relying solely on rule-based quality rewards, exhibits a distinct entropy collapse phenomenon. Incorporating a model-based reward scorer effectively alleviates quality issues; however, diversity remains suboptimal (0.53), and we observe a moderate increase in OOV violations. Trade-offs in Diversity and Quality Rewards Adding the multi-turn diversity reward (+Rmul ) 8

Limitations

languages (cefr) as a tool to unite nations. Continuing Professional Education: Theory and Practice, 79(2):77–88.

Despite the promising results of DDPO in the field of spoken language education, we acknowledge several limitations in our current work:

Kevin Bowden, Yue Fan, Winson Chen, Wen Cui, Davan Harrison, Xin Eric Wang, and Marilyn Walker. 2024. Active listening: Personalized question generation in open-domain social conversation with user model based prompting. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14120–14157, Miami, Florida, USA. Association for Computational Linguistics.

• Reliance on User Simulators: Our training and evaluation primarily rely on a user simulator. While designed to mimic learner behavior, it cannot fully capture the unpredictability, emotional nuances, and diverse error patterns of real human students. Future research should incorporate methods based on user modeling to validate effectiveness in real-world scenarios.

Andrew Caines, Helen Yannakoudakis, Helena Edmondson, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2020. The teacher-student chatroom corpus. In Proceedings of the 9th Workshop on NLP for Computer Assisted Language Learning, pages 10–20, Gothenburg, Sweden. LiU Electronic Press.

• Computational Overhead: The DDPO framework requires sampling multiple trajectories (G groups) and calculating diversity rewards across them. Compared to SFT or basic GRPO implementations, this incurs a higher computational cost during the training phase.

Itsugun Cho, Ryota Takahashi, Yusaku Yanase, and Hiroaki Saito. 2024. Deep reinforcement learning with hierarchical action exploration for dialogue generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4566–4579.

• Model Scale and Generalization: Our experiments were conducted on the Llama-3.1-8B model. While DDPO has demonstrated effectiveness at this scale, we have not yet explored its performance on larger-scale models (e.g., 70B parameters) or different model architectures.

David Coniam, M Milanovic, and W Zhao. 2022. Cefr and cse comparability study: An exploration using the chinese college english test and the languagecert test of english. CEFR Journal—Research and Practice, 5:25–44. Joshua Danish, Asmalina Saleh, Alejandro Andrade, and Branden Bryan. 2017. Observing complex systems thinking in the zone of proximal development. Instructional Science, 45:5–24.

• Static Vocabulary Constraints: The graded vocabulary lists used in the current educational setting are static. Consequently, they may not fully accommodate compound words or morphological variations (e.g., prefixes and suffixes), potentially treating valid derivations as unknown words. Future work needs to investigate the impact of dynamic graded vocabularies.

Bingyu Dong, Jie Bai, Tao Xu, and Yun Zhou. 2024. Large language models in education: A systematic review. In 2024 6th International Conference on Computer Science and Technologies in Education (CSTE), pages 131–134. IEEE. Huifang Du, Shuqin Li, Minghao Wu, Xuejing Feng, Yuan-Fang Li, and Haofen Wang. 2024. Rewarding what matters: Step-by-step reinforcement learning for task-oriented dialogue. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8030–8046, Miami, Florida, USA. Association for Computational Linguistics.

References Ahmad Zaki Amiruddin, Mohammad Taufiq Abdul Ghani, Wan Ab Aziz Wan Daud, Nur Hafezah Hussein, Nur Hazimah Mat Yamin, and Yasmin See Abdullah. 2025. Empowering language learners: Innovations in cefr-based language learning applications. Edelweiss Applied Science and Technology, 9(2):352– 359.

Yingming Gao, Baorian Nuchged, Ya Li, and Linkai Peng. 2023. An investigation of applying large language models to spoken language learning. Applied Sciences, 14(1):224. Jeroen Geertzen, Theodora Alexopoulou, Anna Korhonen, and 1 others. 2013. Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat). In Proceedings of the 31st Second Language Research Forum. Somerville, MA: Cascadilla Proceedings Project, pages 240–254.

Marina Bacher, Sabrina Bacher, and Kateryna Binytska. 2024. A didactical approach to fostering language sensitivity in multilingual classroom settings: The common european framework of reference for

Dominik Glandorf, Peng Cui, Detmar Meurers, and Mrinmaya Sachan. 2025. Grammar control in dialogue response generation for language learning chatbots. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 9820– 9839, Albuquerque, New Mexico. Association for Computational Linguistics.

erence for languages. The Asia-Pacific Education Researcher, 31(6):667–677. Kun Qian, Ryan Shea, Yu Li, Luke Kutszik Fryer, and Zhou Yu. 2023. User adaptive language learning chatbots with a curriculum. In International Conference on Artificial Intelligence in Education, pages 308–313. Springer. Jeremy Redlich and Steven Pattison. 2024. Combining cefr and 4c clil frameworks for principled literature in language teaching. NOBEL: Journal of Literature and Language Teaching, 15(1):1–21.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

Desta Haileselassie Hagos, Rick Battle, and Danda B Rawat. 2024. Recent advances in generative ai and large language models: Current status, challenges, and perspectives. IEEE Transactions on Artificial Intelligence.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.

Nadezhda Stanislavovna Lagutina, Kseniya Vladimirovna Lagutina, Anastasya Mikhailovna Brederman, and Natalia Nikolaevna Kasatkina. 2023. Text classification by cefr levels using machine learning methods and bert language model. Modeling and Analysis of Information Systems, 30(3):202–213.

Gladys Tyen, Mark Brenchley, Andrew Caines, and Paula Buttery. 2022. Towards an open-domain chatbot for language practice. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 234–249.

Shuning Li. 2024. The challenges and dilemmas of spoken english education in the context of the compulsory education system in china. Transactions on Social Science Education and Humanities Research, 11:629–636.

Satoru Uchida, Yuki Arase, and Tomoyuki Kajiwara. 2024. Profiling English sentences based on CEFR levels. ITL-International Journal of Applied Linguistics (Belgium), pages 103–126.

Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995, Taipei, Taiwan. Asian Federation of Natural Language Processing.

Jingxuan Wu and Carsten Roever. 2021. Proficiency and preference organization in second language mandarin chinese refusals. The Modern Language Journal, 105(4):897–918. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading esol texts. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 180–189.

Jionghao Lin, Zifei Han, Danielle R Thomas, Ashish Gurung, Shivang Gupta, Vincent Aleven, and Kenneth R Koedinger. 2024. How can i get it right? using gpt to rephrase incorrect trainee responses. International journal of artificial intelligence in education, pages 1–27.

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, and 1 others. 2025. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.

Yun Long, Haifeng Luo, and Yu Zhang. 2024. Evaluating large language models in analysing classroom dialogue. npj Science of Learning, 9(1):60.

Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2023. A survey of controllable text generation using transformer-based pre-trained language models. ACM Computing Surveys, 56(3):1– 37.

John Mendonça, Isabel Trancoso, and Alon Lavie. 2024. Soda-eval: Open-domain dialogue evaluation in the age of LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11687– 11708, Miami, Florida, USA. Association for Computational Linguistics.

Jixiao Zhang and Chunsheng Zuo. 2025. Grpo-lead: A difficulty-aware reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint arXiv:2504.09696.

Chuan Peng, Jianda Liu, and Hongwen Cai. 2022. Aligning china’s standards of english language ability with the common european framework of ref-

B Main Training and Testing Parameters

Algorithm 1 Vocabulary-Constraint Quality Reward Function

Training Settings

Require: R: list of responses; L: target levels (L1-L4); V: graded vocabulary dictionary Ensure: S: list of reward scores 1: for each (r, l) in (R, L) do 2: Initialization: Nwords ← 0, Ntarget ← 0, F lagviolation ← False 3: Preprocessing: Tokenize r, remove punctuation. 4: for each word w in r do if w is entity (NER), number, or spoken filler then 5: 6: continue 7: end if 8: wlemma ← Lemmatize(w) 9: Nwords ← Nwords + 1 10: if wlemma ∈ / V or Level(wlemma ) > l then 11: F lagviolation ← True 12: end if 13: if Level(wlemma ) == l then 14: Ntarget ← Ntarget + 1 15: end if 16: end for 17: Hard Constraint Check: if sentences count ≤ 1 or ’?’ ∈ / r or count(’?’) ≥ 2 18: or contains non-English then 19: score ← 0.0 20: else 21: Define length range [min, max] based on l: 22: L1: [10, 15], L2: [10, 20], L3/L4: [20, 30] 23: if Nwords ∈ [min, max] and not F lagviolation then 24: if l == L1 then 25: score ← 0.8 26: else 27: score ← 0.5 + min(Ntarget × 0.15, 2.0) 28: end if 29: else 30: score ← 0.2 {Soft penalty for length/vocab violation} 31: end if 32: end if 33: Append score to S 34: end for 35: return S

Qwen • SFT: Full-parameter fine-tuning, learning rate of 2 × 10−5 , warm-up ratio of 0.05, trained for 2 epochs. • DDPO: Learning rate of 3.0×10−6 , warm-up ratio of 0.1, trained for 1 epochs. LLaMA • SFT: Full-parameter fine-tuning, learning rate of 2 × 10−5 , warm-up ratio of 0.05, trained for 2 epochs. • DDPO: Learning rate of 3.0×10−6 , warm-up ratio of 0.1, trained for 1 epochs. Testing Settings For both the generation model and the scoring model, the temperature is set to 0.7 and the beam size is set to 1. For each input, eight samples are collected to be used in diversity calculation.

Table 5 details the four proficiency levels (L1–L4) defined in our dataset, ranging from basic recognition to complex logical expression.

Wen Zhao, Boran Wang, David Coniam, and Bingxue Xie. 2017. Calibrating the cefr against the china standards of english for college english vocabulary education in china. Language Testing in Asia, 7:1– 18. Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. 2023. Controlled text generation with natural language instructions. In International Conference on Machine Learning, pages 42602–42613. PMLR. Aihua Zhu, Samah Ali Mohsen Mofreh, and Sultan Salem. 2023. The effect of goal setting based on china’s standard of english proficiency scales on reading achievement. Sustainability, 15(5):4519.

Proficiency Levels

Level

Description

Can understand and use simple words and phrases in daily life. Recognizes basic information (colors, numbers, greetings), reads short texts with pictures, introduces oneself, and writes basic words. Uses simple grammar for basic communication.

Can understand slow, simple conversations and stories with support. Reads short texts to extract main ideas. Describes familiar people and events; expresses feelings; uses basic tenses and polite expressions. Pronunciation is mostly correct.

Can understand complex daily conversations and broadcasts. Reads and analyzes short articles to infer logic. Communicates coherently on familiar topics, expresses opinions, and participates in discussions using a range of tenses and structures.

Can understand multi-person conversations, news, and lectures. Reads various text types to distinguish facts from opinions. Expresses ideas fluently and logically, adapts to different situations, and uses complex grammar and vocabulary accurately.

Table 5: Detailed descriptions of the four English proficiency levels used in the task.

Vocabulary Reward

Dialogue Model System Prompt

The system prompt used for the dialogue agent (Anna) is presented below. Variables enclosed in curly braces (e.g., {level}) are dynamically filled based on the specific user profile and task configuration.

Table 7 illustrates a comparison of responses generated by the Base model, SFT model, and our DDPO-tuned model. The example highlights how the DDPO model adheres to vocabulary constraints (Level 3) while maintaining conversation flow, whereas baseline models introduce Out-OfVocabulary (OOV) terms.

System Prompt for Dialogue Agent Configuration • Difficulty Level: {level} • Role: Anna (24-year-old spoken English teacher) • Student Ability Reference: {Ability} Role Goals 1. Engage in an English conversation with a student at {grade_level} on the topic: {topic}. 2. In each turn, summarize/expand on the previous input and ask a follow-up question. Task Constraints 1. Length & Vocab: Keep answers within {word_count} words. Use vocabulary suitable for {level}. 2. Grammar Constraint: You may only use the following grammar: {grammar}. Do not use unlisted grammar. 3. Adaptability: Match the student’s ability level—neither too difficult nor too simple. Output Requirements 1. Structure: Response must include a Reaction (respond/expand) and a Question (open-ended follow-up). 2. Guidance: If the conversation strays, guide the student back to the topic.

Case Study: Responses from Different Models

G Detailed Case Study: Entropy Collapse To provide a concrete illustration of the entropy collapse phenomenon discussed in Section 4.1, Table 8 presents a comparison of multiple responses generated by the Base Model, GRPO, and DDPO given the same dialogue context. We generated 8 independent samples for each model using the same temperature setting (T = 1.0). • Base Model: Shows natural diversity but lacks specific pedagogical optimization. • GRPO (Baseline): Exhibits severe entropy collapse. As highlighted in red, the model converges to a single, repetitive pattern: it simply restates the user’s input ("You drink water with your lunch") to satisfy length constraints safely, followed by a repetitive question structure. The semantic diversity is near zero. • DDPO (Ours): Successfully maintains high diversity in phrasing (e.g., "Great", "Wonderful", "Fantastic") and sentence structure while adhering to the educational constraints.

Use of AI Tools

We utilized AI-based tools (e.g., ChatGPT, Gemini) to assist in refining the clarity, grammar, and readability of the manuscript. The conceptualization, experimental design, data analysis, and scientific conclusions remain the sole work and responsibility of the authors. All AI-suggested modifications were reviewed and verified by the authors.

Dialogue Quality Evaluation Prompt

Table 6 presents the detailed system prompt used for the model-based subjective evaluation. The evaluator (e.g., GPT-4) is instructed to act as a rigorous dialogue quality assessor using a 5-point Likert scale across eight specific dimensions. Crucially, these evaluation criteria are identical to the guidelines provided to our human annotators to ensure consistency. 12

System Prompt for Subjective Evaluation You are a rigorous dialogue quality evaluator. Your task is to evaluate the performance of an English spoken dialogue model using a 5-point Likert scale (1: Very Poor to 5: Excellent) and provide a brief justification for each score. Input Data: 1. Dialogue Context (History); 2. User Input; 3. Model Response (Target). General Constraints: • Format: The response must follow: Summary/Extension + Question. The summary must precede the question. • Style: Natural spoken English. No written conventions (e.g., brackets for explanation, "A/B" notation). • Scoring Logic: Start with the Base Score, apply specific deductions/additions, and determine the final score (Min: 1, Max: 5). Evaluation Dimensions & Criteria: 1. Topic Relevance (Base: 5, Deduction) • -1: Led astray by user or slight digression. • -2: Does not align with topic. -3: Completely off-topic. 2. Task Completion (Base: 3, Add/Deduct) • -1: No question asked, wrong length, or prompt violation. • -2: Non-human symbols. -4: Chinese content. • +1: Includes summary/extension. +2: Excellent extension. 3. Information Richness (Base: 3, Addition) • 3: Basic question only. • 4: 1–2 points of extra info. 5: ≥3 points of extra info. 4. Topic Guidance (Base: 3, Addition) • 3: Yes/No question. • 4: Fact-based question (item, color, etc.). • 5: Question requiring thought/elaboration. Output Format: Output strictly in JSON format: {"reason_consistency": "...", "score_consistency": int, ...}

Table 6: The full prompt used for LLM-based subjective evaluation. The prompt defines specific base scores and calculation rules for eight dimensions.

Case Study: Responsibility and Growth (Level 3) Context: Teacher: Interesting observation. Do you think practice can help improve how we deal with these unexpected hurdles? Student: Yes, practice can definitely improve our ability to face and solve them. Teacher: Great point! Can you think of a time when you had to overcome a challenge that made you feel more confident and independent? Student: Yes, like when I finished a big project alone. I felt very proud and grown up. Model Responses: 1. 7B Base Model “Finishing a big project alone is a huge accomplishment. How did taking on that challenge and succeeding make you feel about your own growth and capabilities?” OOV Errors: accomplishment, growth, capabilities (Level > L3) 2. 7B SFT Model “Proud and independent, those are wonderful feelings! How do you think this experience will influence your future approach to similar challenges?” OOV Error: approach (Level > L3) 3. 7B DDPO (Ours) “Fantastic connection! Feeling proud makes a big difference. How does feeling proud make overcoming challenges feel more successful to you?” OOV Error: None

Table 7: Comparison of model responses. The Base and SFT models fail to strictly follow the vocabulary constraints, generating OOV words (marked in red). The DDPO model successfully generates a coherent response within the L3 vocabulary limit.

Context History: [Tutor: "Great! What do you usually have for lunch?", Student: "Rice, vegetables, and meat.", Tutor: "Delicious! Do you drink water with your lunch?"] User Input: "Yes, I drink water with lunch." Model

Sampled Responses (8 independent generations)

Base Model

1. Good habit! Do you like eating fruits after lunch? 2. Good habit! Do you eat rice for breakfast too? 3. Good for you! Do you eat rice, vegetables, and meat for dinner too? 4. Good habit! What’s your favorite meat to eat with rice and vegetables? 5. Good habit! Do you like eating fruit after lunch? 6. Good habit! Do you like eating fruits after lunch? 7. Good habit! Do you like eating fruits with your lunch too? 8. Good for you! Do you like eating fruits with your meals? 1. You drink water with your lunch. Do you eat rice with your lunch? 2. You drink water with your lunch. Do you eat rice with your lunch? 3. You drink water with your lunch! Do you eat meat for lunch? 4. You drink water with your lunch! Do you eat rice with your lunch? 5. You drink water with your lunch! Do you eat rice with your lunch? 6. You drink water with your lunch! Do you eat meat with your lunch? 7. You drink water with your lunch. Do you eat rice for lunch? 8. You drink water with your lunch! Do you eat rice with your lunch? 1. Great! Eating vegetables makes food taste nice! What kind of food do you like? 2. Wonderful! Water makes eating feel good! What kind of drinks do you have? 3. Wonderful! Water helps make food taste good! What makes food so nice? 4. Great! Water makes eating feel good! What kind of drinks do you have? 5. Wonderful! Water makes eating very nice! What kinds of drinks do you have? 6. Great! Water helps make food taste better! What kind of food do you like? 7. Fantastic! Water makes eating feel good! What kind of food do you like? 8. Great! Water makes eating very nice! What kind of drinks do you have?

GRPO

DDPO (Ours)

Table 8: Comparison of generation diversity. The GRPO baseline collapses into a repetitive template (repeating user input + simple question), whereas DDPO generates diverse feedback and follow-up questions.

Record · ID 134585 · SHA-256 47a38e66ab8efacc
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.