arXiv:2606.07226v1 [cs.LG] 5 Jun 2026
DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios Tongzhou Yu∗
Mingjia Li∗
Hong Qian†
[email protected] Nanjing University Nanjing, China Shanghai Innovation Institute Shanghai, China
[email protected] East China Normal University Shanghai, China
[email protected] East China Normal University Shanghai, China Shanghai Innovation Institute Shanghai, China
Wenkai Wang
Zongbao Zhang
Yaoyu Jiang
[email protected] East China Normal University Shanghai, China
[email protected] East China Normal University Shanghai, China Shanghai Innovation Institute Shanghai, China
[email protected] East China Normal University Shanghai, China
Xiangfeng Wang
Aimin Zhou
Jiajun Guo
[email protected] East China Normal University Shanghai, China
[email protected] East China Normal University Shanghai, China
[email protected] East China Normal University Shanghai, China Shanghai Innovation Institute Shanghai, China
Abstract Human creativity has emerged as a critical competency in the era of large language models. Assessing creativity in complex, openended environments is a grand challenge in data mining, currently hindered by a reliance on standardized simple tasks and the scarcity of fine-grained expert data. As an ecologically valid assessment context, debate reflects multiple dimensions of creativity, encompassing both divergent thinking and convergent thinking. Moreover, debate is a data-rich domain, with a large volume of publicly accessible materials. Current mainstream automated scoring methods are poorly suited to complex settings such as debate, and therefore still rely on costly human evaluation. To this end, this paper proposes DEFINED, a data-efficient computational framework for fine-grained creativity assessment in debate scenarios. DEFINED operationalizes debate creativity through a hierarchical eight-dimensional metric system, implemented via a pre-trained autoregressive language model with a hierarchical scoring head that supports both fine-grained and coarse-grained evaluation. Statements and their associated expert scores were obtained from authentic debate competitions, and a constrained data augmentation strategy was employed to address the elite bias inherent in the original data. DEFINED adopts a mixedgranularity training strategy enabling robust learning from limited fine-grained supervision annotated by trained graduate experts. ∗ These authors contributed equally to this work. † Corresponding author.
KDD ’26, Jeju Island, Republic of Korea 2026. ACM ISBN 979-8-4007-2259-2/2026/08 https://doi.org/10.1145/3770855.3817874
To rigorously validate ecological validity beyond synthetic benchmarks, we incorporate an empirical study with debate-naive participants, utilizing these authentic data to serve as a qualitative case study for mid-to-low proficiency populations. Across our evaluation protocol, our scoring model achieves accurate and stable scoring, outperforming prompt-based large language model evaluators and existing debate scoring methods, while mitigating common failure modes observed in current approaches. The code for DEFINED is available on GitHub at https://github.com/tzwo/DEFINED.
CCS Concepts • Applied computing → Education; • Computing methodologies → Machine learning.
Keywords Creativity assessment, Data-efficient learning, Automated scoring, Debate analysis, Intelligent education, Large language models
1
Introduction
Creativity is widely recognized as a pivotal competence for the 21st century, serving as the engine for complex problem-solving and societal innovation [19, 29, 40]. The assessment of creativity constitutes a foundation for research in creativity. Consequently, the accurate and scalable assessment of creativity has become a central pursuit in psychology and education [50]. Traditional assessment paradigms primarily rely on Divergent Thinking (DT) tasks, such as the Alternative Uses Task (AUT) [26] or the Remote Associates Test (RAT) [25], and self-report questionnaires, like the Creative
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
Achievement Questionnaire [5] and the Inventory of Creative Activities and Achievements [17]. While foundational, these methods face significant ecological validity challenges. Questionnaires are susceptible to social desirability bias and affective interference, often failing to reflect actual performance [31, 32]. Similarly, standard DT tasks, often conducted in decontextualized laboratory settings, struggle to capture the situated nature of creativity required in real-world environments [48, 49], where individuals must navigate complex constraints and adversarial dynamics. To bridge the gap between psychometric assessment and realworld creative capability, debate serves as an ideal proxy for assessing ecological creativity. At its core, argumentation is not merely a linguistic exercise but a dynamic integration of creative and critical thinking [10]. Debate demands that participants generate novel arguments (divergent thinking) while adhering to logical coherence and rebuttal constraints (convergent thinking) under time pressure [38, 45]. More importantly, this perspective aligns with process-analytic models of creativity, which posit that creative problem-solving involves a cycle of core processes, including problem construction, information encoding, idea generation, and idea evaluation [27, 28]. Debate naturally instantiates this entire cycle under time pressure, rendering it a well-motivated task paradigm for the present study. However, transitioning from theoretical appreciation to computational assessment in debate scenarios presents three formidable challenges. First, the assessment bottleneck: manual evaluation of creativity in debate is notoriously labor-intensive and expensive. A single fine-grained annotation for a long-form debate statement averages over 40 minutes for a trained expert, rendering largescale annotation computationally intractable [4, 20, 47]. Second, the “granularity gap” in existing resources: current computational argumentation datasets primarily focus on holistic outcomes (e.g., win/loss labels or general quality scores) rather than the specific dimensions of creativity [6, 18, 35]. Winning a debate does not always equate to being creative; conflating these metrics obscures the cognitive mechanisms at play. Third, data distribution bias: existing corpora are dominated by professional debaters (“elite” samples), lacking the mid-to-low proficiency examples typical of student populations or the general public [6, 18, 35]. This domain shift severely hinders the deployment of models in educational settings. In this work, we introduce DEFINED: a data-efficient computational framework designed to automate fine-grained creativity assessment in debate scenarios. We hypothesize that expert adjudicator evaluation is not a monolithic process but a composite judgment derived from distinct cognitive dimensions. The scoring model of DEFINED takes debate statements with context as input and produces both eight-dimensional fine-grained scores and a coarse-grained debate score aligned with human overall judgments. The contributions of the proposed DEFINED framework are threefold: (1) We collect authentic competition statements data scored by top-tier expert adjudicators of Mandarin debate, which serve as the ground truth reflecting human cognitive judgment. To mitigate the elite bias inherent in authentic competition data and to better cover debate statements across a wide range of proficiency levels, we further adopt a
Tongzhou Yu et al.
data augmentation strategy under triple constraints, thereby enhancing the model’s generalization capability. (2) This paper proposes an eight-dimension metric system, decoupling assessment into five creativity-specific dimensions (including divergent thinking and convergent thinking) and three debate-related (non-creativity) dimensions. Building on this foundation, we design a set of detailed scoring rubrics to serve as instructions to human annotators. Based on these scoring rubrics, a limited amount of fine-grained annotation data is produced. By learning the contribution of these dimensions, our scoring model attempts to reverseengineer the cognitive process of expert adjudicators. (3) This work introduces a mixed-granularity training strategy to overcome the scarcity of fine-grained labels. The extensive experiments show that by leveraging only 60 samples of fine-grained expert annotations alongside 4,000 coarsegrained samples, our scoring model achieves high-precision prediction, effectively solving the ”small data” problem in complex psychometric modeling. We also establish a threemodular evaluation protocol to verify system robustness across varying proficiency levels (High vs. Mid-Low) and annotation granularities (Fine vs. Coarse). The experimental results provide comprehensive evidence that the scoring model in DEFINED predicts scores accurately and aligns with the underlying cognitive processes of human assessment.
2
Related Work
In research on automated creativity assessment, early computational approaches are predominantly grounded in semantic distance, which provides theoretical and practical foundations for assessing creative output. Rooted in associative theories of creativity, these methods operationalize originality by computing vector distances between textual concepts in high-dimensional semantic spaces, under the assumption that greater semantic distance reflects higher creative potential. Such approaches have demonstrated empirical validity in classical divergent thinking tasks including verb generation, particularly for short and decontextualized responses [14, 34]. Subsequent approaches have incorporated modern word embedding models such as Word2Vec and GloVe, which demonstrate improved performance over traditional latent semantic analysis (LSA) methods [3]. Some recent studies have begun to emphasize the importance of context-based creativity assessment [46]. However, classical semantic models rely on static word representations and thus lack contextual sensitivity, rendering them incapable of distinguishing polysemous meanings across discourse contexts. Moreover, these methods exhibit intrinsic deficiencies when applied to long-form text: for example, LSA underestimates semantic distance as the length of response increases, a bias that runs counter to human judgments of creativity [9]. As a result, semantic-distance-based measures struggle to capture the complex, multi-word semantic interactions inherent in context-dependent creative tasks [30]. To overcome these limitations, fine-tuned large language models (LLMs) have emerged as one of the most human-aligned solutions for automated creativity and text quality evaluation. By attaching
DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
(a) Data construction and augmentation pipeline Coarse-grained data
Annotation is labor-intensive
Original Coarse-grained debate data
Focus on holistic outcomes
NULL response
Characters of divergent proficiency
Existing matches has elite bias
Eight-dimensional metric system for debate Originality
as prompt
Summary variant
<|Stance|> <|PRO_opening|>
Statement
Augmentation
Flexibility
Pair LLM-as judge -a-Judge
Fluency
Divergent
as example Expert coarse scoring from existing competitions
high
Clarity Apeal
Convergent
Effectiveness
Noncreativity
Fine-grained data
proficiency range
(b) Scoring model architecture Background <Topic> <Stance> <PRO_opening> <CON_opening>
Opponent's speech
Hierarchical scoring head
RMSNorm
Template
data
Logicality Relevance
Creativity-specific dimensions
<|CON_opening|>
Mixed
Smaple from each score range Expert fine-scoring
off-topic replies
Accumulate Authentic matches Background Opponent's <|Topic|> speech
low
Qwen2.5-7BInstruct tokennizer
“Thank”
“you”
“Mr”
“Chairman” ...
GQA
]
“...”
Statement
w2
...
w4
w3
LoRA
[
... RMSNorm SwiGLU MLP
w1
...
Fine-grained score
w5
Coarse score
w6 w7 w8
Figure 1: An overview of the proposed DEFINED framework. (a) Data construction and augmentation pipeline. We construct a mixed-granularity dataset by augmenting authentic high-proficiency debate competitions with synthetic samples generated under a Triple-Constraint Strategy (spanning diverse proficiency levels, noise injection, and summarization variants). A small subset is rigorously annotated by experts to provide fine-grained supervision, while the majority utilizes coarse-grained labels. (b) Scoring model architecture. The scoring model employs a pre-trained LLM (Qwen2.5-7B-Instruct) as a context-aware semantic encoder. The latent representation feeds into a specialized Hierarchical Scoring Head, which first predicts scores for eight fine-grained dimensions and subsequently aggregates them into a holistic debate score via weighted summing. task-specific scoring heads to LLMs and fine-tuning them on largescale datasets paired with human annotations, these approaches learn the latent structure of human evaluation [8]. Recent finetuned models such as Themis [16] and M-Prometheus [33] further demonstrate the effectiveness of supervised learning for text assessment. Across tasks such as AUT and creative problem solving, fine-tuned LLMs have achieved correlations with human judgments exceeding 0.7, substantially outperforming semantic-distance-based approaches [24, 30]. Nevertheless, the principal drawback of this paradigm lies in its heavy reliance on large volumes of high-quality human annotations. Constructing such datasets entails significant financial and temporal costs, thereby limiting scalability and hindering rapid adaptation to new creative tasks or applications in low-resource settings. Prompt-engineering-based LLM evaluation methods have consequently been proposed as a cost-effective alternative. By embedding expert-designed scoring rubrics which explicitly define dimensions into structured prompts, these approaches enable zero-shot or fewshot creativity assessment without task-specific training data, offering clear advantages in speed and efficiency [30]. Empirical studies report that few-shot prompted LLMs achieve moderate correlations with human ratings (approximately 0.6) on creative problem-solving
tasks, in some cases outperforming traditional semantic-distance measures while underperforming fine-tuned models [24]. Recent advancements in the automated evaluation of debate statement quality also adopt this method [21, 43]. However, such methods remain fundamentally constrained by intrinsic biases of LLMs (including verbosity bias, self-enhancement bias) and by instability in score calibration [51]. These limitations cannot be fully mitigated through prompt design alone and pose serious challenges for high-stakes applications such as talent selection or educational assessment, where fairness, objectivity, and reliability are essential.
3 The Proposed DEFINED Framework 3.1 Data Collection and Preprocessing Naturalistic Debate Corpus Construction. To establish a highfidelity benchmark for ecological creativity, we collect authentic debate data from four consecutive editions (2022–2025) of Xin Guo Bian, a premier competitive debating tournament in Mandarin debate. The tournament’s adjudication standards exhibit high interrater reliability, serving as stable ground truth for high-quality
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
argumentation. We operationalize the assessment task as a singleround statement evaluation: each data instance 𝑥 comprises contextual history 𝐶 (debate motion, stance, opening statements, and the opponent’s preceding argument) and focal statement 𝑟 . The corpus spans 56 distinct motions and 108 competitions, yielding 706 valid history-statement pairs obtained via high-fidelity audio transcription, with a total amount of 3,257,458 Chinese characters. Ground Truth Acquisition. For each statement, coarse-grained holistic scores are assigned by 3–5 human annotators on a 0–10 scale. To mitigate varying leniency biases among adjudicators, raw scores were standardized (𝑧-score normalization) and subsequently mapped to a high-proficiency interval. This authentic, highperforming subset is denoted as 𝐷 high , representing the effective upper bound of debate performance in the target domain.
3.2
Data Augmentation Strategy Under Triple Constraints
Authentic debate datasets inherently suffer from elite bias, lacking representation of the mid-to-low proficiency spectrum typical of general educational settings. To bridge this distribution gap, we construct a synthetic dataset 𝐷 simulate using a LLM under a proposed Triple-Constraint Augmentation Strategy. Exemplar-Based Stratified Generation. For each target score interval 𝑦 ∈ [𝑦low, 𝑦mid ], we design prompt templates 𝑝 𝑦 encapsulating specific competency deficiencies (e.g., logical gaps, redundancy, detailed in Appendix B). Conditioned on the context 𝐶 and referencing exemplars in 𝐷 high , the generator (Qwen2.5-72B-Instruct [2]) outputs candidate statements 𝑟ˆ = 𝐺 (𝐶, 𝑝 𝑦 ). Pairwise Preference Alignment. Prior work has shown relative ranking judgments are generally more stable and reliable than absolute scoring in LLM evaluation [1, 22]. Thus, to rectify potential misalignment between generated content and target labels, we introduce a pairwise constraint. Generated samples are subjected to LLM-based pairwise comparison against anchor samples. If the inferred quality ranking 𝑟𝑎𝑛𝑘 (𝑟ˆ𝑖 , 𝑟ˆ𝑗 ) contradicted the assigned pseudo-labels, score swaps are applied to ensure label consistency. In addition, we conducted a small-scale human validation: expert annotators inspected 10 groups of synthetic samples, and the corrected rankings were found to align well with human judgments. Noise Injection and Contextual Variation. We further enrich the data by introducing negative samples 𝐷 neg (e.g., null responses) and a summarization variant 𝐷 summary , where the opponent’s statement is logically compressed by Qwen2.5-7B-Instruct to force the model to attend to core arguments. The final training set 𝐷 train combines 𝐷 high , 𝐷 simulate , 𝐷 neg , and 𝐷 summary , followed by a self-correction pass where a preliminarily trained model re-scores a subset of noisy samples to improve label robustness [39].
3.3
Eight-Dimensional Metric System
Psychometric System Definition. We synthesize a hierarchical evaluation protocol bridging the process-analytic model of creative problem-solving [27, 28] and computational debate metrics [37, 42]. The system decouples assessment into five Creativity-Specific Dimensions: divergent thinking (Fluency, Originality, Flexibility)
Tongzhou Yu et al.
and convergent thinking (Logicality, Relevance), and three NonCreativity Dimensions: Effectiveness, Clarity, Appeal. We provide the detailed definition in Appendix A. Based on this metric system, we curate a fine-grained dataset 𝐷 fine (𝑁 = 120) balanced across 𝐷 high and 𝐷 simulate , annotated by ten trained graduate experts (Cronbach’s alpha = .90-.98). This subset provides the supervision signal for the multi-dimensional score.
3.4
The Architecture of Scoring Model in DEFINED
Overview. The scoring model of DEFINED is designed to give both fine-grained dimensional scores and a holistic quality score for debate. The architecture consists of a semantic encoder based on a pre-trained autoregressive language model and a hierarchical scoring head. Context-Aware Semantic Encoder. In DEFINED, the autoregressive language model serves as the semantic encoder, responsible for mapping long-form inputs in debate into high-dimensional hidden representations. We adopt Qwen2.5-7B-Instruct as backbone model. The model input is constructed by concatenating background information and the current-round statement according to a fixed template, yielding a single sequence of text. This design enables the model to explicitly exploit the full debate context when encoding the current statement, thereby capturing dependencies among arguments and their adversarial dynamics. The concatenated text is tokenized using the Qwen2.5-7B-Instruct tokenizer at the subword level and converted into a token sequence 𝑥 = (𝑥 1, 𝑥 2, . . . , 𝑥𝑇 ), where 𝑇 denotes the sequence length. The input token sequence is first mapped into a continuous vector space through an embedding layer, producing the representation ℎ 0 ∈ R𝑇 ×𝑑 , where 𝑑 is the hidden dimensionality of the model. The model consists of a stack of Transformer decoder layers. The hidden state update at the 𝑙-th layer can be formalized as: ℎ𝑙 = TransformerLayer (𝑙 ) (ℎ𝑙 −1 ) ,
(1)
where each layer comprises a multi-head self-attention mechanism and a feed-forward network (FFN), combined with residual connections and layer normalization. The self-attention mechanism allows the model to perform weighted aggregation over tokens at different positions in the sequence, thereby modeling argumentative structure, semantic coherence, and contextual dependencies. After the final Transformer decoder layer, the model outputs the sequence of hidden states: 𝐻 = (ℎ 1, ℎ 2, . . . , ℎ𝑇 ) ∈ R𝑇 ×𝑑 .
(2)
In this work, we take the hidden state of the final token ℎ𝑙𝑎𝑠𝑡 = 𝐻 [𝑇 , :] in the sequence as a global representation, which serves as a high-level semantic feature for subsequent modules. Hierarchical Scoring Head. The custom scoring head is responsible for mapping the high-dimensional semantic representations into multi-dimensional creativity assessment, and for further deriving a coarse-grained overall score that is consistent with existing debate annotation schemes. This module adopts a multi-layer neural network architecture and explicitly models the hierarchical relationship between fine-grained and coarse-grained scores, thereby accommodating supervision signals of mixed granularity.
DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios
Finegrained
(a) Originality Flexibility Fluency
Coarsegrained
Predicted score
statement1 statement2 Ground truth
LLM-as -a-Judge
Ground truth
Metric: MSE & PCC
Relevance Logicality
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
Clarity
Originality
Appeal
Flexibility
Effectiveness
score1
(b)
score2
Pair ACC
：statement1
Evaluation Protocol
Fluency
(c) Overall Score
statement2
Clarity
Relevance Logicality
Appeal Effectiveness
expert adjudicators
MSE
Finegrained
Metric
Coarsegrained
High- proficiency
Mid- and low- proficiency
Figure 2: The three-modular evaluation protocol to assess the performance of scoring model. This evaluation protocol validates system robustness across varying annotation granularities (Fine vs. Coarse) and proficiency levels (High vs. Mid-Low). (a) Fine-grained evaluation measures the alignment of eight-dimensional predictions with human expert annotations using Mean Squared Error (MSE) and Pearson Correlation Coefficient (PCC). (b) Coarse-grained evaluation in the mid-to-low proficiency range. Given the instability of absolute scoring for lower-quality responses, we utilize Pairwise Accuracy (ACC) to assess the model’s capability to correctly rank synthetic sample pairs. (c) Coarse-grained evaluation in the high-proficiency range. We benchmark the model’s predictions against authentic top-tier expert adjudicator scores using MSE. The input to this module is the high-level semantic representation ℎ last obtained from the autoregressive language model. This representation encodes the overall semantics, argumentative structure, and contextual dependencies of the current utterance under the given debate setting. The custom scoring head consists of two sequential components: (1) a multi-layer non-linear mapping network for predicting fine-grained scores across eight evaluation dimensions; and (2) an explainable aggregator that combines the eight-dimensional scores into a single scalar debate score. First, ℎ last is transformed through a stack of FFN layers. The network is composed of multiple fully connected layers, each followed by a SiLU (Sigmoid Linear Unit) activation function: ℎ (𝑘 ) = SiLU 𝑊 (𝑘 ) ℎ (𝑘 −1) , (3) where ℎ (0) = ℎ last , and 𝑊 (𝑘 ) denotes the weight matrix of the 𝑘-th layer. At the final layer, the hidden representation is projected into an eight-dimensional vector: ŝ = (𝑠ˆ1, 𝑠ˆ2, . . . , 𝑠ˆ8 ) ∈ R8 ,
(4)
where each component corresponds to the predicted score of one dimension in the debate metric system. This design enables the model to learn differentiated responses across dimensions while sharing a common semantic representation. To produce an overall score aligned with overall debate adjudication, the eight fine-grained scores are further passed to an aggregator module. The aggregator computes a weighted sum of the Í8 dimension-wise scores, which can be formalized as 𝑦ˆ = 𝑖=1 𝑤𝑖 𝑠ˆ𝑖 , where w = (𝑤 1, . . . , 𝑤 8 ) are learnable aggregation weights. This structure is intended to emulate the weighting and integration of multiple evaluation dimensions performed by expert adjudicators when assigning an overall score. By introducing this aggregation layer, the model is able to output both fine-grained eightdimensional scores and a coarse-grained overall score within a single forward pass, thereby naturally aligning with datasets annotated at different levels of granularity.
Hybrid Loss Function. To leverage mixed-granularity supervision, we optimize a hybrid objective function. For samples in 𝐷 coarse (and augmented subsets), we minimize the Mean Squared Error (MSE) on the holistic score. For samples in 𝐷 fine , we additionally minimize the sum of MSE across all dimensions. The total loss is defined as: 8 ∑︁ ˆ 𝑦) + 𝜆 · I𝑠 ∈𝐷 fine L = I𝑦 ∈𝐷 MSE(𝑦, MSE(𝑠ˆ𝑗 , 𝑠 𝑗 ) , (5) 𝑗=1
where I is the indicator function and 𝜆 balances the multi-task learning. During training, the appropriate loss term is dynamically selected according to the annotation granularity of each sample and used for back-propagation. This design allows the model to jointly leverage large-scale, easily obtainable coarse-grained data and a limited amount of fine-grained data within a shared parameter space, enabling data-efficient learning of multidimensional scoring.
3.5
Evaluation Protocol
We implement a rigorous Three-Modular Evaluation Protocol to assess the performance across proficiency levels (High vs. MidLow) and granularities (Fine vs. Coarse). For coarse-grained evaluation, we reserve 10% of the authentic competition data (𝐷 high ) and the corresponding summary variants as validation data. We benchmark DEFINED against state-of-the-art approaches of debate evaluation (e.g., InspireDebate [43], Debatrix [21]). Performance is quantified using MSE as metric of highscore range data and Pairwise Accuracy for mid- and low-score range data, leveraging the high reliability of comparative labels in the synthetic subset 𝐷 simulate . For fine-grained evaluation, 𝐷 fine is split (50/50) into training and validation sets, ensuring no overlap with the coarse-grained training data. We compare our scoring model with LLM evaluators (e.g., GPT-4o) using well-designed prompt templates derived from the expert-defined evaluation protocol. These prompts include detailed scoring rubrics, operational definitions, scoring checkpoints, and
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
Tongzhou Yu et al.
Table 1: Pearson correlation coefficients (PCC) between model predictions and expert scores for each dimension. Higher values indicate stronger agreement with human evaluators. Model
Fluency Originality Flexibility Logicality Relevance Effectiveness Clarity Appeal Average
Gemini-2.5-pro GPT-4o Qwen3-max-preview Deepseek-R1
0.71 0.70 0.74 0.79
0.77 0.64 0.77 0.77
0.76 0.67 0.75 0.74
0.71 0.68 0.69 0.74
0.67 0.70 0.75 0.77
0.71 0.70 0.75 0.83
0.44 0.58 0.50 0.55
0.61 0.71 0.74 0.76
0.67 0.67 0.71 0.74
M-Prometheus-7B Themis
0.59 -0.02
0.59 0.31
0.53 0.20
0.60 0.07
0.57 0.16
0.62 0.37
0.56 -0.22
0.55 0.14
0.58 0.12
DEFINED
0.96
0.96
0.94
0.96
0.96
0.96
0.93
0.97
0.96
Table 2: Mean squared error (MSE) of fine-grained score predictions for DEFINED and four representative LLM-based evaluators. Lower MSE indicates closer numerical alignment with expert judgments. Model
Fluency Originality Flexibility Logicality Relevance Effectiveness Clarity Appeal Average
Gemini-2.5-pro GPT-4o Qwen3-max-preview Deepseek-R1
358.75 305.51 243.64 298.73
465.68 347.46 208.93 207.10
444.08 400.42 251.24 226.75
419.44 372.10 300.58 240.95
506.66 355.73 231.15 225.88
555.39 369.64 230.19 166.90
597.76 395.76 440.25 332.37
550.12 332.93 276.44 271.61
487.24 359.94 272.80 246.29
M-Prometheus-7B Themis
352.50 1943.67
402.72 1143.48
465.48 1148.60
405.02 1830.02
488.47 1576.83
416.18 963.40
407.10 2366.62
510.42 1483.35
430.99 1557.00
DEFINED
42.24
34.04
59.60
46.72
35.98
35.76
57.01
33.36
43.09
representative high-/low-score examples to improve scoring consistency and instruction following. The complete prompt template is provided in Appendix D. We adopt MSE and Pearson correlation coefficient (PCC) to evaluate prediction accuracy and alignment with the cognitive dimensions of human annotators.
4
Experiment
In this section, we present the experimental results under the proposed three-modular evaluation protocol (Sec. 4.2-4.4), case study of typical bias patterns in LLMs (Sec. 4.5), ablation study that isolates the coarse-grained data (Sec. 4.6) and interpretability analysis of dimension weights (Sec. 4.7). Given that each evaluation setting involves different data sources and performance metrics, we provide a detailed description for each part. The code for DEFINED is available on GitHub at https://github.com/tzwo/DEFINED.
4.1
Experimental Setup and Costs
All experiments are conducted by fine-tuning Qwen2.5-7B-Instruct as the semantic backbone of DEFINED. Training is performed on 8 NVIDIA H100 GPUs. The total training time is 6,453 seconds, reflecting the data-efficient nature of the proposed framework. The main training hyperparameters are set as follows: learning rate 4 × 10−4 , maximum sequence length 8,192, training batch size 2 with gradient accumulation over 8 steps, validation batch size 8, and a total of 30 training epochs. Model evaluation is performed every 30 steps to monitor convergence throughout training. To enable parameter-efficient adaptation under limited finegrained supervision, we adopt Low-Rank Adaptation (LoRA) [15]
for model fine-tuning. Specifically, the LoRA rank is set to 8 with a scaling factor (alpha) of 32 and a dropout rate of 0.1. LoRA modules are injected into attention-related projection layers, including q_proj and v_proj, which play a central role in contextual representation learning. Optimization is carried out using the AdamW optimizer with 𝛽 1 = 0.9, 𝛽 2 = 0.999, and 𝜖 = 1 × 10−8 . At inference time, DEFINED demonstrates low latency and favorable scalability. Evaluation on the test set is conducted using four NVIDIA H100 GPUs with a batch size of 32. Under this configuration, the model achieves an average inference time of 0.045 seconds per sample, enabling efficient large-scale deployment. Taken together, these results indicate that DEFINED attains strong performance while maintaining moderate training cost and fast inference, making it practical for real-world educational and assessment scenarios where both accuracy and efficiency are critical.
4.2
Accurate Predictions of Scoring Model in Fine-Grained Score
We first evaluate DEFINED’s core contribution, i.e., multi-dimensional creativity assessment based on the fine-grained evaluation protocol (Fig. 2(a)). We compare DEFINED against two categories of baselines: (1) foundation model, including Gemini-2.5-Pro, GPT-4o, Qwen3-Max, and DeepSeek-R1, which rely on prompt-based scoring; and (2) fine-tuned models for text assessment, including Themis and M-Prometheus. All models are evaluated on the same validation dataset using the expert-defined scoring rubrics as prompts. We adopt multiple metrics, including MSE and PCC, to jointly measure prediction accuracy and alignment with human annotators.
DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios
(a)
(b)
Total distribution of data
0-10
Mid and low coarse-score evaluation (ACC(%))
554
10-20
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
146
Debatrix
InspireDebate(sub)
InspireDebate(sub+obj)
DEFINED
100
20-30
566
90
30-40
438
80
40-50
456
70
50-60
60
480
60-70
50
546
70-80
40
454
30
80-90
928
90-100
20
202 0
200
10
400
600
800
0
(c)
Negative sample 354
Summarization variant 2476
1000
High coarse-score evaluation (MSE) Debatrix
InspireDebate(sub)
InspireDebate(sub+obj) 350
DEFINED
300
Accumulated sample 1416
250 200 150
Authentic statement 706
Fine-grained
100
Data 124
50 0
Figure 3: Performance of coarse-grained debate assessment across proficiency spectrums. (a) Dataset composition. The distribution of authentic high-proficiency samples alongside synthetic mid-to-low proficiency variants generated via our augmentation strategy. (b) Evaluation on mid-to-low proficiency synthetic data. Pairwise agreement ACC shows the proposed framework’s robustness in distinguishing relative quality among weaker responses, effectively mitigating the scoring instability observed in baseline models. (c) Evaluation on high-proficiency authentic competition data. DEFINED achieves significantly lower MSE compared to LLM-based baselines (Debatrix, InspireDebate), indicating superior alignment with expert adjudicators. As shown in Tab. 1, DEFINED consistently outperforms all baseline evaluators across all eight dimensions. While API-based models exhibit moderate correlations with expert scores (average PCC ranging from 0.67 to 0.74), fine-tuned models show weaker alignment in this setting. Specifically, M-Prometheus-7B achieves only moderate agreement (average PCC 0.58), while Themis performs poorly overall (average PCC 0.12). In contrast, DEFINED achieves substantially stronger alignment, with an average PCC of 0.96, indicating near-expert-level internal consistency. Correspondingly, DEFINED reduces the average MSE by an order of magnitude compared with the best-performing API baseline (mean MSE 43.09 versus 246.29 for DeepSeek-R1), and dramatically outperforms evaluator-oriented models such as M-Prometheus7B (430.99) and Themis (1557.00), demonstrating significantly improved numerical accuracy in fine-grained scoring (Tab. 2). Among the API-based baselines, Qwen and DeepSeek outperform Gemini and GPT across most dimensions. This gap is possibly attributable to the stronger Chinese language understanding and discourse modeling capabilities of Qwen and DeepSeek, which are trained with a heavier emphasis on high-quality Chinese corpora. These results demonstrate DEFINED not only achieves high predictive accuracy at the aggregate level but also provides stable, and human-aligned fine-grained creativity scores across dimensions. DEFINED has advantages over mainstream open-source and closedsource models in the task of fine-grained scoring of creativity.
For the 10% coarse-grained setting, we performed two additional re-samplings, yielding MSE values of 44.43 ± 1.40 across runs. For the fine-grained setting, we conducted one additional re-sampling, resulting in an MSE of 42.35 ± 0.92. In contrast, API-based methods still exhibit substantially higher errors (MSE > 350). These results demonstrate only minor variation across runs, while maintaining consistent performance advantages over baselines.
4.3
Accurate Coarse-Grained Pairwise Comparisons in Mid- and Low-Score Range
As illustrated in the mid-to-low proficiency module (Fig. 2(b)), absolute ground truth scores are unstable for lower-quality synthetic responses. Therefore, we adopt Pairwise Agreement ACC as the primary metric. For each competition in the set (Fig. 3(a)), we generate an additional set of samples for validation. The original pairwise ordering is used as a ground truth. Model performance is therefore evaluated using Pairwise Agreement ACC, defined as the proportion of statement pairs for which the predicted score ordering matches the original pairwise preference (Fig. 2(b)). We compare DEFINED with two representative debate assessment methods, Debatrix and InspireDebate, both of which rely on LLM as judges. The compared methods differ in their dimensional formulations, which are used as prompts to call LLM to get scores. InspireDebate decomposes debate quality into subjective dimensions (including
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
Tongzhou Yu et al.
Table 3: Ablation study on mixed-granularity supervision using mean squared error (MSE). Method
Logicality
Appeal
Relevance
Clarity
Effectiveness
Flexibility
Originality
Fluency
Average
Fine-grained Only w/o Simulation w/o Summarization
140.35 563.49 81.48
109.66 465.08 100.82
125.33 556.72 90.17
97.31 462.33 81.46
97.87 516.56 64.37
102.41 544.17 104.26
154.66 540.05 94.78
82.76 476.85 114.10
113.79 515.66 91.43
DEFINED (Mixed)
46.72
33.36
35.98
57.01
35.76
59.60
34.04
42.24
43.09
Emotional Appeal, Argument Clarity, Argument Arrangement, and Topic Relevance) and objective dimensions such as Fact Authenticity and Logical Validity, which are then aggregated by means of averaging. In contrast, Debatrix organizes evaluation signals into broader categories of Argument, Source, and Language. As shown in Fig. 3(b), DEFINED achieves a pairwise accuracy of 95.2% in the mid- and low-score range, outperforming baseline methods. In contrast, Debatrix using GPT-4o achieves an accuracy of 69.4%, while InspireDebate reaches 71.4% when using only subjective dimensions and 63.2% when combining subjective and objective dimensions. The margin observe between DEFINED and the baselines indicates that DEFINED more reliably distinguishes relative quality differences among mid- and low-performing statements. Although both Debatrix and InspireDebate exhibit moderately acceptable performance in relative ordering, their absolute scores remain highly unstable in this range (box plots of Fig. 3). In particular, even when the evaluated statement is of clearly inferior quality, API-based methods frequently assign inflated absolute scores (for example, scores approaching 85 or 90), while some high-quality statements in authentic competitions receive scores around 75. The upward bias in absolute scoring substantially limits the practical utility of these methods in educational or formative assessment scenarios, where distinguishing mediocre from strong performance is critical. In the mid- and low-score range, naive LLM-based scoring pipelines tend to overestimate performance and blur meaningful distinctions. By contrast, DEFINED demonstrates strong consistency in both relative ranking and absolute score, reflecting its ability to anchor coarse-grained predictions to expert adjudicator distributions learned from authentic competition data.
4.4
Accurate Coarse-Grained Debate Scoring in High-Score Range
Following high-proficiency evaluation module defined in Fig. 2(c), we assess the model’s ability to approximate expert adjudicators in authentic debate scenarios. We randomly select 10% of the competitions from the dataset as a validation set and focus on debates drawn from authentic competition data (Fig. 3(a)). All methods are evaluated under identical data, and mean squared error (MSE) with respect to expert scores is used as the evaluation metric (Fig. 2(c)). As shown in Fig. 3(a), DEFINED outperforms all baseline methods in the high-score range, achieving an MSE of 18.23, compared with 61.94 for Debatrix (GPT-4o) and 180.53 and 291.16 for InspireDebate using subjective-only and combined subjective–objective dimensions, respectively (Fig. 3(c)). The reduction is over 70% relative to Debatrix and over 90% relative to InspireDebate in MSE. This performance gap indicates that DEFINED achieves a closer numerical
alignment with expert adjudicator judgments in authentic competition data. Moreover, DEFINED contains only 7B parameters, which is smaller than the mainstream models used for comparison, requiring fewer computational resources during inference.
4.5
Case Study of Typical Bias Patterns in the LLM’s Scoring Behavior
To further elucidate the behavioral differences between existing LLM-based evaluators and our proposed DEFINED framework, we conduct a qualitative case study on real debate utterances. We construct a specific experimental scenario and collected speeches from 60 debate-naive participants (32 females, 28 males, undergraduate or graduate students) under this setting. These responses are evaluated using both DEFINED and representative LLMs. By examining the model-generated scores alongside the fine-grained results in the validation set, we manually inspect the three samples exhibiting the largest discrepancies across evaluation dimensions (detailed in Appendix C). This analysis allows us to identify several representative cases that illustrate typical bias patterns in the LLM’s scoring behavior: score instability, surface-level pattern matching, and absolute score mismatch, while highlighting the robustness and interpretability of DEFINED. First, score instability under contextual perturbation emerges as a critical limitation of existing LLM-based evaluators. As illustrated in our examples, identical debate speeches evaluated under slightly altered opponent’s speech can lead to score deviations exceeding 30 points on the same dimension for several strong baseline models. Such volatility suggests that these models do not focus on the intrinsic quality of discourse while distracted by background information. In contrast, DEFINED produces highly consistent scores across these perturbations. This stability is particularly crucial in realistic evaluation scenarios to score the target statement. Second, we observe pattern-matching in baseline models. Statements that explicitly adopt formal discourse markers such as “at first”, “second”, “finally” are frequently rewarded with inflated fluency scores, even when their underlying arguments are shallow or repetitive. This suggests that many models conflate rhetorical templates with fluency. DEFINED can understand the deep argumentative structure of statements, rather than relying on superficial structural words, assigning more reasonable scores accordingly. Third, existing methods exhibit distortions in absolute score, leading to poor discrimination between low-quality and high-quality speeches. In several cases, texts with evident issues, such as oversimplified reasoning or lack of engagement with the opposing argument, receive scores comparable to, or even higher than, rhetorically complex and substantively rich speeches. This undermines
DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
the evaluative utility of automated systems in competitive or highstakes settings. DEFINED produces well-separated score distributions: high-quality debate speeches are consistently assigned higher scores across relevant dimensions, while lower-quality texts are clearly distinguished. This suggests that DEFINED learns a more human-aligned internal scale of creative and argumentative quality. Taken together, these cases demonstrate that DEFINED not only improves correlation with human judgments, but also addresses deeper structural weaknesses of existing LLM-based evaluators, offering a more reliable and principled solution for fine-grained creativity assessment in debate contexts.
dimensions in human cognition. Logical failure leads to immediate penalization, whereas logical adequacy does not necessarily yield additional credit. Notably, Clarity is weighted higher than Logicality, suggesting that comprehensibility takes precedence over formal reasoning rigor: arguments that are easier to understand are favored over those that are logically sound but difficult to follow. Overall, this weighting structure indicates that DEFINED internalizes a human-like evaluation process that prioritizes practical impact and communicative effectiveness over purely formal correctness.
4.6
In this work, we proposed DEFINED, a data-efficient framework that operationalizes fine-grained creativity assessment within complex debate scenarios. By decomposing holistic expert judgments into a hierarchical eight-dimensional metric system, our model effectively reverse-engineers the cognitive process of evaluation, distinguishing specific creativity dimensions from general quality factors. Crucially, the integration of a mixed-granularity training strategy and constrained data augmentation addresses the persistent challenges of annotation scarcity and elite-data bias in computational social science. This architectural design enables the model to leverage abundant coarse-grained signals alongside limited finegrained supervision, achieving robust, scalable, and ecologically valid scoring that significantly outperforms existing methods. Despite these advancements, certain methodological constraints and future directions warrant discussion. First, owing to the unique mixed-granularity supervision paradigm proposed here, conventional supervised fine-tuning methods cannot be directly adapted as baselines. Instead, we ensured rigorous comparative analysis against state-of-the-art API-based LLM evaluators, utilizing expertverified prompts strictly aligned with human annotation guidelines to guarantee fair benchmarking. Challenges also remain regarding cross-domain generalization and potential stylistic artifacts in synthetic data. Future research should focus on identifying taskinvariant creativity features to enable transfer learning across disparate open-ended tasks. Beyond debate, the proposed framework offers a paradigmatic strategy for mining complex psychological constructs under data-scarce conditions: by anchoring fine-grained indicators to large-scale coarse signals, it bridges the gap between psychometric theory and scalable data mining applications.
Ablation Study
To validate the design choice of mixed coarse–fine-grained supervision in DEFINED, we conduct an ablation study that isolates the contribution of coarse-grained training signals. In this ablation setting, the model is trained exclusively on a limited set of finegrained annotations. All other components of the architecture and optimization procedure are kept identical to the full model. Tab. 3 reports the dimension-wise MSE of the full DEFINED model compared with the ablated variant trained using only finegrained creativity scores. Across all eight dimensions, removing coarse-grained supervision leads to a substantial degradation in numerical accuracy. The average MSE increases from 43.09 in the full DEFINED model to 113.79 under fine-grained-only training, corresponding to a relative error increase of more than 2.6×. We further analyze the role of the data augmentation strategy. Removing simulation data causes severe degradation in the midand low-score range: the model frequently fails to assign reasonable scores to lower-quality responses, with prediction deviations often exceeding 30 points. This finding confirms that simulated debate data is essential for mitigating elite bias introduced by expert-only annotations. In addition, removing the summarization variant increases the average MSE to approximately 90, indicating that the model is highly sensitive to contextual diversity. These results collectively demonstrate that mixed-granularity supervision is critical for robust and accurate creativity assessment. Since coarse-grained data is substantially easier to obtain and scale, it effectively improves both the generalization ability and numerical stability of DEFINED.
4.7
Interpretability Analysis
The learned dimension weights—Effectiveness (0.1826), Fluency (0.1689), Originality (0.1396), Clarity (0.1221), Appeal (0.1016), Flexibility (0.0977), Relevance (0.0928), and Logicality (0.0923)—reveal a clear alignment with human adjudication in debate. Effectiveness receives the highest weight, indicating that human evaluators adopt a fundamentally outcome-oriented judgment strategy: arguments are primarily valued by whether they meaningfully advance the debate or successfully address the opponent’s core claims. Fluency and Originality are also assigned relatively high weights, reflecting that evaluators’ first cognitive response is whether a speech “has substance”—that is, whether it generates multiple non-redundant ideas rather than merely repeating previously stated content. By contrast, Logicality and Relevance receive lower weights, not because they are unimportant, but they function as threshold
5
6
Conclusion & Discussion
Acknowledgments
This work is supported by the Shanghai Municipal Special Program for Basic Research on General AI Foundation Models (Grant No. 2025SHZDZX026D08) and Fundamental Research Funds for the Central Universities (Grant No. 2026ECNU-WLJC009).
References [1] Catalin Anghel, Andreea Alexandra Anghel, Emilia Pecheanu, Ioan Susnea, Adina Cocu, and Adrian Istrate. 2025. Multi-Model Dialectical Evaluation of LLM Reasoning Chains: A Structured Framework with Dual Scoring Agents. Informatics 12 (2025), 76. [2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025). [3] Roger E Beaty and Dan R Johnson. 2021. Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior research methods 53, 2 (2021), 757–780.
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
[4] Qian Cao, Xiting Wang, Yuzhuo Yuan, Yahui Liu, Fang Luo, and Ruihua Song. 2025. Evaluating text creativity across diverse domains: A dataset and large language model evaluator. arXiv preprint arXiv:2505.19236 (2025). [5] Shelley H Carson, Jordan B Peterson, and Daniel M Higgins. 2005. Reliability, validity, and factor structure of the creative achievement questionnaire. Creativity research journal 17, 1 (2005), 37–50. [6] Liying Cheng, Lidong Bing, Ruidan He, Qian Yu, Yan Zhang, and Luo Si. 2022. IAM: A Comprehensive and Large-Scale Dataset for Integrated Argument Mining Tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Dublin, Ireland, 2277–2287. [7] Arthur Cropley. 2006. In praise of convergent thinking. Creativity research journal 18, 3 (2006), 391–404. [8] Paul V DiStefano, John D Patterson, and Roger E Beaty. 2025. Automatic scoring of metaphor creativity with large language models. Creativity Research Journal 37, 4 (2025), 555–569. [9] Boris Forthmann, Oluwatosin Oyebade, Adebusola Ojo, Fritz Günther, and Heinz Holling. 2019. Application of latent semantic analysis to divergent thinking is biased by elaboration. The Journal of Creative Behavior 53, 4 (2019), 559–575. [10] Amnon Glassner and Baruch B. Schwarz. 2007. What stands and develops between creative and critical thinking? Argumentation? Thinking Skills and Creativity 2 (2007). [11] Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A Large-Scale Dataset for Argument Quality Ranking: Construction and Analysis. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. New York, NY, 7805–7813. [12] Joy Paul Guilford. 1956. The structure of intellect. Psychological bulletin 53, 4 (1956), 267. [13] J. Paul Guilford. 1967. The Nature of Human Intelligence. McGraw-Hill, (1967). [14] David JP Heinen and Dan R Johnson. 2018. Semantic distance: An automated measure of creativity that is novel and appropriate. Psychology of Aesthetics, Creativity, and the Arts 12, 2 (2018), 144. [15] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In Proceedings of the 10th International Conference on Learning Representations. Virtual. [16] Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, and Xiaojun Wan. 2024. Themis: A reference-free nlg evaluation language model with flexibility and interpretability. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 15924–15951. [17] Emanuel Jauk, Mathias Benedek, and Aljoscha C Neubauer. 2014. The road to creative achievement: A latent variable model of ability and personality predictors. European journal of personality 28, 1 (2014), 95–105. [18] Mahdi Karami and Ali Ghodsi. 2024. Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling. In Advances in Neural Information Processing Systems 38. Vancouver,Canada. [19] T Leopold, Attilio Di Battista, Ximena Jativa, Shuvasish Sharma, R Li, and S Grayling. 2025. Future of jobs report 2025. In World Economic Forum. Geneva, Switzerland. [20] Ruizhe Li, Chiwei Zhu, Benfeng Xu, Xiaorui Wang, and Zhendong Mao. 2025. Automated Creativity Evaluation for Large Language Models: A Reference-Based Approach. Computing Research Repository abs/2504.15784 (2025). [21] Jingcong Liang, Rong Ye, Meng Han, Ruofei Lai, Xinyu Zhang, Xuan-Jing Huang, and Zhongyu Wei. 2024. Debatrix: Multi-dimensional debate judge with iterative chronological analysis based on llm. In Findings of the 62nd Annual Meeting of the Association for Computational Linguistics. Bangkok, Thailand, 14575–14595. [22] Xinyi Liu, Pinxin Liu, and Hangfeng He. 2024. An Empirical Analysis on Large Language Models in Debate Evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Bangkok, Thailand, 470–487. [23] Devin C Lonergan, Ginamarie M Scott, and Michael D Mumford. 2004. Evaluative aspects of creative thought: Effects of appraisal and revision standards. Creativity Research Journal 16, 2-3 (2004), 231–246. [24] Simone A Luchini, Nadine T Maliakkal, Paul V DiStefano, Antonio Laverghetta Jr, John D Patterson, Roger E Beaty, and Roni Reiter-Palmon. 2025. Automated scoring of creative problem solving with large language models: A comparison of originality and quality ratings. Psychology of Aesthetics, Creativity, and the Arts (2025). [25] Martha T Mednick and Sharon Halpern. 1968. Remote associates test. Psychological Review (1968). [26] Sarnoff Mednick. 1962. The associative basis of the creative process. Psychological review 69, 3 (1962), 220. [27] Michael D Mumford and Tristan McIntosh. 2017. Creative thinking processes: The past and the future. The Journal of Creative Behavior 51, 4 (2017), 317–322. [28] Michael D Mumford, Michele I Mobley, Roni Reiter-Palmon, Charles E Uhlman, and Lesli M Doares. 1991. Process analytic models of creative capacities. Creativity research journal 4, 2 (1991), 91–122. [29] OECD. 2024. PISA 2022 Results (Volume III): Creative minds, creative schools. OECD Publications Centre, Paris, France.
Tongzhou Yu et al.
[30] Peter Organisciak, Selcuk Acar, Denis Dumas, and Kelly Berthiaume. 2023. Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models. Thinking Skills and Creativity 49 (2023), 101356. [31] Namgyoo K Park, Monica Youngshin Chun, and Jinju Lee. 2016. Revisiting individual creativity assessment: Triangulation in subjective and objective assessment methods. Creativity Research Journal 28, 1 (2016), 1–10. [32] Philip M Podsakoff, Scott B MacKenzie, Jeong-Yeon Lee, and Nathan P Podsakoff. 2003. Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of applied psychology 88, 5 (2003), 879. [33] José Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig, and André FT Martins. 2025. M-Prometheus: A Suite of Open Multilingual LLM Judges. arXiv preprint arXiv:2504.04953 (2025). [34] Ranjani Prabhakaran, Adam E Green, and Jeremy R Gray. 2014. Thin slices of creativity: Using single-word utterances to assess creative cognition. Behavior research methods 46, 3 (2014), 641–659. [35] Allen Roush, Yusuf Shabazz, Arvind Balaji, Peter Zhang, Stefano Mezza, Markus Zhang, Sanjay Basu, Sriram Vishwanath, Mehdi Fatemi, and Ravid ShwartzZiv. 2024. OpenDebateEvidence: A Massive-Scale Argument Mining and Summarization Dataset. In Advances in Neural Information Processing Systems 38. Vancouver,Canada. [36] Mark A Runco and Garrett J Jaeger. 2012. The standard definition of creativity. Creativity research journal 24, 1 (2012), 92–96. [37] Gabriella Skitalinskaya, Jonas Klaff, and Henning Wachsmuth. 2021. Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Virtual, 1718–1729. [38] Noam Slonim, Yonatan Bilu, Carlos Alzate, Roy Bar-Haim, Ben Bogin, Francesca Bonin, Leshem Choshen, Edo Cohen-Karlik, Lena Dankin, Lilach Edelstein, Liat Ein-Dor, Roni Friedman-Melamed, Assaf Gavron, Ariel Gera, Martin Gleize, Shai Gretz, Dan Gutfreund, Alon Halfon, Daniel Hershcovich, Ron Hoory, Yufang Hou, Shay Hummel, Michal Jacovi, Charles Jochim, Yoav Kantor, Yoav Katz, David Konopnicki, Zvi Kons, Lili Kotlerman, Dalia Krieger, Dan Lahav, Tamar Lavee, Ran Levy, Naftali Liberman, Yosi Mass, Amir Menczel, Shachar Mirkin, Guy Moshkowich, Shila Ofek-Koifman, Matan Orbach, Ella Rabinovich, Ruty Rinott, Slava Shechtman, Dafna Sheinwald, Eyal Shnarch, Ilya Shnayderman, Aya Soffer, Artem Spector, Benjamin Sznajder, Assaf Toledo, Orith Toledo-Ronen, Elad Venezian, and Ranit Aharonov. 2021. An autonomous debating system. Nature 591, 7850 (2021), 379–384. [39] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2023. Learning From Noisy Labels With Deep Neural Networks: A Survey. IEEE Transactions on Neural Networks and Learning Systems 34 (2023), 8135–8153. [40] Branden Thornhill-Miller, Anaëlle Camarda, Maxence Mercier, Jean-Marie Burkhardt, Tiffany Morisseau, Samira Bourgeois-Bougrine, Florent Vinchon, Stephanie El Hayek, Myriam Augereau-Landais, Florence Mourey, et al. 2023. Creativity, critical thinking, communication, and collaboration: Assessment, certification, and promotion of 21st century skills for the future of work and education. Journal of Intelligence 11, 3 (2023), 54. [41] E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and psychological measurement (1966). [42] Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yufang Hou, Graeme Hirst, Iryna Gurevych, and Benno Stein. 2017. Argumentation Quality Assessment: Theory vs. Practice. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vol. 2. Vancouver, Canada, 250–255. [43] Fuyu Wang, Jiangtong Li, Kun Zhu, and Changjun Jiang. 2025. InspireDebate: Multi-Dimensional Subjective-Objective Evaluation-Guided Reasoning and Optimization for Debating. arXiv preprint arXiv:2506.18102 (2025). [44] Lu Wang, Nick Beauchamp, Sarah Shugars, and Kechen Qin. 2017. Winning on the Merits: The Joint Effects of Content and Style on Debate Outcomes. Transactions of the Association for Computational Linguistics 5 (2017), 219–232. [45] Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good. In Proceedings of the 57th Conference of the Association for Computational Linguistics, Vol. 1. Florence, Italy, 5635–5649. [46] Yixuan Wang, Yue Huang, Hong Qian, Yunzhao Wei, Yifei Ding, Wenkai Wang, Zhi Liu, Zhongjing Huang, Aimin Zhou, and Jiajun Guo. 2026. AlphaContext: An Evolutionary Tree-based Psychometric Context Generator for Creativity Assessment. arXiv preprint arXiv:2604.18398 (2026). [47] Weini Weng, Chang Liu, Guoli Zhao, Luwei Song, and Xingli Zhang. 2025. Intelligent Assessment of Scientific Creativity by Integrating Data Augmentation and Pseudo-Labeling. Information 16, 9 (2025), 785. [48] Wenjing Yang, Adam E Green, Qunlin Chen, Yoed N Kenett, Jiangzhou Sun, Dongtao Wei, and Jiang Qiu. 2022. Creative problem solving in knowledge-rich contexts. Trends in Cognitive Sciences 26, 10 (2022), 849–859. [49] Liang Zeng, Robert W Proctor, and Gavriel Salvendy. 2011. Can traditional divergent thinking tests be trusted in measuring and predicting real-world creativity? Creativity research journal 23, 1 (2011), 24–37.
DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios
reflects the strategic use of emotional expression or contextualised narrative to foster resonance with listeners and thereby enhance persuasive force [42, 44]. • Effectiveness. Effectiveness measures whether conclusions or strategic choices, while remaining logically consistent and task-relevant, successfully address the issue at hand and advance the argumentative process, ultimately achieving the practical goal of refuting the opponent or reinforcing one’s own position [42].
[50] Yiwen Zhang, Hong Qian, Xiaowen Wang, Yixvan Wang, Mingjia Li, Jin Wu, Jiajun Guo, Xiangfeng Wang, Chanjin Zheng, and Aimin Zhou. 2026. Research on Automatic Evaluation of Idea Quality in Knowledge Building Communities. China Educational Technology 472 (2026), 85–94. [51] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MTBench and Chatbot Arena. In Advances in Neural Information Processing Systems 36. New Orleans, LA.
Appendix A Detailed Metric System Detailed definition of eight dimensional metric system: • Originality (Creativity). Originality refers to the relative uniqueness or rarity of ideas or solutions proposed within a given context [12, 36, 41]. In debates, this dimension concerns the distinctiveness of claims, analogies, or argumentative pathways. It requires extending or transforming foundational positions when deconstructing opponents’ arguments or articulating one’s own stance [1]. • Fluency (Creativity). Fluency is defined as the capacity to rapidly and continuously generate a large number of distinct ideas, responses, or potential solutions in open-ended settings [12, 41]. In debate scenarios, it is operationalised as the number of relevant arguments or rebuttal points produced under given time and information constraints. • Flexibility (Creativity). Flexibility refers to the ability to shift across categories, perspectives, or reasoning pathways during the thinking process, generating diverse types of answers or solutions [13, 41]. In debates, this dimension reflects cross-domain or multi-strategy reasoning, such as approaching an issue from legal and economic perspectives, moving between macro- and micro-level analyses. • Relevance (Creativity). In creativity research, appropriateness denotes the degree to which an idea fits the task context, goals, and constraints [36]. Within this broader construct, relevance denotes the ability to focus on the core issues within complex problems and to accurately identify unresolved or salient points that warrant further examination. In debates, it refers to identifying critical weaknesses in the opponent’s reasoning or direct challenges to one’s own position, and responding explicitly to the central points of contention without digression [42]. • Logicality (Creativity). In cognitive processes, reasoning follows coherent inferential pathways, applying prior knowledge and rules to analyse information and derive judgments or conclusions in a logically consistent manner [7, 23]. In a debate context, logicality refers to the ability to construct a clear and complete chain of reasoning, organise arguments in a structured way, avoid logical fallacies or inappropriate evidence, and explicitly articulate the relationships between claims and supporting evidence [11, 37]. • Clarity. Clarity denotes the degree to which information is presented in an intelligible and accessible manner, avoiding ambiguity or vagueness, thereby enabling the audience to quickly and accurately grasp the core message [1, 42]. • Appeal. Appeal captures the overall impact of an argument on the audience at both the content and stylistic levels. It
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
B
Augmentation Prompt
Due to space limitations, we present one dimension prompt as an example. Other dimensions follow the same prompt structure. Level 40 (Basic Capability) Instruction A: You have basic argumentation skills. Please strictly follow these requirements: You can only propose 1 relevant point revolving around the topic or the opponent’s view, but with limited depth. Your viewpoint is common, lacking the ability to innovate or redefine the problem. You can maintain roughly coherent logic, but often with jumps or parts lacking explanation. You tend to focus on a single domain or a single type of reasoning path and do not actively switch analytical angles. When responding to the opponent, you might notice a simple loophole but cannot explain its importance or build an effective counterattack. Content is dominated by shallow analysis, completely failing to build systematic frameworks or multiple logical chains, maintaining only rough discourse. Instruction B : You have basic but weak argumentation skills. You are allowed to propose around 1 relevant point (at most 1 main argument + some qualifying explanations), without a second independent argument. The main argument should be a common, obvious conclusion, expressed in common frameworks (causality or comparison); explanations must be limited and superficial, without digging deep or building complete reasoning chains. You may notice a simple loophole in the opponent’s argument but can only point out its existence, not explain its mechanism or scope of impacts. Prohibit cross-domain switching, creating new concepts, or complex analogies; do not use precise data or complex evidence. Output should be roughly coherent but may have jumps or lack explanation. Strict limit: No more than 1 independent main argument; response to the opponent is limited to “mentioning" rather than “deep refutation."
C
Case Study
Detailed demonstration of case studies on typical bias patterns in the scoring behavior of large language models is shown in Fig 4.
D
Fine-grained Prompt
Due to space limitations, we present only one dimension prompt as an example. The remaining dimensions follow the same prompt structure, including theoretical definition, operational definition, scoring rubrics, checkpoints, and illustrative examples. Dimension: Divergent Thinking — Originality Theoretical Definition. The relative uniqueness or rarity of the ideas or solutions proposed within a given context, distinguishing them from conventional or obvious options.
KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea
Tongzhou Yu et al.
(a)
(c)
(b) Score instability
Pattern-matching
Background 1:
Background 2:
Okay, thank you, Mr. Chairman. Hello everyone. First, let's clarify the definition. ... Isaiah Berlin, in his book "The Roots of Romanticism," Perturbation summarized romanticism into a core concept: ... Next, at the social level, the pursuit of romanticism can promote social progress. ...
Thank you, Mr. Chairman, and hello everyone. First of all, romanticism emphasizes ... From an individual perspective, many people in modern society feel a lack of enthusiasm and anxiety, .... At the social level, the pursuit of romanticism has promoted social progress. ... 45
-30
70
40
-30
27
26
-1
Statement: Hello everyone, first of all, I want to say that romanticism is indeed a spiritual trap in contemporary times. ...
Statement 1:
First of all, regarding the statement that using light-hearted and playful remarks in certain environments is not equivalent to using dirty words in such environments, ... Why is it that people are now afraid to speak up? It is not because the so-called "meme culture" hinders expression, ... Moreover, according to research from the University of Cambridge, ... Instead, they should be about exchanging ideas in a relaxed atmosphere, ...
Comment: Logical leap and slippery slope fallacy.
Hello everyone, the opposing debaters mentioned that romanticism can serve as a spiritual sustenance and alleviate the anxiety of contemporary people. However, we must not overlook ... Therefore, in contemporary society, romanticism has not become a true spiritual sustenance. Logicality
80
85
41
75
70
84
Fluency
Fluency
75
Absolute score bias
Comment: Dense key points and diverse perspectives.
Statement 1:
40
70
70
45
Statement 2:
DEFINED
Comment: Clear structure but limited scope and description.
First of all, the opposing side put forward that scenarios like "Daoxiangcun" and "migrant workers" are not suitable for making jokes or teasing, ... Secondly, regarding the "verbal aphasia" mentioned by the opposing side, ... Finally, the opposing side also mentioned whether memes become more inappropriate and more abstract to be funny. ...
Statement 2:
Comment: Rigorous structure and complete logical chain.
First, I want to tell everyone two things not to be afraid of. First, don't worry about public opinion influencing justice. Because, first of all, justice must consider public opinion and its impact. Secondly, being influenced by public opinion and being influenced in terms of fairness are two different things. Many times, ... In all the long speeches from the opposing side just now, they basically said one thing: ... To conclude, to be honest, four years ago, I participated in a debate at the New National Debate Competition on the topic of online issues. Was it a profound topic for discussion? The opposing side is arguing the same as they did four years ago: ...
Figure 4: Case study of typical bias patterns in LLM-based scoring and the robustness of DEFINED. (a) Score instability under contextual perturbation. (b) Surface-level pattern matching. (c) Absolute score mismatch. Correlation (Score 70 80)
Correlation (Score 80 90)
1.00
Fluency 1.00 0.76 0.82 0.63 0.30 0.60 0.55 0.68 0.88
Originality 0.01 1.00 -0.05 0.33 0.37 0.79 0.63 -0.17 0.56
Originality 0.76 1.00 0.78 0.77 0.42 0.54 0.55 0.57 0.86
Flexibility 0.37 -0.05 1.00 0.35 -0.16 0.25 -0.09 0.44 0.35
Flexibility 0.82 0.78 1.00 0.64 0.37 0.58 0.75 0.73 0.90
0.50
Logicality 0.36 0.33 0.35 1.00 0.81 0.35 0.66 0.76 0.77
Logicality 0.63 0.77 0.64 1.00 0.29 0.55 0.64 0.42 0.77
0.25
Relevance 0.47 0.37 -0.16 0.81 1.00 0.42 0.67 0.60 0.77
Relevance 0.30 0.42 0.37 0.29 1.00 0.47 0.40 0.22 0.53
0.00
Effectiveness 0.59 0.79 0.25 0.35 0.42 1.00 0.49 0.06 0.83
Effectiveness 0.60 0.54 0.58 0.55 0.47 1.00 0.53 0.59 0.79
0.25
Clarity 0.05 0.63 -0.09 0.66 0.67 0.49 1.00 0.04 0.57
Clarity 0.55 0.55 0.75 0.64 0.40 0.53 1.00 0.54 0.76
Appeal 0.54 -0.17 0.44 0.76 0.60 0.06 0.04 1.00 0.56
Appeal 0.68 0.57 0.73 0.42 0.22 0.59 0.54 1.00 0.75
Total Score 0.74 0.56 0.35 0.77 0.77 0.83 0.57 0.56 1.00
Total Score 0.88 0.86 0.90 0.77 0.53 0.79 0.76 0.75 1.00
Pearson r
0.75
0.50 0.75
Flu
en cy igi na Fle lity xib Lo ility gic Re ality le Eff vanc ec tiv e en Claess rity Ap pe a To tal l Sc ore
1.00
Or
Or
Flu
en cy igi na Fle lity xib Lo ility gic Re ality le Eff vanc ec tiv e en Claess rity Ap pe a To tal l Sc ore
Fluency 1.00 0.01 0.37 0.36 0.47 0.59 0.05 0.54 0.74
Figure 5: Inter-dimension correlation matrix of eight-dimensional metric system. Operational Definition. The uniqueness of the discourse, analogies, or argumentative paths. It requires extending or transforming the foundational arguments rather than simply restating conventional logic. Scoring Rubrics. • 90–100: Presents a distinctly unique reasoning path; proposes rare but reasonable new perspectives or creative frameworks that redefine the issue. • 70–89: Offers some novelty in perspective; extends or refines common views, though the innovation is limited to specific segments rather than the overall framework. • 40–69: Largely follows common argumentative paths with occasional minor variations; lacks systematic innovation. • 0–39: Primarily restates the prompt or repeats existing positions without new insights; relies on ready-made logic. Checkpoints (Scoring Reference). (1) Presence of unique or rare argumentative angles? (2) Successful extension or transformation of the original argumentative base? (3) Integration of new analogies, concepts, or reasoning chains?
Example. • High Score: Reinterprets AI not as a competition for knowledge, but as a “power shift in defining intellectual authority,” thereby constructing an original structural argument. • Low Score: Merely repeats common clichés (e.g., “AI lacks emotion”) without introducing new perspectives or argumentative transformations.
E
Inter-dimension Correlation Matrix
To further examine the independence and discriminability of the proposed evaluation dimensions, we compute Pearson correlation matrices between all dimensions on the test set under different score intervals (Fig. 5). The analysis reveals that several dimensions exhibit relatively weak correlations despite belonging to the same evaluation framework. For example, within the 80–90 score range, the correlation between Relevance and Logicality is only 0.15, while in the 70–80 range, the correlation between Effectiveness and Appeal decreases to 0.06. These results indicate that the proposed dimensions capture distinct cognitive aspects of debate creativity rather than redundant information.