Preprint. Under review.
☼ S UPER N OVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions Ashima Suvarna, Kendrick Phan, Mehrab Beikzadeh, Hritik Bansal, Saadia Gabriel University of California, Los Angeles § github.com/asuvarna31/supernova
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved large language model (LLM) reasoning in formal domains such as mathematics and code. Despite these advancements, LLMs still struggle with general reasoning tasks requiring capabilities such as causal inference and temporal understanding. Extending RLVR to general reasoning is fundamentally constrained by the lack of high-quality, verifiable training data that spans diverse reasoning skills. To address this challenge, we propose S UPER N OVA, a data curation framework for RLVR aimed at enhancing general reasoning. Our key insight is that instruction-tuning datasets containing expert-annotated ground-truth encode rich reasoning patterns that can be systematically adapted for RLVR. To study this, we conduct 100+ controlled RL experiments to analyze how data design choices impact downstream reasoning performance. In particular, we investigate three key factors: (i) source task selection, (ii) task mixing strategies, and (iii) synthetic interventions for improving data quality. Our analysis reveals that source task selection is non-trivial and has a significant impact on downstream reasoning performance. Moreover, selecting tasks based on their performance for individual target tasks outperforms strategies based on overall average performance. Finally, models trained on S UPER N OVA outperform strong baselines (e.g., Qwen3.5) on challenging reasoning benchmarks including BBEH, Zebralogic, and MMLU-Pro. In particular, training on S UPER N OVA yields relative improvements of up to 52.8% on BBEH across model sizes, demonstrating the effectiveness of principled data curation for RLVR. Our findings provide practical insights for curating human-annotated resources to extend RLVR to general reasoning.
Performance on BigBench Extra Hard (BBEH) Baseline (Qwen3)
20
+29.4%
15 10 5
+86.5% 0.6B
Pass@8 (%)
Pass@1 (%)
arXiv:2604.08477v1 [cs.AI] 9 Apr 2026
Abstract
+50.0%
1.7B
Model Size
4B
35 30 25
SuperNova
+64.5%
+42.9% +50.8%
20 15 0.6B
1.7B
Model Size
4B
Figure 1: SuperNova elicits strong general reasoning. We show that training with our curated S UPER N OVA data leads to consistent pass@k improvements across varying values of k on a challenging benchmark, BBEH-test. We highlight that S UPER N OVA is effective on various models sizes from Qwen3 family.
1
Preprint. Under review.
1
Introduction
Large language models (LLMs) have shown remarkable progress in reasoning capabilities for formal domains such as mathematics and code (Guo et al., 2025; Lambert et al., 2024; Guha et al., 2025; Ma et al., 2025; Zeng et al., 2025; Hu et al., 2025; Chen et al., 2025). However, real-world problem solving requires a broader spectrum of reasoning skills beyond formal domains. For example, it may involve different forms of reasoning, such as determining that a street might be wet because it rained (causal inference), or understanding that an event scheduled "next Friday" cannot conflict with one scheduled "last Tuesday" (temporal reasoning). We refer to this broader set of capabilities as general reasoning—the ability to derive novel conclusions from existing knowledge using skills such as logical deduction, causal reasoning, spatial understanding, and pragmatic inference (Newell et al., 1972; Johnson-Laird, 2010; Griffiths, 2020). These capabilities are critical for solving tasks in benchmarks such as Big-Bench Extra Hard (BBEH), which evaluate complex forms of general reasoning. A widely adopted approach for improving reasoning in LLMs is reinforcement learning with verifiable rewards (RLVR). Specifically, RLVR relies on the ability to verify model outputs against a ground-truth final answer (Guo et al., 2025). The wide availability of humanverified data (e.g., MATH (Hendrycks et al., 2021), competitions1 , CodeForces) has led to the rapid scaling of RLVR pipelines for STEM reasoning (Chen et al., 2025; Yu et al., 2025; Akter et al., 2026; Hu et al., 2025). However, we find that exposure to STEM reasoning (Bhaskar et al., 2025; Huan et al., 2025; Zhou et al., 2025) does not transfer reasoning capabilities to general reasoning tasks. For example, OpenReasoner-7B (Hu et al., 2025) and OpenThinker7B (Guha et al., 2025) outperform the base model by +50% on challenging math benchmarks such as AIME24 (Zhang & Math-AI, 2024), while reducing performance by −8% on general reasoning tasks in BBEH. Prior work such as General Reasoner (Ma et al., 2025) has attempted to scale RLVR beyond STEM. Specifically, it focuses on deriving domain-specific (e.g., science, business, finance, and history) question–answer pairs from the web. However, this approach suffers from key limitations: (a) expanding to new domains does not necessarily improve the skills required for general reasoning (e.g., models achieve strong MMLU scores but still perform poorly on BBEH), and (b) data sourced from the internet is often difficult to verify due to noise and varying quality. On the other hand, obtaining high-quality human-verified data for teaching general reasoning skills via RLVR is expensive and labor-intensive. To this end, we make a crucial observation: there exists a plethora of high-quality, humanannotated data in resources curated for instruction-following. Specifically, datasets such as SuperNI (Wang et al., 2022) and FLAN (Wei et al., 2021) contain thousands of expertannotated tasks, including event understanding, question generation, and object counting (Appendix Table 8). However, these datasets cannot be directly used for RLVR for several reasons: (a) many open-ended tasks do not allow easy verification, (b) not all tasks are useful for eliciting strong reasoning capabilities, and (c) the principles for curating RLVR data for general reasoning remain underexplored. To address these challenges, we propose S UPER N OVA, a data curation framework for RLVR to advance general reasoning. S UPER N OVA is a multi-stage pipeline for curating high-quality RLVR data to improve downstream general reasoning performance (Figure 2). Importantly, we explore several data design decisions through 100+ compute-matched RL experiments. First, we start with a set of candidate tasks from SuperNI and assess their ability to elicit complex general reasoning across the 23 sub-tasks in BBEH. § 3.1. This involves reformatting several open-ended tasks into easily verifiable formats (e.g., converting them into multiple-choice questions). We find that source task selection has a dramatic impact on downstream general reasoning performance (pass@k). Second, we evaluate two strategies for mixing diverse tasks in the source data. Macro mixing selects a shared set of top-performing tasks across all sub-tasks based on average performance, while micro mixing selects the top-performing tasks separately for each sub-task. Interestingly, we observe that micro mixing consistently outperforms macro mixing, suggesting that different 1 https://artofproblemsolving.com/
2
Preprint. Under review.
Task Selection
Task Mixing
Task Interventions
Reformatting
Filtering
Macro
Aggregate
Multi-hop
Long-Range
RL Training
Ranking
Micro
Per-Task
Anti-prior
Constraints
Figure 2: S UPER N OVA Framework: In this work, we curate reasoning data from natural instruction to enhance general reasoning capabilities in LLMs. First, we study the impact of task selection on downstream reasoning performance. Then, we explore strategies to mix diverse tasks in source data. Finally, we examine whether synthetic data interventions can enhance data quality and improve downstream reasoning. reasoning skills benefit from different source tasks. Third, we examine whether synthetically generated data interventions (e.g., introducing long-context dependencies in questions) improve data quality. Surprisingly, we find that augmenting the original data with these interventions does not improve performance under a fixed training budget. Finally, we combine these insights to construct the S UPER N OVA dataset, comprising 25K RLVR samples that achieve state-of-the-art performance on challenging general reasoning benchmarks. Specifically, we train Qwen3 models of various sizes (0.6B–4B) on S UPER N OVA (Figure 1). We find that S UPER N OVA-4B achieves relative gains of 29.4% and 42.9% on pass@1 and pass@8, respectively, on BBEH-test (§ 6). These results show that (a) S UPER N OVA elicits strong general reasoning capabilities, and (b) performance improves with increased test-time compute (i.e., from 1 to 8 attempts per problem), highlighting improved exploration during reasoning. Notably, S UPER N OVA-4B outperforms the larger Qwen3-8B model by 8.2% on pass@8 on general reasoning tasks, demonstrating that S UPER N OVA enables training smaller yet stronger general reasoners. Furthermore, S UPER N OVA models exhibit strong generalization across additional reasoning benchmarks, including BBH (Suzgun et al., 2023), MMLU-Pro (Wang et al., 2024), and Zebralogic (Lin et al., 2025) (§ 6). In particular, S UPER N OVA-4B achieves a relative improvement of 12.3% on pass@8 over the baseline model across these benchmarks. Overall, our experiments provide practical guidelines for principled RLVR data curation for training strong general reasoners.
2
Preliminaries
Reinforcement Learning with Verifiable Rewards (RLVR). RLVR is widely adopted for training LLMs for reasoning in domains that rely on automatically verifiable ground truth such as mathematics and code. Given an input-target pair (q, t), RLVR samples G rollouts oi iG=1 from a behavior policy πθold and optimizes the GRPO (Shao et al., 2024) objective: "
JGRPO (θ ) = E(q,t)∼D, {oi }G ∼πθ i =1
old
(·|q)
# 1 G min ρi (θ ) Âi , clip(ρi (θ ), 1−ϵ, 1+ϵ) Âi , G i∑ =1 (1)
π (o |q)
where ρi (θ ) = π θ (io |q) is the importance sampling ratio. The group-centered advantage θold
i
Âi for each output is computed as Âi = ri − G1 ∑G j=1 r j where ri = r ( oi , q ), the computed reward. Following Yu et al. (2025), we skip the KL penalty to improve training efficiency in our experiments. Task-Specific Instruction Datasets. Instruction-tuning datasets such as SuperNI (Wang et al., 2022), and Flan-Collection (Wei et al., 2021) are a collection of well-structured, distinct tasks spanning diverse reasoning abilities. These datasets are constructed from high-quality human supervision including task definitions, instructions and ground-truth annotations. We observe that these large instruction-tuning datasets often encode reasoning structures that are not explicitly annotated but can be inferred from the examples and task structure. 3
Preprint. Under review.
Consider an instruction dataset D = { D1 , D2 , D3 ...DK } comprising K tasks where each subset Dk is a well-defined task targeting a particular skill. General Reasoning Benchmarks. Evaluation benchmarks that aim to evaluate models on general reasoning, such as BBEH and BBH, can be decomposed into sub-tasks that target a particular skill. This decomposability allows us to systematically assess how task-specific data translates to broad reasoning gains. We adopt BBEH as our validation benchmark as it consists of 23 diverse sub-tasks spanning linguistic, logical and commonsense reasoning (Appendix Table 5). Formally, we define our validation set as V = {V1 , V2 .....VN } where Vi denotes the sub-tasks in the benchmark. Problem Setup. In this work, we focus on the curation of high-quality training data to enable strong general reasoning capabilities via reinforcement learning. Given a pool of candidate datasets D = { D1 , D2 , . . . , DK }, a model M, and a training algorithm A, we seek a subset of tasks S ⊆ D that maximizes downstream performance after training. Following the data curation formulations proposed for SFT in math reasoning (Guha et al., 2025) and multimodal reasoning (Bansal et al., 2025), we define our objective as: S∗ = argmaxS⊆ D Φ( A( M, S), V )
(2)
where A( M, S) denotes the model after A to M on the selected subset S, V is the validation set and ϕ measures downstream performance on V.
3
S UPER N OVA ☼
We outline our S UPER N OVA framework (Figure 2), which consists of multiple stages: (a) task selection, which assesses the impact of task choice (§ 3.1); (b) mixing, which identifies the best strategy to mix the diverse tasks (§ 3.2); and (c) data interventions, which aim to enhance the quality of our data (§ 3.3) 3.1
Task Selection
Extracting reasoning data from instructions. The quality of the input and the coverage of reasoning types is critical for determining the reasoning skills imparted to the LLM. For example, a LLM exposed to temporal graphs will excel in downstream temporal understanding tasks (Xiong et al., 2024). In this work, we leverage instruction-tuning data D to source diverse tasks Dk for general reasoning. Since, instructions are formatted for supervised-finetuning they are not directly usable for RLVR as they may incorporate hard to verify ground-truth. Thus, for every instruction p in Dk , we reformat the instruction to a verifiable question q. To further identify the most effective data from D, we sample 8 rollouts from model M for each q and compute the per-question win-rate. Finally, we remove all questions which are too easy for the model (win-rate=1) or too challenging (win-rate=0). Task Ranking. For each task Dk , we define a task-utility score uk ∈ R that indicates how effective Dk is for RLVR training. Then, we rank the K tasks according to their utility scores, producing a ranking: u D1 > u D2 > u D3 > · · · > u DK ) . The task utility scores enable us to select high-quality tasks while downweighting poor and irrelevant tasks. We explore various approaches to compute task-utility: (a) we compute the semantic and lexical similarity between the task questions and the questions from our validation benchmark V; (b) we compute the difficulty of the task based on the average win-rate of the task under model M; and (c) we train model M on Dk and evaluate performance on V. We adopt approach (c) in our main experiments. 3.2
Mixing
After obtaining high-quality tasks, we determine how to combine them to construct an effective training mixture. Mixing strategy is a key design choice in data curation and prior work in LLM reasoning have shown to yield superior datasets by mixing subsets from 4
Preprint. Under review.
various sources. Consider the K tasks from § 3.1 and number of tasks to be mixed N ∈ {1, 2, 4, 8, 16}, we want to determine the optimal value of N under two mixing strategies: • Macro Mixing: Consider the ranking from § 3.1: u D1 > u D2 > u D3 .. > u DK where u Dk is the macro average of model performance on VBBEH . We select the top-ranked N tasks u D1 > u D2 > u D3 > · · · > u DN for our mixture. • Micro Mixing: Here, we leverage the sub-tasks of our V and produce a ranking (i )
for each sub-task Vi . Specifically, we define uk as the performance of model M trained on Dk and evaluated on sub-task Vi , yielding a per-sub-task ranking: (i )
(i )
(i )
u D1 > u D2 > · · · > u DK for each Vi ∈ V. We then select the top-ranked N tasks per sub-task and take the unique set of selected tasks for our mixture. 3.3
Data Interventions
Starting from the best mixture from §3.2, we assess whether we can enhance the data quality through targeted data interventions. RLVR datasets primarily focus on the questions since interventions on the target may hinder the verifiability of the answer. Thus, we apply a set of interventions to transform the difficulty of the questions while preserving the target answer. These interventions aim to increase the difficulty of the questions by introducing diverse reasoning types such very long-context dependency, information that prompts model to go against a strong prior or needle in haystack. Let Dbase = {(q, t)} be the base data with original question-target pairs. We apply an intervention I that transforms each question while preserving the target, producing D ′ = {( I (q), t)}. We provide the implementation details of applying these interventions in Appendix § G.
4
Experimental Setup
Training Data. We use SuperNI (Wang et al., 2022) as our data source. SuperNI consists of 1600 tasks spanning various tasks types such as question answering, question generation and commonsense reasoning. Each task consists of the task description and the instructionresponse pair, annotated by experts. For our experiments, we select a candidate pool of 83 tasks. We provide the candidate pool selection strategy in Appendix § B. Training. We train models from Qwen3 Yang et al. (2025) family (0.6B, 1.7B and 4B) with GRPO (Shao et al., 2024) for all our experiments (§ 2). For our data curation experiments, we use Qwen3-0.6B for faster training iterations. All our data curation experiments were run for 250 RL steps. Finally, we train the S UPER N OVA models for 5000 RL steps. We present more details about the training setup in Appendix B. Evaluation. We evaluate our models on various benchmarks that target diverse reasoning capabilities. For our data curation experiments, we choose BBEH-mini as our validation benchmark. BBEH-mini is a curated subset of BBEH (Kazemi et al., 2025) consisting of 460 examples spanning 23 tasks that target diverse reasoning capabilities. We use the remaining BBEH examples, which are not included in BBEH-mini as the unseen test set, BBEH-test. After creation of our S UPER N OVA dataset, we evaluate our models on 4 additional unseen benchmarks including BBH (Suzgun et al., 2023), Zebralogic (Lin et al., 2025), MMLUPro (Wang et al., 2024) and MATH500‘(Lightman et al., 2023). To ensure consistency, we use an identical prompt across all evaluations that encourages the model to think before answering, provided in Appendix B. Evaluation Metric. We adopt pass@k as our evaluation metric, which is well-suited for evaluating RL-trained models (Chen et al., 2021; Yue et al., 2025). As shown in Appendix § 6, we find that pass@8 provides 2.5 times greater discriminability than pass@1 (σ: 0.76 → 1.92). We therefore utilize pass@8 for our data curation experiments. 5
Preprint. Under review.
20
+39 +33 +32 +32
+31 +31 +31 +29 +29 +29 +28 +28 +28 +26 +25
Baseline (15.43%) 1 2 2 2 2 2 4 4 6 6 8 8 8 9 9
15 10 5 0
task738_perspectrum task003_mctaco task717_mmmlu task210_logic2text task249_enhanced task1385_anli task1296_wiki task828_copa task073_csqa task013_mctaco task018_mctaco task697_mmmlu task827_copa task383_matres task065_timetravel task1088_array task190_snli task1333_check task016_mctaco task1208_atomic task1386_anli task1516_imppres task199_mnli task1210_atomic task217_rocstories task1155_bard task218_rocstories task224_scruples task1211_atomic task213_rocstories
Pass@8 (%)
25
Figure 3: Impact of Task Selection. We train the baseline (Qwen3-0.6B) on each task individually under compute-matched settings. We report relative pass@8 gains on BBEHmini for each task and highlight the tasks that improve and degrade the baseline.
Baselines Models. We evaluate several models as baselines for our experiments. (1) Qwen3 and Qwen3.5 family: included to measure the gains obtained from training on S UPER N OVA. (2) General-Reasoner-Qwen3-4B (Ma et al., 2025): an all-domain reasoning model that is closest in motivation to our models. (3) OpenThinker3-7B (Guha et al., 2025): a strong math reasoning model supervised finetuned on large math corpus. (4) OpenReasoner-Nemotron-7B (Ahmad et al., 2025): a strong reasoning model. (5) Olmo37B-Think (Olmo et al., 2025): a state-of-art reasoning model. To further compare the quality of S UPER N OVA with other reasoning datasets, we train Qwen3-0.6B under a computematched setup using three baseline datasets including Nemotron-CrossThink (Akter et al., 2026), which targets various domains beyond math; General-Reasoner (Ma et al., 2025), which curates reasoning data across diverse STEM-focused domains; and DAPO (Yu et al., 2025), a high-quality math reasoning dataset sourced from competition websites. We provide additional details in Appendix § C.
5
Experiments
5.1
Impact of Task Selection
We train Qwen3-0.6B on each task and report model performance on BBEH in Fig. 3. Our experiments show that task selection has a substantial impact on downstream reasoning performance. Specifically, we observe a 7.6 percentage point (pp) gap between the lowestperforming task (task213-rocstories, −9pp vs. baseline) and the highest-performing task (task738-perspectrum, +39pp vs. baseline). This large spread indicates that tasks do not contribute equally to downstream reasoning. Notably, several tasks degrade performance relative to the baseline, underscoring that not all tasks are beneficial for improving general reasoning under RLVR. Furthermore, we find that tasks involving multi-hop reasoning yield the largest gains over the baseline model (Appendix § F). Additionally, we explore the efficacy of semantic similarity and lexical similarity between the source and validation tasks as a way to assess task utility (Appendix § H). However, we observe weak correlation between similarity scores and downstream performance on BBEH. 5.2
Impact of Task Mixing
We present the results of two mixing strategies: Macro Mixing and Micro Mixing in Table 1. Across both strategies, we find that mixing questions from the top two tasks yields the best results, with Micro Mixing achieving the highest pass@8 of 22.8%.This suggests a 6
Preprint. Under review.
Intervention
Top 1 Top 2 Top 4 Top 8 Top 16 Micro Mixing pass@1 pass@8
7.5 18.3
8.9 22.8
7.5 18.7
7.6 18.0
7.5 20.2
6.4 17.0
7.5 18.3
Macro Mixing pass@1 pass@8
7.6 21.5
8.2 21.7
6.6 17.4
Table 1: Impact of mixing. We mix questions from tasks following two strategies: micro mixing and macro mixing. We find that micro mixing with top 2 tasks achieves the best performance (bold).
Model
pass@8
Micro-Top2 Going Against Prior Long-Context Inductive Reasoning Finding Errors Many-hop Reasoning Knowledge-intensive Reasoning Compositional Understanding Learning on the Fly
22.8 22.6 21.3 20.4 20.0 20.0 19.8 19.6 18.3
Table 2: Impact of interventions. We compare the performance of models trained on datasets transformed using synthetic interventions. We find that the base dataset is superior to all interventions.
BBEH-mini
BBEH-test
pass@1
pass@8
pass@1
pass@8
Openthinker3-7B OpenReasoning-Nemotron-7B Olmo-3-7B-Think Qwen3-8B
3.4 3.5 6.4 11.9
7.6 8.5 13.0 21.7
4.4 4.8 6.8 14.1
10.1 11.0 14.9 24.2
Qwen3-0.6B Qwen3-0.8B S UPER N OVA-0.6B
5.7 6.6 9.8
15.4 22.2 24.6
5.2 7.4 9.7
15.2 23.8 25.0
Qwen3-1.7B Qwen3.5-2B S UPER N OVA-1.7B
7.6 7.5 12.4
18.0 22.4 25.7
7.8 8.1 11.7
17.7 25.8 26.7
Qwen3-4B General-Reasoner-4B S UPER N OVA-4B
12.0 11.5 14.8
23.7 30.0 31.7
13.6 12.0 17.6
23.2 32.9 33.3
Table 3: Performance of models trained with S UPER N OVA data. We compare the pass@1 and pass@8 of models on BBEH. We find that models trained on S UPER N OVA achieve best-in-class performance across model sizes. S UPER N OVA-4B even beat Qwen3-8B.
trade-off in task diversity: mixing too few tasks limits data diversity, while mixing too many degrades performance. Furthermore, Micro Mixing consistently outperforms Macro Mixing regardless of the number of tasks combined. These results indicate that selecting top-ranked tasks per sub-task (Micro Mixing) better preserves coverage across diverse reasoning skills, whereas selecting tasks based on overall ranking (Macro Mixing) biases the mixture toward a narrower set of abilities.
5.3
Impact of Data Interventions
We apply several data intervention strategies to the best-performing dataset from §5.2 (Micro-Top2) and report results on BBEH-mini in Table 2. Surprisingly, none of the interventions improve over the original data. While Going Against Prior achieves the highest performance among the interventions (22.6%), it still falls short of Micro-Top2. This suggests that synthetically generated interventions can degrade data quality, and that improving already high-quality data through such interventions is non-trivial. 7
Preprint. Under review.
Model Qwen3-0.6B S UPER N OVA-0.6B Qwen3-1.7B S UPER N OVA-1.7B Qwen3-4B S UPER N OVA-4B
MMLU-Pro
BBH
Zebralogic
MATH500
Average
55.3 56.2 64.3 61.5 71.2 76.0
52.4 81.5 80.3 79.7 84.4 89.3
34.4 49.1 53.3 75.5 55.6 77.0
71.9 71.4 73.2 84.0 74.1 77.9
53.5 64.6 67.8 75.2 71.3 80.1
Table 4: Performance of S UPER N OVA models on unseen benchmarks. We report pass@8 across four benchmarks that were unseen during data curation. We find that LLMs trained on S UPER N OVA show improved pass@8 across all model sizes.
6
Training Reasoners with S UPER N OVA
S UPER N OVA elicits strong general reasoning. We evaluate the performance of S UPER N OVA models across different scales (0.6B, 1.7B, and 4B) and compare them against several strong reasoning LLMs. Results are reported in Table 3. Notably, we find that models trained with S UPER N OVA achieve the best pass@1 and pass@8 on BBEH across all model sizes. In particular, S UPER N OVA-1.7B achieves relative gains of 44pp and 3.5pp over Qwen3.5-2B at pass@1 and pass@8, respectively while S UPER N OVA-4B outperforms of General-Reasoner4B by 46pp and 1.2pp. Remarkably, S UPER N OVA-4B outperforms Qwen3-8B—a 2× larger model—by 8.2pp on pass@8, highlighting the effectiveness of S UPER N OVA in training strong general reasoners even at smaller scales. We report per-sub-task pass@8 scores on BBEH-test in Appendix Table 5. S UPER N OVA beats SOTA reasoning datasets. We compare S UPER N OVA against three state-of-the-art reasoning datasets that target diverse reasoning skills. To ensure a fair comparison of data quality, we perform a compute-matched analysis of all datasets (details in Appendix § C). Results on BBEH-mini are shown in Figure 4. We find that S UPER N OVA achieves gains of 42pp on pass@1 and 28pp on pass@8 over the strongest baseline, Nemotron-Crossthink. In contrast, both math reasoning datasets, DAPO and NemotronCrossthink (Math) show little to no improvement over the baseline. Overall, under fixed compute, S UPER N OVA delivers substantially better reasoning performance than existing datasets.
Performance on BBEHmini (%)
S UPER N OVA generalizes to Out-of-Distribution (OOD) Benchmarks We evaluate S U PER N OVA on challenging reasoning benchmarks that are unseen during data curation, as shown in Table 4. We observe consistent improvements across all benchmarks and model sizes. Notably, S UPER N OVA achieves substantial gains on Zebralogic, where S UPER N OVA-
20 15
+28.0%
Baseline DAPO NemotronCrossThink (Math) General Reasoner NemotronCrossThink SuperNova
10
+42.6%
5 0
Pass@1
Evaluation Metric
Pass@8
Figure 4: Comparison with Other Reasoning Datasets. We report the relative gains achieved by training Qwen3-0.6B on S UPER N OVA and existing reasoning datasets. 8
Pass@8 on BBEHmini(%)
+50.5%
Baseline SuperNova
+15.8%
30 25 20 15
Pass@k on BBEHmini (%)
Preprint. Under review.
Qwen3.52B LLaMA3.23BInstruct Model
Qwen30.6B SuperNova0.6B
40 30 20 10 1
2
4
8
16
32
64
128
Number of Samples per Question (k, log scale)
Figure 5: (Left) We train Qwen3.5-2B and LLaMA3.2-3B-Instruct with S UPER N OVA and show relatives gains over the baseline model on BBEH-mini. (Right) We show the performance comparison between the baseline model and S UPER N OVA-0.6B by scaling values of k till 128 on BBEH-mini.
4B outperforms Qwen3-4B by 21pp. This suggests that training on S UPER N OVA enhances logical reasoning capabilities, particularly for constraint satisfaction tasks. We also find that S UPER N OVA-1.7B achieves an average score of 75.2%, exceeding Qwen3-4B (71.3%) despite being half its size, highlighting the efficacy of S UPER N OVA. Finally, S UPER N OVA models maintain competitive performance on MATH500, with modest gains, indicating that training on S UPER N OVA does not degrade mathematical reasoning capabilities. S UPER N OVA gains are consistent at larger values of k. We analyze whether the performance gains from S UPER N OVA persist at higher values of k. As shown in Figure 5 (Right), S UPER N OVA-0.6B maintains consistent gains over Qwen3-0.6B across all values of k up to 128. This suggests that training on S UPER N OVA expands the model’s exploration space even at large sample sizes, enabling more diverse reasoning behaviors than the baseline. S UPER N OVA shows cross-model generalization. We study how training on S UPER N OVA generalizes across model families (Figure 5 (Left)). In particular, LLaMA3.2-3BInstruct (Grattafiori et al., 2024) trained on S UPER N OVA achieves gains of 15.8pp over its baseline. We further observe similar improvements on Qwen3.5-2B (Team, 2026), suggesting that data curation insights derived from earlier-generation models (e.g., Qwen3) transfer to newer-generation models. Overall, the benefits of S UPER N OVA generalize across both model families and generations.
7
Conclusion
In this work, we propose S UPER N OVA for RLVR data curation to enhance the general reasoning capabilities of LLMs. S UPER N OVA leverages large-scale instruction-tuning datasets to curate RLVR data for general reasoning. Through controlled experiments, we surface several key insights. We show that task selection and micro mixing are critical for training strong general reasoners. Additionally, we find that augmenting data with synthetically generated interventions fails to improve reasoning performance. Additionally, we demonstrate that the gains from S UPER N OVA generalize across model families and challenging benchmarks. While our results demonstrate the effectiveness of S UPER N OVA, several directions remain for future work. First, we evaluate general reasoning on a fixed set of academic benchmarks that may not fully capture real-world problem solving. Second, our experiments are conducted under limited compute, future work may explore how our findings persist with unbounded compute and data. Finally, we believe S UPER N OVA has provided several practical insights that can spur further data curation efforts for RLVR beyond STEM domains. 9
Preprint. Under review.
Ethical Concerns Disclosure of LLM use in both research and reviewing. We use ChatGPT and Claude as in our experiments and have provided the relevant prompts. Claude was used in formatting latex tables and code generation for the figures. Finally, we used ChatGPT and Claude to assist with grammar and proof-reading in our paper writing.
References Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. OpenCodeReasoning: Advancing Data Distillation for Competitive Coding. arXiv preprint arXiv:2504.01943, 2025. URL https://arxiv.org/abs/2504.01943. Syeda Nahida Akter, Shrimai Prabhumoye, Matvei Novikov, Seungju Han, Ying Lin, Evelina Bakhturina, Eric Nyberg, Yejin Choi, Mostofa Patwary, Mohammad Shoeybi, et al. Nemotron-crossthink: Scaling self-learning beyond math reasoning. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 984–1002, 2026. Hritik Bansal, Devandra Singh Sachan, Kai-Wei Chang, Aditya Grover, Gargi Ghosh, Wentau Yih, and Ramakanth Pasunuru. Honeybee: Data recipes for vision-language reasoners. arXiv preprint arXiv:2510.12225, 2025. Adithya Bhaskar, Xi Ye, and Danqi Chen. Language models that think, chat better, 2025. URL https://arxiv.org/abs/2509.20357. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotron: Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505.16400, 2025. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Thomas L Griffiths. Understanding human intelligence through human limitations. Trends in Cognitive Sciences, 24(11):873–883, 2020. Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint arXiv:2506.04178, 2025. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025. Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue. Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning, 2025. URL https://arxiv. org/abs/2507.00432. 10
Preprint. Under review.
Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1. Philip N Johnson-Laird. Mental models and human reasoning. Proceedings of the National Academy of Sciences, 107(43):18243–18250, 2010. Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K Jain, Virginia Aglietti, Disha Jindal, Yuanzhu Peter Chen, et al. Big-bench extra hard. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 26473–26501, 2025. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The twelfth international conference on learning representations, 2023. Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of llms for logical reasoning. arXiv preprint arXiv:2502.01100, 2025. Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, Yunan Huang, Mozhi Zhang, Pengyu Zhao, Junjie Yan, and Junxian He. Synlogic: Synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond, 2025. URL https://arxiv.org/abs/2505.19641. Yang Liu, Jiaqi Li, and Zilong Zheng. Rulereasoner: Reinforced rule-based reasoning via domain-aware dynamic sampling, 2026. URL https://arxiv.org/abs/2506.08672. Ximing Lu, David Acuna, Jaehun Jung, Jian Hu, Di Zhang, Shizhe Diao, Yunheng Zou, Shaokun Zhang, Brandon Cui, Mingjie Liu, Hyunwoo Kim, Prithviraj Ammanabrolu, Jan Kautz, Yi Dong, and Yejin Choi. Golden goose: A simple trick to synthesize unlimited rlvr tasks from unverifiable internet text, 2026. URL https://arxiv.org/abs/2601.22975. Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. Generalreasoner: Advancing llm reasoning across all domains. arXiv preprint arXiv:2505.14652, 2025. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20286–20332, 2025. Allen Newell, Herbert Alexander Simon, et al. Human problem solving, volume 104. Prenticehall Englewood Cliffs, NJ, 1972. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3. arXiv preprint arXiv:2512.13961, 2025. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging bigbench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051, 2023. Qwen Team. Qwen3. 5: Towards native multimodal agents. URL: https://qwen. ai/blog, 2026. 11
Preprint. Under review.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 conference on empirical methods in natural language processing, pp. 5085–5109, 2022. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10452–10470, 2024. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892, 2025. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction tuning for large language models: A survey, 2025. URL https://arxiv.org/abs/2308.10792. Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024. https://huggingface.co/datasets/math-ai/aime24, 2024. Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 million open-source distilled reasoning dataset to empower large language model training. arXiv preprint arXiv:2503.19633, 2025. Ruochen Zhou, Minrui Xu, Shiqi Chen, Junteng Liu, Yunqi Li, Xinxin Lin, Zhengyu Chen, and Junxian He. Does learning mathematical problem-solving generalize to broader reasoning?, 2025. URL https://arxiv.org/abs/2507.04391.
12
Preprint. Under review.
A
Related Work
A.1
General-Purpose Reasoning in LLMs
Several works have explored expanding the general reasoning capabilities of LLMs. Ma et al. (2025) constructs a large-scale dataset spanning multiple domains such as history, finance, and physics from web-scraped sources. Akter et al. (2026) goes beyond mathematics by curating synthetically derived questions from CommonCrawl and open-source QA datasets. Lu et al. (2026) leverages transformed pretraining data with structured templates and distractors to generate verifiable reasoning data in domains such as cybersecurity. However, these approaches largely rely on internet-sourced data, which can be noisy and of low quality. Other work has focused on rule-based tasks (Liu et al., 2026) and logic puzzles (Liu et al., 2025). While effective for specialized reasoning, these approaches rely on highly curated logic and rule-based datasets that are challenging to scale and cover a limited range of reasoning types. In contrast, S UPER N OVA leverages instruction-tuning datasets, which are human-annotated and generally of higher quality than raw internet data, enabling broader general reasoning capabilities. A.2
Data Curation for Reasoning
High-quality reasoning data is critical for training strong LLM reasoners. Prior work has focused on large-scale datasets for supervised fine-tuning (SFT) (Hugging Face, 2025; Zhao et al., 2025) and RLVR (Chen et al., 2025; Hu et al., 2025), typically by scraping competition websites or distilling knowledge from larger models. On the other hand, (Muennighoff et al., 2025; Ye et al., 2025) demonstrate that carefully curated, high-quality reasoning datasets can yield strong gains even with relatively small datasets. Guha et al. (2025) systematically studies data design principles for SFT reasoning data at scale through controlled experiments, in a manner similar to S UPER N OVA. However, these efforts primarily focus on reasoning in formal domains using SFT. SFT aims to improve instruction-following by mimicking gold responses (Zhang et al., 2025; Wang et al., 2022), while RLVR optimizes a sparse, outcome-based reward (Guo et al., 2025). Moreover, SFT typically requires complete reasoning traces and solutions for training, whereas RLVR only requires the final answer. As a result, SFT-oriented data curation strategies do not directly transfer to RLVR. S UPER N OVA addresses this gap by providing key insights to drive data curation for RLVR with a focus on general reasoning.
B
Detailed Experimental Setup
B.1
Task Selection
We utilize Claude-Opus-4.6, to select a candidate set of 83 tasks from the 1600 tasks of SuperNI. To ensure we can conduct a controlled study with our limited compute and keep our search space tractable, we prompt the LLM with a minimal prompt and do not consider the validation benchmark while preparing this candidate pool. This was done to ensure that the task ranking is done purely on task utility scores. The prompt follows: This is an instruction-following task use to train LLMs. Consider, the given task description and examples. Now assess the suitability of the task for RL training reasoning models. Think step by step and only respond with yes/no. Task ID: {task_id} Task Description: {description} Example Input: {input} Example Output: {output} For reformatting the instruction tuning tasks to verifiable questions, we prompt GPT-5-mini with B.1. To estimate the quality of the reformatting, we manually inspect 100 samples and find that GPT-5-mini follows the prompt accurately on 100% of the samples while preserving the ground-truth and original task structure.
13
Preprint. Under review.
Prompt for Reformatting Instruction-Tuning Dataset 1 2 3
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Role : You a r e an e x p e r t D a t a s e t Engineer s p e c i a l i z i n g i n Reinforcement Learning from Human Feedback (RLHF) and V e r i f i a b l e Rewards . O b j e c t i v e : Transform a raw t a s k d e s c r i p t i o n , input , and output i n t o a s t r u c t u r e d Problem and S o l u t i o n p a i r . This p a i r must be s u i t a b l e f o r RL t r a i n i n g where t h e reward i s c a l c u l a t e d v i a e x a c t −match v e r i f i c a t i o n . Constraints : The Problem : Must i n c o r p o r a t e a l l n e c e s s a r y c o n t e x t from t h e input and output without g i v i n g away t h e output . The S o l u t i o n : Must c o n t a i n only t h e f i n a l answer . No e x p l a n a t i o n s , no " The answer i s . . . " , and no punctuation u n l e s s i t i s p a r t o f t h e value . V e r i f i a b i l i t y : The s o l u t i o n must be uniquely e x t r a c t a b l e v i a simple s t r i n g matching or regex . Formatting Logic : Open−ended : Use t h i s i f t h e answer i s a unique value ( e . g . , a number , a s p e c i f i c name , or a c o n s t a n t ) . MCQ ( M u l t i p l e Choice ) : Use t h i s i f t h e t a s k i s s u b j e c t i v e , has m u l t i p l e v a l i d answers , or i n v o l v e s Yes/No . MCQ Format : Provide o p t i o n s l a b e l e d (A) through ( J ) . I f t h e t a s k i s multi − c o r r e c t , t h e s o l u t i o n should be a comma− s e p a r a t e d l i s t o f l e t t e r s ( e . g . , "A, C " ) . Output Format : Return a v a l i d JSON o b j e c t with t h e keys " f o r m a t t i n g l o g i c " , " problem " and " s o l u t i o n " . Task D e s c r i p t i o n : { d e f _ t a s k } Input : { example_input } Output : { example_output }
Finally, for win-rate filtering we generate 8 samples from Qwen3-0.6B at temperature=0.7 (generation length: 4096) , our baseline model and compute the per-question win-rate across these 8 samples. We filter all questions with a win-rate of 0 (too hard) and a win-rate of 1(too easy). B.2
Training
All our experiments were done on 4xH100 gpus. We use the GRPO implementation from TRL2 for our training. All our data curation experiments utilize Qwen3-0.6B with 500 prompts, learning rate of 1e-6, 8 generations per prompt, batch size of 8, decoding temperature of 0.7 and maximum generation length 4096. We run our training for 250 steps (1 epoch). For our large scale experiments, we run 5000 steps (1 epoch) across 10,000 prompts and use a learning rate of 1e-6 for 0.6B models and 4e-6 for 1.7B and 4B models. B.3
Evaluation
For our evaluations, we use the following prompt across all benchmarks Think step by step, and when you are ready to provide the final answer, use the prefix "The answer is:" followed by the answer directly, with no formatting and no markup. For instance: "The answer is: 42", or "The answer is: yes", or "The answer is: (a)" For multi-choice questions, provide the letter, e.g. "The answer is: (a) 2 https://github.com/huggingface/trl
14
Preprint. Under review.
Model
Avg.
Geo.
Bool.
Shuf.
M.Ar.
Zebra
Hyp.
WoL
Dis.
Word
Brd.
NYCC
T.Ar.
Mov.
Caus.
Sarc
Dyck
SuperNova-0.6B Qwen3-0.6B Qwen3.5-0.8B
25.0 15.2 23.8
35.3 0 5.9
56.8 40.5 27
48.8 2.3 20.9
0 0 0
48 20 2
5.6 2.8 13.9
7.1 19 23.8
60.6 15.2 78.8
14.6 2.4 17.1
53.1 46.9 71.4
27.1 20.8 41.7
18.9 16.2 16.2
47.1 25.5 37.3
76.7 53.5 81.4
44 50 64
7.5 5 10
SuperNova-1.7B Qwen3-1.7B Qwen3.5-2B
27.6 17.7 25.5
61.8 0 11.8
37.8 32.4 37.8
51.2 4.7 33.7
0 0 3.3
34 4 1
0 2.8 19.4
38.1 23.8 27.4
57.6 45.5 77.3
9.8 9.8 32.9
67.3 51 40.8
16.7 14.6 40.6
40.5 35.1 23
62.7 31.4 57.8
53.5 60.5 79.1
44 44 56
2.5 5 8.8
SuperNova-4B Gen.-Reasoner-4B Qwen3-4B
33.5 32.9 23.2
54.4 55.9 0
56.8 62.2 5.4
41.9 23.3 0
25.6 6.7 2.2
33 20 10
22.2 8.3 0
47.6 45.2 26.2
65.2 72.7 51.5
42.7 24.4 36.6
71.4 79.6 65.3
30.2 35.4 25
51.4 37.8 48.6
65.7 78.4 64.7
66.3 72.1 67.4
22 38 30
20 12.5 52.5
Qwen3-8B OLMo-3-7B-Think OpenReasoning-7B OpenThinker3-7B
24.2 14.9 11.1 10.1
2.9 0 0 0
5.4 0 0 0
2.3 0 0 0
0 0 0 0
10 0 0 0
0 0 0 0
31 2.4 2.4 0
57.6 48.5 60.6 54.5
36.6 24.4 9.8 17.1
59.2 6.1 4.1 12.2
20.8 20.8 29.2 12.5
67.6 16.2 0 8.1
51 78.4 19.6 9.8
79.1 62.8 65.1 58.1
38 44 34 28
42.5 2.5 0 5
Table 5: BBEH pass@8 results (%) on 16 selected tasks, grouped by model size. 7 nearzero tasks omitted (buggy tables, linguini, object counting, object properties, spatial reasoning, sportqa, temporal sequence). Best in each size class is bolded. Task abbreviations: Geo.=Geometric Shapes, Bool.=Boolean Expressions, Shuf.=Shuffled Objects, M.Ar.=Multistep Arithmetic, Hyp.=Hyperbaton, WoL=Web of Lies, Dis.=Disambiguation QA, Word=Word Sorting, Brd.=Boardgame QA, T.Ar.=Time Arithmetic, Mov.=Movie Recommendation, Caus.=Causal Understanding.
All evaluations were conducted on 1xH100 with a batch size of 8. We use decoding temperature=0.7 with maximum generation length of 4096 across all our experiments.
C
Implementation Details of Training Baseline Datasets
For fair comparison across dataset quality, we train Qwen3-0.6B on the fixed budget of 250 RL steps across 500 prompts and the same learning rate for all datasets. Since NemotronCrossthink, Dapo and General-Reasoner are large-datasets, we report their performance as average pass@8 across three runs trained on three random samples of 500 prompts.
D
Additional Results on BBEH
We provide the per sub-task pass@8 scores of S UPER N OVA and baseline models on BBEHtest in Table 5. We observe negligible gains on 7 out of the 23 sub-tasks. We observe that S UPER N OVA is able to improve on tasks like Hyperbaton, Multi-step Arithmetic and Shuffling Objects where the base model has near zero performance.
E
Pass@k Analysis
Following Chen et al. (2021) and Yue et al. (2025), we analyze the pass@k curves of our taskspecific models. Across 80+ RL curves, we observe that the spread and distinguishability of model performance increases at k=8, with maximum overlap at k=1. We show the pass@k curves in Figure 6.
F
Task Analysis
We prompt an LLM (Claude-Opus-4.6) with the task descriptions from each task and generate coarse category labels. We find that Multi-hop Reasoning and Coreference resolution emerge as the strongest categories, while narrative and surface-formatting tasks (e.g., Story Coherence, Date/Temporal format) consistently underperform (Figure 7). However, these aggregate trends obscure variations at the task-level. Despite Textual Entailment & NLI ranking in the middle at the category-level, task738_perspectrum emerges as the top-ranked task by large margin. This highlights that coarse category labels are insufficient for task selection and effective data curation for RL should be driven by fine-grained task utility analysis. 15
Pass@k (%)
Preprint. Under review.
20.0 17.5 15.0 12.5 10.0 7.5 5.0
=1.92
Base model =1.44 =1.07 =0.76
1
2
4
8
k
Figure 6: Pass@k accuracy of task-specific models across various values of k.
Baseline 15.43%
Ethical & Value judgment Date / Temporal format Science QA & World knowledge Story coherence & Abductive String Program Execution Textual Entailment & NLI Logical & Formal reasoning Commonsense & Social reasoning Multistep & Math reasoning Temporal & Causal Reasoning Coreference resolution Multihop & Knowledge retrieval 14
16
Above baseline Below baseline
18
20
Mean Pass@8 (%)
22
Figure 7: (a)We categorize the source tasks based on target reasoning skill and task type. We report Mean Pass@8 across each task category and highlight the categories which degrade baseline (Qwen3-0.6b) model on BBEH-mini.
16
Preprint. Under review.
G
Data Interventions
Following Kazemi et al. (2025), we design the given 7 interventions to improve data quality (Table 6) and prompt GPT-5-mini with G. Since, we want to preserve the ground-truth answer, we apply these interventions only to the problem statement. Finally, to ensure the that the final answer is preserved, filter the augmented data with based on win-rate computed again with the augmented problem statements. In our experiments, we combine the original data and the intervened data in a ratio of 1:1. Dimension
Description
Many-hop reasoning
Add information that increases the number of reasoning steps needed to reach the answer. Going against strong prior Add context that creates a misleading prior belief which conflicts with the correct answer, tempting the model to answer incorrectly based on surface-level associations. Learning on the fly Introduce a new rule, definition, or convention within the problem that must be understood and applied to solve it. Long-context Pad the problem with additional (but non-answer-changing) context to increase overall length. Finding errors in reason- Include a flawed reasoning chain within the problem that the ing traces model must recognize as incorrect. Inductive reasoning Provide a set of examples that establish a pattern, requiring the model to induce and apply the pattern. Add extra constraints that the model must track, even though Constraint satisfaction they do not affect the final answer. Compositional under- Fuse an independent sub-problem into the main problem, requirstanding ing the model to separate and solve them independently. Knowledge-intensive rea- Add domain-specific terminology or context that requires spesoning cialized knowledge to parse, even though it does not change the answer. Table 6: Following Kazemi et al. (2025), we design these interventions to improve the data quality. We provide the interventions and their definitions here.
Prompt for Data Intervention 1 2 3 4 5 6 7 8 9 10 11
You a r e an e x p e r t data augmentation a s s i s t a n t . Your t a s k i s t o t a k e an e x i s t i n g ( problem , answer ) p a i r from an NLP d a t a s e t and i n j e c t a d i s t r a c t o r i n t o t h e problem . The d i s t r a c t o r must make t h e problem harder f o r an AI model t o solve , but i t must NOT change t h e ground− t r u t h answer . ## DISTRACTOR TYPE You MUST use t h e f o l l o w i n g d i s t r a c t o r type : * * { distractor_name } * * : { d i s t r a c t o r _ d e s c r i p t i o n } ## RULES 1 . * * Answer p r e s e r v a t i o n ( CRITICAL ) * * : The ground− t r u t h answer MUST remain e x a c t l y t h e same a f t e r d i s t r a c t o r i n j e c t i o n . Do not a l t e r t h e c o r e r e a s o n i n g cha in . 2 . * * N a t u r a l n e s s * * : The d i s t r a c t o r must read n a t u r a l l y within t h e problem . I t should not f e e l a r t i f i c i a l l y i n s e r t e d or out o f place . 3 . * * P l a u s i b i l i t y * * : The d i s t r a c t o r should be p l a u s i b l e and c o n t e x t u a l l y r e l e v a n t enough t h a t a model might be misled by i t .
17
Preprint. Under review.
12 13
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58
4 . * * Minimal i n v a s i o n * * : Modify only what i s n e c e s s a r y . Do not r e w r i t e t h e e n t i r e problem . I n j e c t t h e d i s t r a c t o r i n t o or around the e x i s t i n g t e x t . 5 . * * D i f f i c u l t y c a l i b r a t i o n * * : The d i s t r a c t o r should make t h e problem meaningfully harder , not t r i v i a l l y so . Aim f o r a d i f f i c u l t y i n c r e a s e t h a t would cause a mid− t i e r model t o f a i l while a s t r o n g model would s t i l l succeed . ## OUTPUT FORMAT You MUST respond with a v a l i d JSON o b j e c t and nothing e l s e . No markdown , no e x p l a n a t i o n o u t s i d e t h e JSON . Use t h e f o l l o w i n g schema : {{ " original_problem " : " " , " original_solution " : "" , " augmented_problem " : " " , " augmented_solution " : " " , " distractor_metadata " : { { " distractor_types_used " : [ {{ " name " : " " , " description " : "" }} ], " injected_text_summary " : " " , " why_answer_unchanged " : " " , " estimated_difficulty_increase " : "" }} }} ## IMPORTANT GUIDELINES − Think s t e p by s t e p b e f o r e g e n e r a t i n g t h e output . − F i r s t , understand what t h e problem i s asking and why t h e given answer i s c o r r e c t . − Second , i d e n t i f y which p a r t s o f t h e problem can be augmented without breaking t h e answer . − Third , apply t h e s p e c i f i e d d i s t r a c t o r type as n a t u r a l l y as possible . − Fourth , d r a f t t h e d i s t r a c t o r t e x t . − F i f t h , v e r i f y t h a t t h e answer i s s t i l l c o r r e c t with t h e d i s t r a c t o r in place . − Only then produce t h e f i n a l JSON output . ## FINAL CHECKLIST ( v e r i f y b e f o r e o u t p u t t i n g ) − [ ] I s t h e output v a l i d JSON? − [ ] I s t h e answer i n augmented_problem i d e n t i c a l t o t h e o r i g i n a l answer ? − [ ] Does t h e d i s t r a c t o r read n a t u r a l l y i n c o n t e x t ? − [ ] I s t h e s p e c i f i e d d i s t r a c t o r type used with a c l e a r description ? − [ ] I s t h e why_answer_unchanged f i e l d f i l l e d with a l o g i c a l explanation ? − [ ] Would t h e augmented problem genuinely be harder f o r a model t o s o l v e ? Now, p r o c e s s t h e f o l l o w i n g input and r e t u r n t h e JSON output : Problem : Statement : { problem } Solution : { solution } """
18
Pass@8 on BBEHmini(%)
Preprint. Under review.
r = 0.17
r = 0.10
r = 0.20
20 18 16 14 25
30
35
40
45
Semantic Similarity (%)
50
55
6
8
10
12
Lexical Overlap (%)
14
0
20
40
60
Win Rate (%)
80
Figure 8: (Left) Correlation between semantic similarity and task performance. (Middle) Correlation between lexical similarity and task performance. (Right) Correlation between win rate and task performance. We observe weak correlation for all three approaches with downstream task performance on BBEH-mini.
Figure 9: Distribution of reasoning skills in S UPER N OVA.
H
Similarity and Task Difficulty
We find that semantic similarity and lexical similarity between the tasks and validation benchmark are poor predictors of task utility for RLVR. As shown in Fig. 8, both measures exhibit weak correlation with model performance on BBEH. While these approaches are attractive because they are cheap, fast to implement and model agnostic, our findings suggest that surface similarity is insufficient for task selection. We also investigated whether task difficulty, measured by the average win-rate of the base model, predicts downstream reasoning performance in Fig. 8. Similar to surface similarity, we observe only a weak correlation between task difficulty and model performance on BBEH, indicating that task difficulty is also a poor predictor of task utility for RLVR. Overall, our findings underscore that effective task selection relies on controlled, iterative, and compute-matched RL training.
I
Micro Mixing
We provide the top tasks ranked per sub-task in Table 7. For Micro-Top1, 16 unique tasks are selected while 31 unique tasks are included in Micro-Top2. Additionally, we show the distribution of reasoning skills as categorized in § F in S UPER N OVA which is scaled from Micro-Top2 and comprises 31 unique tasks. Table 7: Top-5 training tasks per BBEH task. BBEH Task movie recommendation
Rank Task ID 1
task827
Training Task copa_commonsense_reasoning continued on next page
19
Preprint. Under review.
Table 7 – continued from previous page BBEH Task
Rank Task ID
Training Task
2 3 4 5
task069 task212 task1297 task1209
abductivenli_classification logic2text_classification qasc_question_answering atomic_classification_objectuse
word sorting
1 2 3 4 5
task828 task1548 task1385 task835 task383
copa_commonsense_cause_effect wiqa_binary_classification anli_r1_entailment mathdataset_answer_generation matres_classification
object counting
1 2 3 4 5
task1210 task1211 task1155 task827 task004
atomic_classification_madeupof atomic_classification_hassubevent bard_analogical_reasoning_trash_or_treasure copa_commonsense_reasoning mctaco_answer_generation_event_duration
geometric shapes
1 2 3 4 5
task249 task1209 task1385 task697 task717
enhanced_wsc_pronoun_disambiguation atomic_classification_objectuse anli_r1_entailment mmmlu_answer_generation_formal_logic mmmlu_answer_generation_logical_fallacies
nycc
1 2 3 4 5
task1297 task073 task212 task213 task828
qasc_question_answering commonsenseqa_answer_generation logic2text_classification rocstories_correct_ending_classification copa_commonsense_cause_effect
boardgame qa
1 2 3 4 5
task004 task116 task062 task1726 task1387
mctaco_answer_generation_event_duration com2sense_commonsense_reasoning bigbench_repeat_copy_logic mathqa_correct_answer_generation anli_r3_entailment
buggy tables
1 2 3 4 5
task007 task1390 task600 task391 task004
mctaco_answer_generation_transient_stationary wscfixed_coreference find_the_longest_common_substring_in_two_strings causal_relationship mctaco_answer_generation_event_duration
linguini
1 2 3 4 5
task004 task1209 task640 task085 task738
mctaco_answer_generation_event_duration atomic_classification_objectuse esnli_classification unnatural_addsub_arithmetic perspectrum_classification
boolean expressions
1 2 3 4 5
task850 task600 task1390 task018 task210
synthetic_longest_palindrome find_the_longest_common_substring_in_two_strings wscfixed_coreference mctaco_temporal_reasoning_presence logic2text_structured_text_generation
multistep arithmetic
1 2 3 4 5
task004 task1210 task1211 task007 task1390
mctaco_answer_generation_event_duration atomic_classification_madeupof atomic_classification_hassubevent mctaco_answer_generation_transient_stationary wscfixed_coreference
time arithmetic
1 2 3 4 5
task212 task835 task383 task1209 task1153
logic2text_classification mathdataset_answer_generation matres_classification atomic_classification_objectuse bard_analogical_reasoning_affordance
object properties
1
task828
copa_commonsense_cause_effect continued on next page
20
Preprint. Under review.
Table 7 – continued from previous page BBEH Task
Rank Task ID
Training Task
2 3 4 5
task004 task1210 task1211 task007
mctaco_answer_generation_event_duration atomic_classification_madeupof atomic_classification_hassubevent mctaco_answer_generation_transient_stationary
hyperbaton
1 2 3 4 5
task249 task213 task393 task600 task827
enhanced_wsc_pronoun_disambiguation rocstories_correct_ending_classification plausible_result_generation find_the_longest_common_substring_in_two_strings copa_commonsense_reasoning
sarc triples
1 2 3 4 5
task210 task640 task970 task850 task717
logic2text_structured_text_generation esnli_classification sherliic_causal_relationship synthetic_longest_palindrome mmmlu_answer_generation_logical_fallacies
zebra puzzles
1 2 3 4 5
task828 task863 task210 task1726 task738
copa_commonsense_cause_effect asdiv_multiop_question_answering logic2text_structured_text_generation mathqa_correct_answer_generation perspectrum_classification
spatial reasoning
1 2 3 4 5
task738 task087 task019 task080 task697
perspectrum_classification new_operator_addsub_arithmetic mctaco_temporal_reasoning_category piqa_answer_generation mmmlu_answer_generation_formal_logic
shuffled objects
1 2 3 4 5
task249 task018 task827 task1297 task717
enhanced_wsc_pronoun_disambiguation mctaco_temporal_reasoning_presence copa_commonsense_reasoning qasc_question_answering mmmlu_answer_generation_logical_fallacies
temporal sequence
1 2 3 4 5
task004 task1210 task1211 task007 task1390
mctaco_answer_generation_event_duration atomic_classification_madeupof atomic_classification_hassubevent mctaco_answer_generation_transient_stationary wscfixed_coreference
sportqa
1 2 3 4 5
task270 task210 task062 task600 task391
csrg_counterfactual_context_generation logic2text_structured_text_generation bigbench_repeat_copy_logic find_the_longest_common_substring_in_two_strings causal_relationship
web of lies
1 2 3 4 5
task1152 task828 task1211 task080 task640
bard_analogical_reasoning_causation copa_commonsense_cause_effect atomic_classification_hassubevent piqa_answer_generation esnli_classification
causal understanding
1 2 3 4 5
task383 task004 task1390 task828 task291
matres_classification mctaco_answer_generation_event_duration wscfixed_coreference copa_commonsense_cause_effect semeval_2020_task4_commonsense_validation
disambiguation qa
1 2 3 4 5
task697 task1296 task717 task018 task065
mmmlu_answer_generation_formal_logic wiki_hop_question_answering mmmlu_answer_generation_logical_fallacies mctaco_temporal_reasoning_presence timetravel_consistent_sentence_classification
dyck languages
1
task1390
wscfixed_coreference continued on next page
21
Preprint. Under review.
Table 7 – continued from previous page BBEH Task
Rank Task ID 2 3 4 5
J
task1386 task004 task1210 task1152
Training Task anli_r2_entailment mctaco_answer_generation_event_duration atomic_classification_madeupof bard_analogical_reasoning_causation
Additional Details about SuperNI tasks
We provide the task descriptions of the 83 tasks that we include in our candidate pool in Table 8. These tasks are arranged in the order of maximum performance on BBEH-mini. Table 8: Task descriptions. Task Name
Summary
task738 perspectrum classification
Decide whether the given perspective supports or undermines the given claim. Writing questions that involve commonsense understanding of “event duration”. Answering multiple choice questions on logical fallacies. Given a sentence and a pronoun, decide which one of the choices the pronoun is referring to. Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise. Given a subject, a relation, and a context, find the object with that relation to the subject. Given a pair of sentences, judge whether the second sentence is the cause or effect of the first one. Answer questions based on commonsense knowledge. Checking the presence of temporal reasoning in a question. Answering multiple choice questions on formal logic. Given a premise and two alternatives, select the alternative that more plausibly has a causal relation with the premise. Given a context and a verb, answer if the given verb can be anchored in time or not. Choosing the option that makes a given short story consistent. Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise. Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise. Given a mathematical question involving multiple operations, find the most suitable numerical answer. Given a tuple, determine whether the Head is used for the Tail or not.
task003 mctaco question generation event duration task717 mmmlu answer generation logical fallacies task249 enhanced wsc pronoun disambiguation task1385 anli r1 entailment task1296 wiki hop question answering task828 copa commonsense cause effect task073 commonsenseqa answer generation task018 mctaco temporal reasoning presence task697 mmmlu answer generation formal logic task827 copa commonsense reasoning task383 matres classification task065 timetravel consistent sentence classification task640 esnli classification task1387 anli r3 entailment task863 asdiv multiop question answering task1209 atomic classification objectuse
Continued on next page
22
Preprint. Under review.
Task Name
Summary
task212 logic2text classification
Given a command, classify the command in one of seven logic types. Given a mathematical question, find the most suitable numerical answer. Answering questions that involve commonsense understanding of event ordering. Given two facts and a multiple-choice question, answer the question. Answering questions that involve commonsense understanding of transient vs. stationary events. Given a context, a pronoun, and a noun, determine if the pronoun in the context refers to the noun or not. Given two strings return the longest common substring in those two strings. Generate a solution to a goal regarding physical knowledge about the world. Generate correct answers for math questions.
task750 aqua multiple choice answering task010 mctaco answer generation event ordering task1297 qasc question answering task007 mctaco answer generation transient stationary task1390 wscfixed coreference task600 find the longest common substring in two strings task080 piqa answer generation task1726 mathqa correct answer generation task835 mathdataset answer generation task580 socialiqa answer generation task1393 superglue copa text completion task1727 wiqa what is the effect task170 hotpotqa answer generation task133 winowhy reason plausibility detection task004 mctaco answer generation event duration task019 mctaco temporal reasoning category task229 arc answer generation hard task106 scruples ethical judgment task178 quartz question answering task1152 bard analogical reasoning causation task090 equation learner algebra task850 synthetic longest palindrome task1422 mathqa physics task393 plausible result generation task085 unnatural addsub arithmetic
Find the numerical answer for a math word problem. Given a context, a question and three options, provide the correct answer based on the context. Given a premise sentence, two possible options and a question word, choose the best option. Find the effect of an event on another event, based on an introduced process. Given a set of context and supporting facts, answer the question asked. Detect if a reason that explains an answer to a pronoun coreference resolution question is correct or not. Answering questions that involve commonsense understanding of event duration. Verifying the temporal reasoning category of a given question. Given a hard science question, provide the answer based on scientific facts and reasoning. Given two actions choose the one that is considered less ethical. Given a question, select the correct answer from the given options using an explanation. Given an analogy that relates actions with their consequences, give the appropriate consequence of the given action. Answer the given equation. Given a string find the longest substring that is a palindrome. Given a problem on physics and options to choose from, find the correct option that answers the problem. Given a sentence, write another sentence that is a likely result of it. Performing arithmetic with swapped operator symbols. Continued on next page
23
Preprint. Under review.
Task Name
Summary
task1529 scitail1.1 classification
Determining if there is entailment between hypothesis and premise. Given a mathematical question involving multiple operations, find the most suitable numerical answer. Given a command and corresponding interpretation, classify whether it is the right interpretation or not. Binary classification based on steps in wiqa. Fact checking based on given context.
task867 mawps multiop question answering task211 logic2text classification task1548 wiqa binary classification task966 ruletaker fact checking based on given context task935 defeasible nli atomic classification task116 com2sense commonsense reasoning task087 new operator addsub arithmetic task206 collatz conjecture task970 sherliic causal relationship task086 translated symbol arithmetic task270 csrg counterfactual context generation task392 inverse causal relationship task105 story cloze-rocstories sentence generation task1507 boolean temporal reasoning task1404 date conversion task1153 bard analogical reasoning affordance task069 abductivenli classification task062 bigbench repeat copy logic task1088 array of products task190 snli classification task1333 check validity date ddmmyyyy task016 mctaco answer generation frequency task1208 atomic classification xreason task1386 anli r2 entailment task1516 imppres naturallanguageinference
Given a premise, hypothesis and an update, identify whether the update strengthens or weakens the hypothesis. Decide whether a sentence is plausible and matches commonsense. Performing arithmetic with newly defined operator symbols. Given a list of integers, compute the next number in the 3n+1 problem. Determine if A and B share a causal relationship. Performing arithmetic with translated operator symbols. Given a premise, initial context with ending, and new counterfactual ending, generate counterfactual context which supports the new story ending. Given two sentences, decide whether the first sentence can be the result of the second one. Given four sentences, predict the next coherent sentence. Given a statement about date and time values, deduce whether it is true or false. Given a date in a particular format, convert it into some other format. Given an analogy that signifies affordances, give the appropriate affordance of the given action. Choosing text that completes a story based on given beginning and ending. Generating text that follows simple logical operations such as repeat, before, after etc. Given an integer array, return an array such that its element at each location is equal to the product of elements at every other location in the input array. Given two sentences choose whether they agree, disagree, or neither with each other. Given a date in dd/mm/yyyy format, check if it is a valid date or not. Answering questions that involve commonsense understanding of event frequency. Given a tuple, determine whether the Tail is the reason for the Head or not. Given a premise and hypothesis, determine if the hypothesis entails, contradicts, or is neutral to the premise. Classify a given premise and hypothesis pair. Continued on next page
24
Preprint. Under review.
Task Name
Summary
task199 mnli classification
Given 2 sentences, determine if they clearly agree or disagree with each other or if they cannot be answered. Given a tuple, determine whether the Head is made of the Tail or not. Given a five sentence story in shuffled order and the title, put the story in the correct order. Given an analogy that relates items to whether they are trash or treasure, determine if the given item is trash or treasure. Given a five sentence story and the title, determine which two sentences must be swapped so that the story makes complete sense. Given a tuple, determine whether the Head includes an event or an action in the Tail or not. Given the title and the first four sentences of a five sentence story, choose the correct story ending.
task1210 atomic classification madeupof task217 rocstories ordering answer generation task1155 bard analogical reasoning trash or treasure task218 rocstories swap order answer generation task1211 atomic classification hassubevent task213 rocstories correct ending classification
25