Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders Yi Jing* Zao Dai* Jinwu Hu Zijun Yao Lei Hou Juanzi Li Xiaozhi Wang Tsinghua University [email protected] [email protected]
Abstract
Question
arXiv:2605.27354v1 [cs.LG] 26 May 2026
How can internals guide post-training data engineering?
Model internals encode rich information about how a large language model (LLM) processes its training data; however, post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals. We propose S AE RL, a data engineering framework for LLM reinforcement learning (RL). It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool. Each property grounds a concrete data engineering operation: SAE-space clustering with moderate batch mixing for batch diversity control, a difficulty proxy for easy-to-hard curriculum ordering, and a quality probe for data filtering. S AE RL improves average accuracy by 3.00% over vanilla GRPO and reaches target accuracy with 20% fewer training steps on Qwen2.5-Math-1.5B, with consistent gains across model scales and RL algorithms. Experiments show that SAE transfers effectively across model families and scales, serving as a lightweight and reusable data engineering tool. These results demonstrate that model internals are a powerful and practical source of signals for post-training data engineering.
1
1) Why are external signals limited? Verifier Outcomes Rollout Pass Rates
Costly
Difficulty Labels
Miss Model-intrinsic Structure
2) What intrinsic properties do internals reveal? SAE Space SAE Data Sample Encoder √
Diversity
√ √ √
Difficulty
√
𝐸𝑆𝐴𝐸 √
√ √ √ √
√ √ √ √
√ √ √ √
Quality √ High
Low
3) How do internals improve data engineering? SAE
Diversity Curriculum Learning
Difficulty
Quality
Data Selection
Figure 1: Conceptual overview of S AE RL. Sparse Autoencoder (SAE) activations characterize three intrinsic data properties (diversity, difficulty, and quality), enabling SAE-based curriculum learning and data selection for LLM post-training.
Introduction
ing human preferences (Ouyang et al., 2022; Lambert et al., 2024), verifier outcomes (DeepSeek-AI et al., 2025; Shao et al., 2024; Yu et al., 2025), rollout pass rates (Sun et al., 2025; Xu et al., 2025; Zheng et al., 2025), and difficulty signals (Narvekar et al., 2020; Shi et al., 2025; Gao et al., 2025; Zhao et al., 2025). These signals have proven useful for data selection and curriculum learning.
Post-training, especially reinforcement learning, has become central to advancing the capabilities of large language models (OpenAI, 2026; Anthropic, 2026; Zeng et al., 2026; DeepSeek-AI, 2026). Its effectiveness depends heavily on data engineering: which samples are used, how to sort the samples, and batching strategies. These choices shape the training signal at every step, making data engineering an important factor for improving both training efficiency and final performance. Existing post-training data engineering pipelines typically rely on external feedback signals, includ-
However, external signals are often costly to obtain and to apply throughout training (Casper et al., 2023), leaving the rich data-feedback signals embedded in model internals largely underexplored. Recent work has shown that internal representations can guide data selection in pre-training (Sam
* Equal contribution.
1
et al., 2025; Rathi and Radford, 2026) and supervised fine-tuning (Ivison et al., 2025; Ma et al., 2025; Chen et al., 2026; Yang et al., 2025b), suggesting that model internals encode structure actionable for training. Whether they can play a similar role in post-training data engineering for reinforcement learning remains an open question. Mechanistic interpretability research (Meng et al., 2022; Wang et al., 2022; Somvanshi et al., 2026) continuously explores how to obtain and understand model internals. As a recent advance, Sparse Autoencoders (SAEs) decompose LLM hidden representations into sparse, fine-grained feature activations (Bricken et al., 2023; Gao et al., 2024; Templeton et al., 2024), providing fine-grained and disentangled perspectives of LLM internals. While recent pioneering work (Wang et al., 2025a) adopts LLM hidden representations in RL data selection, exploring the fine-grained feature space offered by SAE may lead to more holistic and precise modeling of data properties with model internals. Therefore, we study the method using SAE activations to capture three intrinsic properties of posttraining data: (1) Diversity: distances and clusters in the internal space can measure how broadly a batch covers distinct feature regions and reasoning patterns. (2) Difficulty: sparse activation patterns can reflect the actual demands that a problem imposes on the model, going beyond shallow features such as length or topic. (3) Quality: internal activations can help distinguish samples from the target distribution from noisy or off-distribution raw data. These three properties correspond to concrete data engineering operations: batching strategy, curriculum ordering, and data filtering. Based on these findings, we propose S AE RL, an intrinsic framework for RL post-training data engineering based on SAE activations. S AE RL uses SAE to model three data properties: quality, difficulty, and diversity. S AE RL then proceeds in three steps: (1) an SAE-based quality probe filters the data pool toward target-distribution samples; (2) samples are clustered in SAE space and sorted by calibrated difficulty within each cluster, forming local easy-to-hard trajectories; (3) batches are interleaved across clusters and moderately mixed by swapping a small tail portion between nearby batches, improving coverage while preserving within-batch coherence. Experiments on mathematical reasoning show that S AE RL improves performance and efficiency across model scales and RL algorithms. Ablation
studies show that batching strategy, curriculum ordering, and data filtering each contribute to the final results. These results suggest that S AE RL improves post-training data engineering by jointly modeling data diversity, sample difficulty, and data quality with SAEs. Our contributions are twofold: (1) We frame model internals as actionable signals for posttraining data engineering. (2) We propose S AE RL, which grounds SAE-based quality, difficulty, and diversity signals in concrete data engineering operations for efficient LLM post-training. We hope that this work can facilitate future research on intrinsic data engineering and actionable mechanistic interpretability (Orgad et al., 2026).
2
Motivating Finding
We conduct a preliminary study to examine whether SAE activations encode actionable signals for posttraining data engineering. We find that they capture three intrinsic data properties—diversity, difficulty, and quality—motivating the design of S AE RL. 2.1
SAE Can Predict Data Diversity
SAE representations encode diversity-relevant semantic information. Since data diversity corresponds to coverage over distinct topics and skills, we examine whether SAE activations capture such semantic variation by testing their ability to predict external topic labels. We use D EEP M ATH (He et al., 2025), a largescale mathematical reasoning dataset with annotated topic labels, in our pilot study. Given an SAE representation zi for a data sample, we train a linear probe to predict topic labels at three levels of granularity: t̂i = fT (zi ). (1) As shown in Table 1, SAE features substantially outperform the majority-class baseline across all granularities, including 82 leaf topics. This indicates that SAE activations encode topic-level semantic structure, making SAE space a reliable basis for measuring data coverage and diversity in post-training data engineering. 2.2
SAE Can Predict Data Difficulty
SAE representations encode difficulty-relevant information. Data difficulty is reflected in internal activation patterns—problem meanings, symbolic structure, and required skills—making SAE activations a natural interface for extracting difficulty 2
Reveal
SAE Representation Data Sample
Diversity
Difficulty
Quality
MiniBatchKMeans (K=10)
ElasticNet Regressor
SGD-trained Linear Classifier
Small Difficulty-labeled Subset
Metadata e.g. Length
Model Activations
SAE Encoder
Three Intrinsic Properties
Train
… weight
𝐸𝑆𝐴𝐸
High-quality features
Low-quality
Calibration Quality Score
SAE-guided Curriculum
𝐶𝐾
Difficulty Score
SAE-guided Filtering
Tail Swap Threshold / Top-k
… …
…
Clusters 𝐶𝑘 (𝑘 = 1, … , 𝐾) 𝐶1 𝐶2
Mean / Max Pooling SAE Representation …
Difficulty Score
Cluster ID
Token-level Sparse Activations Prompt Span Solution Span … …
Quality Score
…
High-quality Samples
Batches
Figure 2: Overview of S AE RL. Token-level SAE activations are pooled into a shared representation encoding diversity, difficulty, and quality. These three properties ground two data engineering operations: curriculum construction and data selection.
Target L2 topic L3 topic Leaf topic
Labels
Majority
SAE
Regime
Train
Test
9 36 82
31.8 17.2 7.5
54.6 37.7 26.6
In-domain OOD Adapted OOD
DM3k DM3k DM3k + DSR800
DMrem DSR10k DSR10k
Table 1: Linear probe accuracy (%) predicting D EEP M ATH topic labels from prompt-side SAE activations. Targets are external dataset metadata, providing a noncircular test of whether SAE representations encode semantic axes relevant to sample diversity.
are only partially captured by surface statistics such as length, step count, or TeX ratio. We use PRM800K (Lightman et al., 2024) as the validation setting, as its step-level process labels provide a reliable proxy for solution quality. We convert these labels into numeric scores (+1 → 1, 0 → 0.5, −1 → 0) and average them within each example to obtain a continuous samplelevel quality score. Given the SAE representation zi , we train a ridge regressor to predict this score:
(2)
As shown in Table 2, SAE features strongly predict in-distribution difficulty and retain a positive signal under distribution shift, indicating that SAE activations capture difficulty-relevant structure beyond shallow cues such as length or topic. This makes them a reliable basis for difficulty-aware curriculum construction. 2.3
0.749 0.135 0.286
Table 2: Difficulty prediction from SAE activations using ElasticNet. ρ denotes Spearman correlation; DM and DSR denote DeepMath and DeepScaleR.
signals. Given the SAE representation zi , we train an ElasticNet (Zou and Hastie, 2005) regressor to predict a continuous difficulty score: dˆi = fD (zi ).
ρ
q̂i = fQ (zi ).
(3)
As shown in Table 3, SAE features outperform both the mean baseline and a metadata-only baseline, improving test Pearson correlation from 0.2100 to 0.3715 over metadata features. This suggests that SAE activations capture quality-relevant structure beyond shallow cues, supporting their use for quality-aware data filtering.
SAE Can Predict Data Quality
SAE representations encode quality-relevant information. Data quality reflects whether a training example is reliable, well-formed, and aligned with the target reasoning distribution. These properties 3
Feature
RMSE ↓
MAE ↓
Pearson ↑
Mean Metadata SAE
0.2161 0.2113 0.2007
0.1772 0.1718 0.1608
– 0.2100 0.3715
Moderate batch mixing. Each batch is paired with a partner batch drawn from a nearby curriculum stage, matched by similar average difficulty and sequence length but required to have a different dominant cluster, with a small tail portion exchanged between the two batches.
Table 3: Quality prediction on PRM800K. SAE features outperform metadata features, indicating qualityrelevant signal beyond surface statistics.
3
3.3
We model sample difficulty from SAE representations and use it to construct a cluster-first easy-tohard curriculum.
Methodology
Based on the motivating findings above, we propose S AE RL, an offline data engineering framework for reinforcement learning post-training that uses SAE to model three intrinsic data properties— diversity, difficulty, and quality—and maps them to concrete operations: batching strategy, curriculum ordering, and data filtering. 3.1
Difficulty proxy and calibration. As described in Section 2.2, we train a lightweight ElasticNet regressor on a small difficulty-labeled subset (|L|=3k) to estimate sample difficulty, producing a raw score dˆi = fD (ϕ(xi )) for each sample. Since scores may vary in scale across clusters, we apply cluster-wise calibration using a global mapping with shrinkage-based cluster corrections: ri = Calibrate dˆi , ci , (5)
SAE Representation
SAEs decompose dense model activations into sparse, interpretable feature activations (Gao et al., 2024), providing a structured interface for extracting content-level signals from model internals. Given a sample xi , we extract token-level SAE activations separately from its prompt and solution spans, aggregating each via mean and max pooling to capture both sustained and localized activation patterns. The unified representation is ϕ(xi ) = zi , mi , (4)
where ci is the cluster assignment of xi and ri is the final ranking score used for curriculum ordering. Cluster-first curriculum. Within each cluster, samples are sorted by ri into fixed-size batches, forming local easy-to-hard trajectories. The global curriculum then interleaves batches across clusters stage by stage, with moderate batch mixing applied within each stage. 3.4
where zi concatenates the pooled SAE activations over both spans, and mi is a small set of shallow metadata features (e.g., length statistics, TeX ratio, digit ratio); the SAE part contains 960 features and mi contains 26. 3.2
Difficulty-driven Curriculum Ordering
Quality-driven Data Filtering
We model sample quality from SAE representations to filter noisy data before curriculum ordering. The probe formalizes this as binary classification: given a sample xi , it outputs the probability of belonging to the target distribution, si = pψ yi = 1 | ϕ(xi ) , (6)
Diversity-driven Batching Strategy
We model batch diversity by clustering samples in SAE space and applying moderate batch mixing. Empirically, we find that batch diversity in SAE space has a concave relationship with downstream performance: moderate cross-cluster mixing improves over pure-cluster batches, while excessive mixing hurts optimization (Section 5.2). Appendix A provides a bias–variance perspective analysis on this finding.
implemented as a SGD-trained linear classifier (Bottou, 2010) over SAE activations, trained on a subset of source-labeled samples. High-scoring samples are then selected by a fixed threshold Dτ = {xi : si ≥ τ } or top-k ranking Dk = TopKxi (si ), filtering the noisy data pool toward the target distribution and providing a higherquality data source for post-training.
Clustering. We cluster samples using SAE features and metadata via MiniBatchKMeans (Sculley, 2010) at K=10, capturing model-internal structure such as mathematical semantics, problem format, and skill patterns.
4
Main Experiment
We evaluate S AE RL in the mathematical reasoning domain, focusing on downstream performance, training efficiency, and noisy-data selection. 4
Experiment Setup
30.0 40.0 33.3 40.0 33.3 40.0 40.0
53.4 55.6 55.0 55.6 53.1 56.2 55.9
81.1 81.9 81.6 78.9 79.2 83.5 84.6
71.4 70.7 72.1 69.4 70.8 72.0 72.0
26.0 26.6 26.5 23.6 25.5 27.3 28.4
34.8 34.4 34.7 32.4 34.9 35.7 34.6
49.4 51.5 50.5 49.9 49.4 52.4 52.5
GRPO DIFF ADARFT GAINRL S AE RLG
46.6 50.0 53.3 53.3 53.3
68.1 68.7 63.1 68.4 68.4
90.3 90.9 87.5 90.1 91.5
79.1 79.1 76.1 79.8 80.3
33.6 32.9 31.8 34.6 35.4
42.0 42.6 38.6 41.7 43.0
59.9 60.7 58.4 61.3 61.9
Table 4: Accuracy (%) at step 900. S AE RLG and S AE RLD denote S AE RL trained with GRPO and DAPO, respectively. DIFF denotes Difficulty Curriculum Learning. Bold indicates the best result, while underline denotes the second best.
Evaluation. We instantiate S AE RL in the mathematical reasoning domain and evaluate on six benchmarks spanning a wide difficulty range: GSM8K (Cobbe et al., 2021) and AMC23 (lower), MATH500 (Lightman et al., 2024) and MinervaMath (Lewkowycz et al., 2022) (mid), and OlympiadBench (He et al., 2024) and AIME24 (competition-level), which are referred to as GSM8K, AMC, MATH, MNV, OLPD, and AIME, respectively. We report Pass@8 for AIME24 and Avg@8 for the remaining five benchmarks.
AIME AMC GSM8K MATH MNV OLPD Avg
Qwen2.5 Math-1.5B
Model Method GRPO DAPO DIFF ADARFT GAINRL S AE RLG S AE RLD
40 20 60 40 40 20 20
680 580 340 540 760 580 100
540 320 440 900 780 400 240
560 400 440 820 480 380 340
560 260 540 860 600 520 220
440 400 420 900 480 380 320
470 330 373 676 523 380 206
GRPO DIFF ADARFT GAINRL S AE RLG
40 20 20 80 40
320 140 740 240 120
240 160 360 220 200
180 160 400 180 200
220 420 900 220 280
200 180 440 220 200
200 180 476 193 173
Table 5: Training steps required to reach the target accuracy on each benchmark. For each model–benchmark pair, the target accuracy is set to the minimum final accuracy among all compared methods in Table 4, ensuring that every method can reach it by step 900. Lower values indicate higher training efficiency. Method notation follows Table 4. Bold indicates the fewest steps, while underline denotes the second fewest.
Baselines. We compare S AE RL against five baselines. Vanilla GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025) serve as RL algorithm baselines without curriculum, and we pair S AE RL with both to test whether its benefits are consistent across RL algorithms. Difficulty Curriculum Learning (Narvekar et al., 2020) uses externally provided difficulty labels, testing whether SAE-based signals add value beyond human annotations. ADARFT (Shi et al., 2025) estimates difficulty from rollout accuracy, representing rolloutbased curriculum methods. GAINRL (Wang et al., 2025a) selects data via compressed hiddenstate representations, serving as the closest internalsignal baseline to directly test whether sparse SAE features outperform dense alternatives. 4.2
Qwen2.5 Math-1.5B
Models and training. We train two model scales, Qwen2.5-Math-1.5B and Qwen2.5-Math7B (Yang et al., 2024), on DeepMath-103K (He et al., 2025) with a batch size of 128 to test the generality of S AE RL. We denote S AE RL trained with GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025) as S AE RLG and S AE RLD , respectively. We train an SAE on layer-27 activations of Qwen3-1.7B (Yang et al., 2025a) as the shared encoder for all data engineering operations, demonstrating that a single SAE trained on one model can effectively guide post-training data engineering for other model families and larger scales. Additional details are provided in Appendix C.2.
AIME AMC GSM8K MATH MNV OLPD Avg
GRPO DAPO DIFF ADARFT GAINRL S AE RLG S AE RLD
Qwen2.5 Math-7B
Model Method
Qwen2.5 Math-7B
4.1
ADARFT, and GAINRL, S AE RL obtains stronger overall performance, indicating that sparse SAE activations provide a more useful signal than external difficulty labels, rollout accuracy, or compressed hidden states. At the 7B scale, S AE RLG again achieves the best average result among the compared methods, suggesting that a shared SAE trained on a smaller model can still guide data engineering for larger models.
Training Performance 4.3
Table 4 shows that S AE RL improves average accuracy across RL algorithms, baselines, and model scales. At the 1.5B scale, S AE RL improves both GRPO and DAPO, showing that the SAE-based curriculum is not specific to a particular RL algorithm. Compared with Difficulty Curriculum Learning,
Training Efficiency
S AE RL improves training efficiency by reducing both training steps and preparation cost. Table 5 evaluates convergence speed by measuring how many training steps each method needs to reach a shared target accuracy. At the 1.5B 5
Rule
Kept
DM
F ULL 210,043 103,022 P95-T 103,121 98,342 P99-T 87,664 86,767 T OP -50 K 50,000 49,962 T OP -90 K 90,000 88,855
Purity (%) Recall (%) 49.05 95.37 98.98 99.92 98.73
100.00 95.46 84.22 48.50 86.25
AIME AMC GSM8K MATH MNV OLPD Avg
S AE RL − Diff − Diff & Mix − Clus & Mix
40.0 33.3 33.3 36.6
56.2 52.1 55.3 55.0
83.5 81.6 81.2 82.2
72.0 71.3 71.1 71.3
27.3 25.0 24.8 25.3
35.7 35.0 35.0 34.5
52.4 49.7 50.1 50.8
Table 7: Ablation results at step 900 on Qwen2.5-Math-1.5B, reported in accuracy (%). The first row denotes the full S AE RL. Rows prefixed with “−” remove the corresponding component(s), where Diff, Mix, and Clus denote difficulty sorting, moderate batch mixing, and cluster-first grouping, respectively. Bold indicates the best result.
Table 6: SAE-probe-based DeepMath-like sample selection from the mixed raw pool. DM denotes DeepMath samples; P95-T and P99-T denote percentile-threshold selection rules.
scale, S AE RL accelerates both GRPO and DAPO. S AE RLD gives the fastest average convergence, and S AE RLG requires fewer average steps than GRPO, ADARFT, and GAINRL. At the 7B scale, S AE RLG reaches the target in the fewest average steps. These results show that SAE-guided data engineering improves convergence across different model scales and RL algorithms. S AE RL also demonstrates efficiency gains. The Difficulty baseline and ADARFT achieve comparable convergence speed but require LLM-generated labels or multiple rollouts per problem at substantial cost—ADARFT takes approximately 17.33 H100 GPU hours with a reduced rollout budget (Appendix C.3). In contrast, S AE RL trains the difficulty proxy from a small labeled subset of 3,000 samples, and SAE encoding for the full dataset of 103,022 samples takes about 0.5 H100 GPU hours. Thus, S AE RL obtains its convergence gains with substantially lower preprocessing overhead. 4.4
Method
split, indicating that D EEP M ATH-like high-quality samples are highly separable in the SAE activation space. As shown in Table 6, after applying the fixed probe to Draw , the p95 threshold retains 103,121 samples, with 95.37% D EEP M ATH purity and 95.46% recall. Direct top-50k selection by the probe score further improves the D EEP M ATH purity to 99.92%. These results suggest that the SAEbased probe captures fine-grained D EEP M ATH-like activation signatures, enabling stable high-quality data selection from noisy data.
5
Analysis
We analyze the sources of S AE RL’s gains across four dimensions: component contribution, batch diversity control, robustness, and interpretability. 5.1
Ablation Study
S AE RL relies on the joint effect of batching strategy, curriculum ordering, data filtering. Difficulty sorting defines the easy-to-hard trajectory, clusterfirst grouping preserves local coherence in SAE activation space, and moderate batch mixing adds limited cross-cluster coverage without disrupting the trajectory. Table 7 shows that removing difficulty sorting causes the largest degradation, confirming that the easy-to-hard trajectory is central to S AE RL. The w/o Clus & Mix variant removes cluster assignments and therefore cannot perform moderate batch mixing, leaving a difficulty-only curriculum. Its drop indicates that difficulty sorting alone is insufficient, and SAE-space grouping provides useful local coherence. Comparing w/o Diff with w/o Diff & Mix shows that mixing without difficulty sorting does not improve the curriculum and can even weaken it. In contrast, the full S AE RL outperforms the variants that remove either difficulty sorting or
Noisy Data Selection
We further evaluate whether SAE activations support the selection of high-quality samples from a target distribution within a larger mixed noisy pool. We use D EEP M ATH as the target distribution: it is constructed from N UMINA M ATH (Li et al., 2024b) and other open mathematical sources through decontamination, difficulty filtering, and answer-verifiability filtering (He et al., 2025), making recovery from its source family a meaningful test of quality discrimination. We formulate the task as follows. The raw pool Draw consists of 103,022 D EEP M ATH samples mixed with 107,021 samples from the source corpus N UMINA M ATH 1.5, giving |Draw | = 210,043. The probe is trained to recover the D EEP M ATH subset using only d = 960 SAE features obtained by mean/max pooling over prompt and solution tokens. The SAE-only source/style probe achieves 0.9911 ROC-AUC and 0.9910 AP on the holdout 6
450 (b)
8
46.0
4
45.5
16
45.0 44.5
32
0 8.5
Steps to 43% ↓
Mean@8 (%)
46.5 (a)
330
0
4
10.5
16
270
8.5
KNN diversity ×10−4
9.5
KNN diversity ×10−4
Figure 3: SAE-space batch diversity versus downstream reinforcement learning performance. (a) Average mean@8 at step 800 as a function of the mean in-batch k-NN distance in SAE space, with k = 5. (b) Number of training steps required to reach the fixed average mean@8 threshold τ = 43.0%. Moderate diversity, represented by mix8, achieves the best step-800 performance and the fastest threshold crossing.
81.1 83.5
71.4 72.0
26.0 27.3
34.8 35.7
51.8 56.5
80.3 82.1
69.9 71.1
24.5 25.4
33.8 34.9
128
GRPO 30.0 S AE RLG 40.0
85.0 87.5
94.2 94.6
86.2 86.8
43.3 43.0
52.7 54.8
512
GRPO 36.6 S AE RLG 33.3
85.0 85.0
93.4 93.7
84.6 85.8
40.4 41.1
52.4 53.3
Table 8: Accuracy (%) across batch sizes and evaluation metrics. Results are evaluated on Qwen2.5-Math-1.5B at step 900 for B = 128 and step 300 for B = 512, where B denotes the training batch size. S AE RLG denotes S AE RL trained with GRPO. Bold indicates the best result.
grades both final accuracy and convergence speed— despite mix32 achieving the highest measured diversity. This suggests that beyond a moderate level, cross-cluster mixing disrupts within-batch gradient coherence more than it reduces cluster-local bias. This pattern is consistent with the bias–variance decomposition in Appendix A, which shows that the mixing utility is a concave function of mixing strength with a unique interior optimum. The practical takeaway is that effective batch construction requires balancing two competing objectives: preserving local SAE-space coherence to stabilize optimization, while introducing limited cross-cluster coverage to reduce directional bias.
cluster-based batch construction, indicating that moderate batch mixing is most effective when it is applied on top of an already structured cluster-first, easy-to-hard curriculum. 5.2
53.4 56.2
Pass @8
10.5
AIME AMC GSM8K MATH MNV OLPD
GRPO 13.7 128 Avg S AE RLG 14.1 @8 GRPO 12.5 512 S AE RLG 13.7
8
210 9.5
Metric B Method
32
390
Batch Diversity Analysis
The cluster-first curriculum introduces moderate cross-cluster batch mixing to balance within-batch gradient coherence and cross-cluster coverage. The mixing strength, controlled by the number of tail samples swapped between batches, directly governs this trade-off. To verify that moderate mixing is indeed optimal and to characterize how sensitivity to mixing strength affects downstream performance, we compare five curriculum variants that differ only in this parameter:
5.3
Batch Size Analysis
Table 8 shows that S AE RL remains effective across batch sizes. Under Avg@8, S AE RL outperforms GRPO at both B = 128 and B = 512, indicating that the curriculum remains effective beyond the default training batch size. Under Pass@8, increasing the batch size narrows the gap between the two methods. This suggests that larger batches may dilute the structural benefit of an ordered learning trajectory.
M = {mix0, mix4, mix8, mix16, mix32}, where mix0 is the cluster-first curriculum with no mixing, and larger indices correspond to stronger cross-cluster mixing. All other components of S AERL are held fixed. We quantify batch diversity by the mean inbatch k-NN distance (k=5) computed in the twodimensional SAE projection space, and measure downstream performance by the average mean@8 across the six evaluation benchmarks used in the main experiments. Figure 3 reports both peak performance at step 800 and the number of steps required to reach a fixed threshold τ = 43.0%. The results reveal a clear non-monotonic relationship. Performance improves steadily from mix0 to mix8, and mix8 reaches τ in the fewest training steps. Beyond this point, further increasing the mixing strength to mix16 and mix32 de-
5.4
Interpretability Analysis
Beyond downstream performance, we examine whether S AE RL exposes interpretable structure at the cluster and feature levels during curriculum construction. Additional details are provided in Appendix D. Cluster-level structure. Comparing C with D EEP M ATH topic metadata T yields low alignment (purity = 0.1095, NMI = 0.0881), indicating that SAE clusters do not reproduce the human7
defined topic taxonomy. Rather, inspection of cluster statistics and representative examples reveals that clusters capture curriculum-relevant properties including problem format, reasoning structure, solution profile, and difficulty. Several clusters also show enrichment for recognizable mathematical areas such as limits, combinatorics, group theory, and integration. SAE clusters thus characterize the data along axes more relevant to curriculum construction than external topic labels.
(Ye et al., 2025; Li et al., 2025b; Chen et al., 2025; Shi et al., 2025; Sun et al., 2025; Zhao et al., 2025; Gao et al., 2025), and then to (3) optimization- and resource-aware selection using gradients, influence, rollout utility, or distribution schedules (Li et al., 2025a; Yang et al., 2026; Zhu et al., 2025; Wang et al., 2025a; Xu et al., 2025; Zheng et al., 2025; Wang et al., 2025b; Rajaraman et al., 2026). However, they still rely mainly on external or scalar signals; S AE RL instead grounds data engineering in model-internal structure.
Feature-level signals. The difficulty proxy relies primarily on SAE activations rather than shallow metadata. Among the top-20 features ranked by LightGBM (Ke et al., 2017) gain, 19 are SAEderived and only 1 is a metadata feature; among the top-100, only 3 are metadata features. Within the SAE features, solution-side mean activations dominate, consistent with sustained solution-side patterns providing the strongest correlational signal for difficulty. Prompt-side max activations also contribute, reflecting localized cues in the problem statement such as symbolic structure or diagrammatic format. A high-activation audit in Appendix D further shows that individual high-gain features exhibit recurring semantic tendencies spanning abstract algebra, advanced analysis, geometry, combinatorics, and number-theoretic reasoning. Taken together, these analyses show that S AE RL provides not only an effective curriculum ordering, but also an auditable decision pathway. Each sample can be inspected through its activation group, difficulty-related feature signals, and position within the curriculum.
6
6.2
Model internals have moved from post-hoc analysis toward training-time feedback. Existing work uses (1) logit- or loss-based signals for data filtering and token or instruction selection (Li et al., 2024c; Lin et al., 2024), (2) gradients or influence estimates for data selection and weighting (Xia et al., 2024; Li et al., 2025a; Yang et al., 2026; Zhu et al., 2025), and (3) hidden states or activations for efficient example selection or representation-level intervention (Wang et al., 2025a; Wu et al., 2024). These approaches show that internals can guide training, but they often reduce internal structure to coarse signals. S AE RL instead uses sparse autoencoder features (Bricken et al., 2023; Templeton et al., 2024; Gao et al., 2024), which provide sparse and fine-grained activation signals for data engineering, extending beyond prior SAE-based uses for tuning-data diversity (Yang et al., 2025b) and preference modeling (Liu et al., 2025) to the RLVR post-training setting.
7
Related Work
We discuss two trends behind our work: posttraining data engineering is becoming more adaptive, and model internals are increasingly used as training signals. 6.1
Model Internals for LLM Training
Conclusion
We propose S AE RL, a post-training data engineering framework grounded in model-internal sparse representations. S AE RL uses SAE activations as a shared representation space to model three intrinsic data properties—diversity, difficulty, and quality— and grounds each in a concrete data engineering operation: batching strategy, curriculum ordering, and data filtering. Experiments on mathematical reasoning demonstrate consistent gains in accuracy and convergence efficiency across model scales and RL algorithms, with a single SAE transferring effectively across model families. These results suggest that model internals are a powerful and practical source of signals for post-training data engineering, opening a direction complementary to external feedback-based approaches.
Post-training Data Engineering
Post-training data engineering controls which examples are used, when they appear, and how training budget is allocated, which strongly shapes final model behavior (Ouyang et al., 2022; Zhou et al., 2023; Lambert et al., 2024; DeepSeek-AI et al., 2025). Existing methods have evolved from (1) task-, goal-, or replay-based curricula (Narvekar et al., 2020; Li et al., 2024a; Tzannetos et al., 2024), to (2) quality- and capability-aware selection of compact high-value data or boundary-level prompts 8
Limitations
Intended Use. S AE RL is intended as a research framework for post-training data engineering, including curriculum construction, batch organization, data filtering, and interpretability-oriented analysis. It is not intended for developing harmful models, evading safety mechanisms, or optimizing data for malicious capabilities.
Domain scope. Our empirical validation focuses on mathematical reasoning with verifiable rewards. This setting provides a controlled testbed for studying curriculum construction, since both training feedback and evaluation outcomes can be measured reliably. However, the extent to which the same SAE-space structure transfers to other post-training settings remains to be established, including code-centric RL, agentic RL, tooluse and multi-step decision-making, and general instruction-following.
Documentation of Artifacts. We will document the released artifacts with sufficient detail to support reproducibility, including the data-processing pipeline, SAE feature extraction, curriculum construction, data filtering, and evaluation setup.
Limited supervision. Although SAERL reduces the need for large-scale labeling or rollout-based scoring, it is not fully unsupervised. The difficulty proxy uses a small difficulty-labeled subset, and the quality probe relies on source or distribution labels as supervision. Future work may explore weaker forms of supervision, self-calibrated scoring, or fully unsupervised criteria for constructing SAEguided curricula.
AI Assistants in Research or Writing. We use AI assistants for code development assistance and language polishing. All AI-assisted content is reviewed and edited by the authors, who remain responsible for the final scientific claims, experiments, and writing.
Theoretical scope. Our analysis treats proximity in SAE space as a proxy for semantic similarity and gradient coherence. This yields an optimization-level interpretation of the observed coherence–coverage trade-off, but it does not prove a causal relationship between SAE distance and training dynamics. Establishing such a guarantee would require direct gradient-level measurements on the representation space.
Anthropic. 2026. Claude Opus 4.6. https:// www.anthropic.com/transparency. Model transparency report, released February 2026.
Ethical Considerations
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. Preprint, arXiv:2307.15217.
References
Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT 2010, pages 177–186. Physica-Verlag HD. Trenton Bricken et al. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread.
This section discusses the ethical considerations and broader impact of this work. Potential Risks. S AE RL uses model-internal SAE representations for post-training data engineering. Although intended to improve efficiency and inspectability, such signals could be misused to optimize data for unsafe behaviors. We restrict our experiments to mathematical reasoning with verifiable rewards and recommend safety filtering and human oversight for broader applications.
Xin Chen, Junchao Wu, Shu Yang, Runzhe Zhan, Zeyu Wu, Min Yang, Shujian Huang, Lidia S. Chao, and Derek F. Wong. 2026. Neuron-aware data selection in instruction tuning for large language models. Preprint, arXiv:2603.13201.
Intellectual Property. The models, datasets, benchmarks, and software frameworks used in this work are publicly available research artifacts and are used in accordance with their respective licenses and terms of use. Any released code or processed artifacts will follow the corresponding license requirements.
Xinjie Chen, Minpeng Liao, Guoxin Chen, Chengxi Li, Biao Fu, Kai Fan, and Xinggao Liu. 2025. From data-centric to sample-centric: Enhancing LLM
9
reasoning via progressive optimization. Preprint, arXiv:2507.06573.
Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. 2024. Tulu 3: Pushing frontiers in open language model post-training. Preprint, arXiv:2411.15124.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. Preprint, arXiv:2206.14858.
DeepSeek-AI. 2026. Deepseek-v4: Towards highly efficient million-token context intelligence. https://huggingface.co/deepseek-ai/ DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf. Technical report. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, et al. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Preprint, arXiv:2501.12948.
Hu Li, Xuezhong Qian, and Wei Song. 2024a. Prioritized experience replay based on dynamics priority. Scientific Reports, 14(6014). Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024b. NuminaMath 1.5. https://huggingface.co/datasets/AI-MO/ NuminaMath-1.5.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. Preprint, arXiv:2406.04093. Zhaolin Gao, Joongwon Kim, Wen Sun, Thorsten Joachims, Sid Wang, Richard Yuanzhe Pang, and Liang Tan. 2025. Prompt curriculum learning for efficient LLM post-training. Preprint, arXiv:2510.01135.
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. 2024c. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14255–14273. Association for Computational Linguistics.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850. Association for Computational Linguistics.
Shipeng Li, Zhiqin Yang, Shikun Li, Xiaobo Xia, Hengyu Liu, Xinghua Zhang, Gaode Chen, Dong Fang, Ying Tai, and Zhe Peng. 2025a. LearnAlign: Data selection for LLM reinforcement learning with improved gradient alignment. Preprint, arXiv:2506.11480.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. DeepMath-103K: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. Preprint, arXiv:2504.11456.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025b. LIMR: Less is more for RL scaling. Preprint, arXiv:2502.11886. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In International Conference on Learning Representations.
Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. 2025. Large-scale data selection for instruction tuning. Preprint, arXiv:2503.01807.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. 2024. Rho1: Not all tokens are what you need. Preprint, arXiv:2404.07965.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, volume 30, pages 3146–3154.
Dengcan Liu, Jiahao Li, Zheren Fu, Yi Tu, Jiajun Li, Zhendong Mao, and Yongdong Zhang. 2025. SparseRM: A lightweight preference modeling with sparse autoencoder. Preprint, arXiv:2511.07896.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri,
10
Da Ma, Gonghu Shang, Zhi Chen, Libo Qin, Yijie Luo, Lei Pan, Shuai Fan, Lu Chen, and Kai Yu. 2025. Task-specific data selection for instruction tuning via monosemantic neuronal activations. Preprint, arXiv:2503.15573.
DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300. Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. 2025. Efficient reinforcement finetuning via adaptive curriculum learning. Preprint, arXiv:2504.05520.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. Preprint, arXiv:2202.05262.
Shriyank Somvanshi, Md Monzurul Islam, Amir Rafe, Anannya Ghosh Tusti, Arka Chakraborty, Anika Baitullah, Tausif Islam Chowdhury, Nawaf Alnawmasi, Anandi Dutta, and Subasish Das. 2026. Bridging the black box: a survey on mechanistic interpretability in ai. ACM Computing Surveys, 58(8):1–35.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. 2020. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1–50.
Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. 2025. Improving data efficiency for LLM reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay. Preprint, arXiv:2506.05316.
OpenAI. 2026. GPT-5.5 System Card. https:// deploymentsafety.openai.com/gpt-5-5. Published April 23, 2026. Hadas Orgad, Fazl Barez, Tal Haklay, Isabelle Lee, Marius Mosbach, Anja Reusch, Naomi Saphra, Byron Wallace, Sarah Wiegreffe, Eric Wong, et al. 2026. Interpretability can be actionable. arXiv preprint arXiv:2605.11161.
Adly Templeton et al. 2024. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Preprint, arXiv:2203.02155.
Georgios Tzannetos, Parameswaran Kamalaruban, and Adish Singla. 2024. Proximal curriculum with task correlations for deep reinforcement learning. Preprint, arXiv:2405.02481. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: A circuit for indirect object identification in GPT-2 small. Preprint, arXiv:2211.00593.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849.
Qinsi Wang, Jinghan Ke, Hancheng Ye, Yueqian Lin, Yuzhe Fu, Jianyi Zhang, Kurt Keutzer, Chenfeng Xu, and Yiran Chen. 2025a. Angles don’t lie: Unlocking training-efficient RL through the model’s own signals. Preprint, arXiv:2506.02281.
Nived Rajaraman, Audrey Huang, Miro Dudik, Robert Schapire, Dylan J. Foster, and Akshay Krishnamurthy. 2026. Learning to reason with curriculum I: Provable benefits of autocurriculum. Preprint, arXiv:2603.18325.
Zhenting Wang, Guofeng Cui, Yu-Jhe Li, Kun Wan, and Wentian Zhao. 2025b. DUMP: Automated distribution-level curriculum learning for RL-based LLM post-training. Preprint, arXiv:2504.09710.
Neil Rathi and Alec Radford. 2026. Shaping capabilities with token-level data filtering. Preprint, arXiv:2601.21571.
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2024. ReFT: Representation finetuning for language models. Preprint, arXiv:2404.03592.
Dylan Sam, Ayan Chakrabarti, Afshin Rostamizadeh, Srikumar Ramalingam, Gui Citovsky, and Sanjiv Kumar. 2025. Analyzing similarity metrics for data selection for language model pretraining. Preprint, arXiv:2502.02494.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. LESS: Selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 54104–54132. PMLR.
D. Sculley. 2010. Web-scale k-means clustering. In Proceedings of the 19th International Conference on World Wide Web, pages 1177–1178. Association for Computing Machinery.
Yixuan Even Xu, Yash Savani, Fei Fang, and J. Zico Kolter. 2025. Not all rollouts are useful: Downsampling rollouts in LLM reinforcement learning. Preprint, arXiv:2504.13818.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024.
11
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
Minlie Huang, and Hongning Wang. 2025. Dataefficient RLVR via off-policy influence guidance. Preprint, arXiv:2510.26491. Hui Zou and Trevor Hastie. 2005. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301–320.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement. Preprint, arXiv:2409.12122. Ningyuan Yang, Weihua Du, Weiwei Sun, Sean Welleck, and Yiming Yang. 2026. GradAlign: Gradientaligned data selection for LLM reinforcement learning. Preprint, arXiv:2602.21492. Xianjun Yang, Shaoliang Nie, Lijuan Liu, Suchin Gururangan, Ujjwal Karn, Rui Hou, Madian Khabsa, and Yuning Mao. 2025b. Diversity-driven data selection for language model tuning through sparse autoencoder. Preprint, arXiv:2502.14050. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. LIMO: Less is more for reasoning. Preprint, arXiv:2502.03387. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. 2025. DAPO: An open-source LLM reinforcement learning system at scale. Preprint, arXiv:2503.14476. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. 2026. Glm5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763. Yang Zhao, Kai Xiong, Xiao Ding, et al. 2025. UFORL: Uncertainty-focused optimization for efficient reinforcement learning data selection. Preprint, arXiv:2505.12457. Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. 2025. Act only when it pays: Efficient reinforcement learning for LLM reasoning via selective rollouts. Preprint, arXiv:2506.02177. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less is more for alignment. Preprint, arXiv:2305.11206. Erle Zhu, Dazhi Jiang, Yuan Wang, Xujun Li, Jiale Cheng, Yuxian Gu, Yilin Niu, Aohan Zeng, Jie Tang,
12
A
SAE feature set, and let at ∈ R|F | be the SAE activation vector at token t. For each span, we summarize token-level SAE activations using mean pooling and max pooling. The SAE representation zi is obtained by concatenating the mean-pooled and max-pooled activations from both the prompt and solution spans: max s zi = āSip ; amax (9) S p ; āSi ; aSis .
A Bias–Variance View of Moderate Batch Mixing
We provide a bias–variance perspective on why moderate cross-cluster batch mixing can improve optimization. Setup. For each sample xi , let gi = ∇θ ℓ(xi ; θ) denote its per-sample gradient. Since SAE activations zi approximate the model’s internal representation of xi , we assume gi = G(zi ) + εi for a locally Lipschitz G : Rd → Rp , where εi captures SAE approximation error and residual nonlinear effects. Under this assumption, samples nearby in SAE space tend to produce similar gradients.
i
For clustering and difficulty estimation, we further append shallow metadata features mi , including length statistics, TeX ratio, and digit ratio, and use ϕi = [zi ; mi ] as the full feature vector. In our experiments, zi is 960-dimensional and mi is 26dimensional.
Pure-cluster bias. Consider a batch of size b drawn from cluster c, with gradient mean µc and covariance Σc , and letP Gt denote the target gradient. The MSE of Ĝc = 1b bi=1 gi decomposes as MSE0 = ∥Gt − µc ∥2 +
1 tr(Σc ), b
B.2
We cluster samples in the feature space using MiniBatchKMeans with K = 10. Each sample is assigned to the nearest cluster centroid based on its full feature vector ϕi . To estimate sample difficulty, we use a small difficulty-labeled subset L with |L| = 3000. For each labeled sample, the difficulty label is denoted by d⋆i . We train an ElasticNet difficulty proxy fD on the labeled subset and use it to produce the raw difficulty prediction dˆi = fD (ϕi ). Because only a limited number of difficulty labels are available, the raw proxy score is calibrated using a global calibration map fitted on the small labeled subset, together with a shrinkage-based cluster residual. Let g denote the global calibration map fitted on L. For each cluster, we compute the average residual between the labeled difficulty d⋆i and the globally calibrated prediction g(dˆi ) over labeled samples in that cluster. If a cluster has no labeled samples, its residual is set to zero. The residual is then scaled by a shrinkage weight λc = nc /(nc + τsh ), where nc is the number of labeled samples in cluster c and τsh > 0 controls the strength of shrinkage toward the global calibration. The final difficulty score is
(7)
where the two terms are cluster-local bias and estimation variance, respectively. Pure-cluster batches have low variance but may be biased when µc deviates from Gt . Effect of mixing. Suppose the mixed batch contains ⌊ρb⌋ samples from cluster d and (1−ρ)b from cluster c, with the two clusters uncorrelated. Let rc = Gt − µc and v = µd − µc . The net MSE change relative to equation 7 is ρ tr(Σd − Σc ), b (8) When v ̸= 0, ∆(ρ) is a convex quadratic in ρ with minimizer ρ† = clip(ρ∗ , 0, 1), where ρ∗ = A/2C, A = 2⟨rc , v⟩ − 1b tr(Σd − Σc ), and C = ∥v∥2 ; when v = 0, the optimum is attained at an endpoint. If ρ∗ ∈ (0, 1), the mixing utility U (ρ) = −∆(ρ) admits an interior maximum: too little mixing leaves cluster-local bias uncorrected, while too much weakens within-batch gradient coherence. ∆(ρ) = −2ρ⟨rc , v⟩ + ρ2 ∥v∥2 +
B
Technical Details of the S AE RL Pipeline
ri = g(dˆi ) + λci ∆ci .
This appendix provides additional technical details for the offline data-processing steps used in Section 3. B.1
Clustering and Difficulty Calibration
(10)
A larger ri corresponds to a higher estimated difficulty. B.3
SAE Sample Representation
Each sample xi is divided into a prompt span Sip and a solution spanSis . Let F denote the retained
Curriculum Ordering and Moderate Batch Mixing
Within each cluster, samples are sorted by the calibrated difficulty score ri in ascending order, re13
expansion factor of 64 to support fine-grained feature extraction. The training corpus consists of FineWeb-Edu (Penedo et al., 2024) and Wikipedia, totaling 80GB. Training was conducted on 4 A100 GPUs and completed in approximately 29 hours, giving a total cost of 116 A100 GPU hours.
sulting in a local easy-to-hard curriculum. The ordered samples in each cluster are then partitioned into fixed-size batches. The global curriculum interleaves these batches across clusters stage by stage, which preserves local easy-to-hard trajectories while maintaining coverage across different clusters. After the cluster-first curriculum has been constructed, we apply moderate batch mixing. For an ordered batch of size b, we keep the first b − h samples fixed and exchange only the last h tail samples with another batch. The partner batch is selected from a local curriculum window and must satisfy three conditions: it should have similar average calibrated difficulty, similar average sequence length, and a different dominant cluster. The dominant cluster of a batch is defined as the most frequent cluster label among its samples. Since only the tail block is exchanged, this operation introduces limited cross-cluster mixing while largely preserving the local curriculum structure within each batch. B.4
C.2
We maintain consistent core hyperparameters across both model scales (1.5B and 7B) to support a controlled comparison. Table 9 summarizes the detailed training configuration on verl. Hyperparameter Algorithm Learning Rate Train Batch Size Max Prompt Length Max Response Length Sampling Temperature Rollouts per Sample (N )
Raw data selection is formulated as a binary classification problem over SAE representations. Let Draw denote the raw candidate pool. For each candidate sample, yi = 1 indicates membership in the target DeepMath-like distribution, while yi = 0 indicates otherwise. The quality probe uses only the SAE representation zi , without metadata. We train a linear classifier with SGD and use its predicted probability as the selection score:
C.3
(11)
D
Implementation Detail
Interpretability Details of SAE-Guided Curriculum Construction
This appendix supplements the interpretability analysis in Section 5.4. We examine three types of evidence: the relation between SAE clusters and external topic annotations, the feature groups that provide predictive signal for difficulty estimation, and the intermediate variables exposed by the curriculum construction procedure.
This appendix provides the training hyperparameters and baseline implementation details used in the experiments. C.1
Baseline Implementations
For the Difficulty Curriculum Learning baseline (Narvekar et al., 2020), we sort the DeepMath-103K dataset (He et al., 2025) by its provided difficulty labels in ascending order and sample progressively throughout training. To ensure a fair comparison, we train the ADARFT baseline (Shi et al., 2025) using GRPO (Shao et al., 2024). ADARFT estimates problem difficulty from rollout accuracy, originally using Avg@128. Given the scale of the 103K problem dataset, we adapt this standard and use Avg@16 as the difficulty proxy for our ADARFT implementation, allowing us to simulate the ADARFT curriculum strategy within computational viability constraints.
where ψ = (w, b) and σ is the logistic sigmoid. Given the score si , we use either threshold-based selection or fixed-size top-k selection. The threshold rule selects samples with scores above a quality threshold γ, which controls selection precision. The top-k rule selects the k highest-scoring samples, which fixes the number of selected samples.
C
Value GRPO 1 × 10−6 128 1024 3072 0.6 8
Table 9: Default hyperparameters for training with the verl framework.
SAE-based Data Selection
si = pψ (yi = 1 | zi ) = σ(w⊤ zi + b),
Hyperparameters
Sparse Autoencoder Training Details
We train the SAE using the OpenSAE framework on layer-27 activations of Qwen3-1.7B, with an 14
Cluster Conservative label 0 1 2 3 4 5 6 7 8
9
Main evidence
Interpretation
Derivative-centered calculus Derivative applications; medium-high Local change, optimization, and derivativedifficulty; moderate length based transformations. Number-theoretic symbolic Congruences; long solutions; medium Congruence-style reasoning with symbolic reasoning difficulty manipulation and extended derivations. Abstract algebra and proof Group theory; high topic entropy; Algebraic structures and proof-oriented reastructure medium-high difficulty soning beyond a single subfield. Discrete and combinatorial Combinatorics; highest mean difficulty Counting, construction, and combinatorial reasoning proof patterns with high difficulty. Integral and continuous rea- Integral applications; long solutions; Integration-related problems with continusoning high difficulty ous reasoning and longer derivations. Broad calculus and transfor- Derivative-related topics; high entropy; A heterogeneous continuous-math group inmations long solutions volving functions and transformations. Limits and sequence-style Limits; highest top-topic share; lowest The most topic-concentrated cluster, cenreasoning entropy tered on limits and sequences. High-load limits and analy- Limits; higher difficulty than Cluster More complex and heterogeneous variants sis 6; long solutions of limit or analysis-style reasoning. Short-form elementary alge- Simple equations; lowest mean diffi- A mixed low-to-mid difficulty group with bra culty; shortest solutions; highest en- short algebraic structure. tropy Integration and symbolic Integration techniques; medium diffi- Procedural symbolic transformation, espeprocedures culty; moderate length cially integration-related reasoning.
Table 10: Cluster-level semantic audit of SAE activation groups on DeepMath. Summaries are generated by a GPT-5.4-based agent and manually reviewed by the authors. Labels are conservative summaries rather than one-to-one topic annotations.
Feature group
Top 20
Top 100
sol_mean prompt_max prompt_mean sol_max meta
10 5 2 2 1
36 26 24 11 3
descriptions of activation-space structure, rather than as one-to-one topic labels. This distinction is necessary because the external topic taxonomy and the SAE representation characterize different aspects of a sample. Topic labels describe the subject category assigned by the dataset, whereas SAE clusters are formed according to model-internal activation patterns. A cluster may therefore contain multiple mathematical topics while still preserving structure relevant to curriculum construction, such as similar solution profiles, reasoning formats, or difficulty ranges.
Table 11: Feature-group composition of the LightGBM difficulty proxy by gain.
D.1
Cluster-Level Structure
We compare the SAE cluster assignments on DeepMath with human-annotated topic metadata. The overall cluster–topic alignment is low: cluster-topic purity is 0.1095, NMI is 0.0881, and ARI is 0.0394. These results indicate that SAE clusters do not simply reproduce the human-defined mathematical topic taxonomy. We therefore further inspect each cluster using top leaf topics, topic entropy, difficulty distribution, solution length, and representative samples. Table 10 provides a summary of the resulting cluster structure. Some clusters exhibit clear topical enrichment, such as limits, combinatorics, group theory, and integration. At the same time, clusters also differ in problem format, solution length, symbolic density, proof orientation, and mean difficulty. These summaries should therefore be understood as coarse
D.2
Feature-Level Signals
We next examine which input feature groups provide predictive signal for difficulty estimation. The curriculum pipeline uses prompt-side and solutionside SAE features, together with a small set of metadata features. For feature-group analysis, we train an auxiliary LightGBM model on the same difficulty-labeled subset and rank input columns by gain. This model is used only for interpretability analysis; the actual curriculum ranking scores are still produced by the calibrated difficulty proxy described in Section 3. Table 11 reports the source distribution of the top-20 and top-100 gain-ranked features. The top-ranked features come primarily from SAE activations rather than metadata. Among 15
High-gain feature
Observed high-activation pattern
sol_mean/50831
Abstract algebra, rings, groups, and proof- Solution-side activation associated with abheavy problems stract algebraic structure and proof load.
sol_mean/28006
Measure, integration, and high-level analy- Solution-side pattern related to advanced analsis ysis and integration-style reasoning.
prompt_max/16071
Graphs, geometry, and Asymptote-style Prompt-side activation capturing localized diaprompts grammatic or formatting cues. Combinatorics, graphs, and discrete struc- Solution-side signal associated with discrete tures reasoning and combinatorial structure. Simple equations and algebraic word prob- Lower- to mid-difficulty algebraic patterns inlems volving short symbolic reasoning. Congruences, primes, and numeric expres- Solution-side number-theoretic and numeric sions reasoning patterns.
sol_mean/122476 sol_mean/60349 sol_mean/88548
Interpretation
Table 12: Semantic audit of individual high-gain features using high-activation samples. Summaries are produced by a GPT-5.4-based agent and manually reviewed by the authors.
the top-20 features, 19 are SAE-derived and 1 is metadata; among the top-100 features, 97 are SAEderived and 3 are metadata. Within these features, solution-side mean features form the largest group, which is consistent with sustained solution-side activations providing correlational signals for difficulty estimation. Prompt-side max features also appear frequently, indicating that localized strong activations in the problem statement provide additional predictive cues. To complement the feature-group statistics, we further inspect high-activation samples for several high-gain SAE features. Table 12 summarizes the semantic tendencies observed in these samples. These high-activation samples exhibit several recurring patterns, including abstract algebra, measure and integration, discrete combinatorics, geometric formats, elementary algebra, and numbertheoretic expressions. These descriptions indicate semantic tendencies only. SAE features are not strictly monosemantic: the same SAE feature may be activated in different contexts, and the same mathematical pattern may be distributed across multiple SAE features. D.3
Procedure-Level Inspectability
The final curriculum order is constructed from a sequence of explicit intermediate variables. For each sample, the pipeline records its SAE representation, metadata, cluster assignment, raw difficulty prediction, calibrated ranking score, batch assignment, and, when moderate mixing is applied, its tail-swap partner. These variables allow researchers to inspect how a sample moves from the representation space to its final position in the curriculum.
16