HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures Ziyun Qiao1,2,*,† 1
Yue Min1,*
Ruining Chen3
Yujun Li1,†
Wizard Quant, Beijing, China 2 Peking University, Beijing, China 3 University of Science and Technology of China, Hefei, China * Equal contribution. † Correspondence to: [email protected], [email protected] Abstract
arXiv:2607.02266v1 [cs.LG] 2 Jul 2026
Most data-mixing methods assume the corpus has already been partitioned into groups, and the choice of those groups determines what a mixer can express. Existing labels (provenance, topic or format taxonomies, flat embedding clusters) commit to one semantic axis at one granularity; changing the resolution rebuilds the labels. We argue the bottleneck is the label system, not the mixer, and provide a hierarchical one. HERMES is a data-derived labeling substrate: a Learned Semantic Transform followed by 3-stage residual vector quantization annotates each document once into a coarse-to-fine code whose prefix length controls granularity up to ∼130k cells. At coarse granularity HERMES sits at a plateau with KMeans-family methods on standard clustering metrics, so the contribution is the substrate, not the clusterer. On 1B-parameter, 25B-token pre-training, the hierarchy exposes an interaction fixed-granularity pipelines cannot test: at one prefix length, a combined Stage-2 rule contrast (equal-subbucket coverage vs. sizeproportional within-bucket quality top-30%) lifts a 16-task capability macro-average by +0.0253; at the next finer level, the same rule loses its measurable edge as candidate pools contract ∼5×. HERMES reframes data mixture design from choosing among fixed label sets to navigating a reusable, data-derived granularity hierarchy.
1
Figure 1: Three corpus-control paradigms. Top: fixed taxonomies (source, topic, format) scale but fix one semantic axis at one granularity. Bottom: per-sample selection (DSIR, MATES, LESS) reaches the document at per-document compute. Middle: HERMES exposes a data-derived hierarchy whose prefix reads (L1 , L12 , L123 ) deliver multiple granularities from one offline annotation, with no re-clustering.
granularity, and flat clusterings require recomputation to change K. We address this with HERMES (Hierarchical Embedding-based RVQ for Mixture Selection), a one-shot hierarchical residual vector quantizer that annotates each document with a coarse-to-fine code (c1 , c2 , c3 ) and exposes granularity as a prefix length on a single trained RVQ codebook stack (Figure 1). The detailed comparison to existing provenance, distilled-taxonomy, and embedding-cluster label families is deferred to Section 2.
Introduction
Data curation has become a decisive factor in LLM pre-training (Penedo et al., 2023, 2024), shifting attention from parameter scaling alone toward the mixing of heterogeneous sources. Pre-training data mixing has two separable layers: a label system that partitions the corpus, and a mixer or sampler that consumes those labels. What has constrained group-level pipelines is the label layer, not the mixer machinery: provenance is coarse, taxonomies commit to a single semantic axis and
At the K = 256–130k label scale, regressionstyle mixture optimizers (RegMix (Liu et al., 2025b), Data Mixing Laws (Ye et al., 2025)) sit outside the proxy budget, since their fitting bases scale at least linearly in cell count; we therefore instantiate only O(1)-proxy outer weights, Uniform and 1
DoReMi. Per-sample selectors that avoid labels pay per-document compute at web scale (Section 4.3). HERMES meets these constraints. A Learned Semantic Transform, in the spirit of optimized product quantization (Ge et al., 2013), feeds a 3-stage residual vector quantizer (Liu et al., 2015; Huijben et al., 2024); the prefix length on the resulting code defines granularity over a single trained RVQ codebook stack: L1 = c1 gives 256 buckets, L12 gives ∼65k, L123 gives ∼130k observed. HERMES does not aim to be a better clustering algorithm: at K = 256 it sits on a plateau with four standard 256-way clusterers on compactness/mass-balance metrics (Table 2), and is downstream-matched to KMeans within 0.0002 Avg. (Section 5.1). Its value is that one annotation supports an entire granularity sweep with no re-clustering between K = 256 and K = 130k. HERMES exposes two interactions invisible to fixed-granularity pipelines, on a 1B/25B-token training regime evaluated by a 16-task capability macro-mean (Avg.). First, at fixed L12 granularity under DoReMi-L1, switching the Stage2 sampler from max-entropy coverage (equal sub-bucket mass, all docs) to quality top-30% (size-proportional mass, within-bucket top-30%; corrected FineWeb-Edu reader) raises Avg. by +0.0253 (z +2.09) at fixed corpus, model, codebook, and outer weight (Section 5.2). Second, the same Stage-2 advantage does not persist at L123 : under the quota-preserving pair (per-L1 L123 local random coverage vs. L123 quality top-30%), the two rules sit at 0.3986 and 0.3988 (∆ = +0.0002, numerically tied under single-seed evaluation; Section 5.3). This collapse is consistent with candidate competition: the median candidate pool contracts 5.3× from L12 (2,271) to L123 (429), a regime in which within-sub-bucket ranking is no longer a stable proxy for global ranking over coverage. Together these show that granularity and the Stage-2 sampler are jointly determined on the substrate; a paired Uniform vs. DoReMi L1 characterisation is in Appendix L. Contributions. (i) HERMES, a one-shot hierarchical RVQ substrate that exposes granularity as a prefix length on a trained codebook stack, annotating ∼50M documents into up to ∼130k cells. (ii) A controlled Stage-2 sampler study at fixed codebook, corpus, model, budget, and granularity (L12 under DoReMi-L1): switching from max-entropy coverage to quality top-30% (corrected reader) yields +0.0253 Avg. (Finding A); the contrast moves two
axes jointly (sub-bucket mass and per-document eligibility) and is read as a combined Stage-2 rule effect. (iii) A candidate-competition diagnostic: holding DoReMi-L1 outer weights fixed and comparing the same two rules in their quota-preserving forms, the L12 advantage (+0.0253) collapses to +0.0002 at L123 (0.3988 vs. 0.3986 per-L1 local random coverage), consistent with median pool shrinkage from 2,271 to 429 documents (Finding B, Section 5.3).
2
Related Work
Corpus label systems. The label system used to partition a corpus is a design choice independent of the mixer that consumes it. Prior work falls into three families. Provenance labels use the source domain or web shard a document originated from (Xie et al., 2023a), easy to obtain but committed to a single coarse axis. Distilled taxonomies build humanor LLM-defined category sets at a fixed granularity: WebOrganizer (Wettig et al., 2025) releases topic and format taxonomies trained on LLM annotations, and Topic-over-Source (Peng et al., 2025) derives topic labels via clustering, LLM summarization, and classifier distillation, at a single fixed topical resolution. Embedding clusters are unsupervised and data-derived: KMeans variants, plain RVQ (Liu et al., 2015), and CLIMB’s ∼20 flat cells (Diao et al., 2025). These three families differ along prior structure, semantics, and granularity. Topic-over-Source shares the data-derived prior but yields a single fixed topic level via an LLM-in-theloop pipeline; HERMES exposes granularity as a prefix length on one quantizer pass, with no LLM in the loop. Mixers and samplers that consume labels. Given a label system, several outer-weight families decide how much sampling probability each cell receives. DoReMi (Xie et al., 2023a) runs group-DRO on a small reference model. Data Mixing Laws (Ye et al., 2025) and RegMix (Liu et al., 2025b) fit regression proxies whose bases scale linearly in cell count and are typically reported with ≤20 groups. CLIMB (Diao et al., 2025) couples clustering and weight search in a joint loop. Chameleon (Xie et al., 2025) computes leveragescore weights over a learned domain-affinity matrix. None of these works treat the label system as a separately tunable axis: label resolution is a property of the labeling step they presuppose, not a dial they sweep. HERMES is complementary: it 2
Hierarchical Codes ( )
Learned Semantic Transform (LST)
Residual Vector Quantization (RVQ)
Stage 1 (Coarse)
Input Data
Codebook
W
⋮
Stage 3 (Fine)
Stage 2 (Intermediate)
Codebook
Codebook
Transformed Embeddings
Reusable Semantic Buckets
select
select
...
...
select
Level 1 Prefix:
...
(Coarse)
... Prefix counts:
Level 2 Prefix:
Structured sematic tokens
Frozen encoder sample embeddings
(Intermediate)
...
LST Loss
Prefix counts:
Level 3 Leaf: (Fine)
Preserves semantic neighborhoods
Reconstruction:
Quantization friendliness
Hierarchical codes: Leaf counts:
Near-orthogonal transform ✓
▮▮ ▮▮ ▦ σ
Structure-preserving & quantization-friendly near-orthogonal transform
...
Prefix counts: number of items Leaf counts: number of leaves Density: data mass / bucket Quality prior: recon. error stats
Figure 2: HERMES annotation pipeline. A frozen encoder produces document embeddings (left); the Learned Semantic Transform rotates them into a quantization-friendly geometry under structure-preservation, reconstruction, and orthogonality losses (centre-left). Three cascaded RVQ stages then encode each rotated embedding into a hierarchical code (centre-right). Reading the code at different prefix lengths (right) exposes ∼256, ∼65k, and ∼130k buckets, reused across samplers without re-clustering. Sections 3.2–3.3.
annotator with K = 256 throughout,
provides the label substrate, and any of these outerweight families can in principle consume it at a chosen prefix length, subject to their proxy budget. In this paper we plug in two O(1)-proxy families compatible with K = 256–130k cells: Uniform and DoReMi. Per-sample selectors instead score documents individually (Section 4.3). Vector quantization machinery. HERMES borrows the rotate-then-quantize machinery of optimized product quantization (Ge et al., 2013), improved RVQ (Liu et al., 2015), and VQ-VAE (van den Oord et al., 2017), all of which optimize reconstruction. We reuse it for labeling: the code is consumed by samplers rather than a decoder, judged by the data-mixing interactions it exposes; see Chen et al. (2026) for the broader static-vsdynamic landscape.
3
Method
3.1
Notation and overview
LST
RVQ
ei −−−→ hi ∈ Rd −−−−→ (c1 , c2 , c3 ) ∈ [K]3 , a Learned Semantic Transform (LST) rotates the embedding into a quantization-friendly hi and a 3-stage Residual Vector Quantizer (RVQ) emits a hierarchical code. We write the bucket id at prefix length ℓ as bℓ (xi ) = (c1 , . . . , cℓ ), so b1 is an ancestor of b2 which is an ancestor of b3 . We refer to the three prefix granularities as L1, L12, and L123 and use prefix length as the granularity dial. Codes are produced once, so any difference across samplers cannot be attributed to a different grouping. 3.2
Learned Semantic Transform (LST)
LST is a single linear-plus-normalize layer hi = normalize(W ei +b) with W ∈ Rd×d initialized to the identity. It is trained jointly with the RVQ codebooks under three named losses (pairwise structure preservation, quantization-aware reconstruction, and orthogonality), following the rotate-thenquantize lineage of Ge et al. (2013); full equations and the per-stage commitment loss are in Appendix D. Optimizer and schedule are also in Appendix D.
HERMES is designed for the middle of the costresolution spectrum: one annotation per document, computed offline against frozen embeddings, that supports a prefix-length granularity dial up to ∼130k cells without re-clustering. Let D = {xi }N i=1 with each document mapped to a fixed embedding ei ∈ Rd by a frozen encoder (d = 1024, N ≈ 5 × 107 ); embeddings are computed once and never updated. HERMES is a two-stage offline
3.3
Residual Vector Quantization
Residual vector quantization (RVQ) is a workhorse for hierarchical discrete representations across audio coding (Zeghidour et al., 2022; Défossez 3
Table 1: Three granularities defined by code prefix. L123 is naturally sparse: only ∼0.77% of nominal triples are populated, as the stage-3 residuals concentrate on a thin manifold of the joint code space (Appendix B). Granularity
Code
L1 L12 L123
c1 (c1 , c2 ) (c1 , c2 , c3 )
Nominal
Observed
256 65,536 1.677×107
256 65,408 129,955
Stage 1: L1 outer weight. Each of the 256 L1 buckets receives a non-negative weight summing to 1. Uniform: equal-per-bucket, the max-entropy choice on L1 and the canonical no-outer-learning baseline. DoReMi: weights from group-DRO at g = L1 on a 120M-parameter / 2.5B-token proxy (Xie et al., 2023a; Sagawa et al., 2020), learned once at L1 and reused without further learning in every non-Uniform HERMES experiment. WebOrganizer rows under DoReMi use group-DRO computed natively on the WO Topic/Format labels. These are the only two O(1)-proxy outer-weight families tractable at K = 256 (Section 1). We evaluate both at L1 as a sanity audit; the paired characterisation is reported in Appendix L. Our main granularity claim is conditioned on fixed DoReMi outer weights, not on DoReMi being optimal.
et al., 2023), approximate nearest-neighbour search (Jégou et al., 2011; Ge et al., 2013; Liu et al., 2015), and generative modelling (van den Oord et al., 2017; Lee et al., 2022); we repurpose it here as a grouping substrate for pre-training data. The transformed vector hi is encoded by L cascaded vector quantizers, each with codebook e(k) ∈ RK×d . Writing r1 = hi , stage (k) k picks ck = arg maxj∈[K] cos(rk , ej ), emits
Stage 2: Per-granularity sub-bucket sampler. Stage 2 decides which sub-buckets inside each L1 ancestor are eligible and how documents are drawn inside them; it does not re-distribute Stage-1 L1 weights. Samplers differ along two axes: subbucket mass is size-proportional or max-entropy (equal regardless of size); per-document eligibility is all or top-30% by FineWeb-Edu quality score within that sub-bucket. Concretely, letting Cℓ (b1 ) denote the L1 ancestor’s active level-ℓ children, max-entropy samplers set
(k)
qk = eck , and passes residual rk+1 = rkP− qk to the next stage. The reconstruction ĥi = L k=1 qk enters only Lquant ; the downstream label is the code (c1 , . . . , cL ), and prefix lengths ℓ < L name strictly coarser partitions. Codebooks are updated by EMA with k-means initialization and a per-stage stop-gradient commitment term (full form in Appendix D). We set L = 3 and K = 256 in the main experiments; the L1 capacity ablation (Appendix A) varies K. After training, encoding is a deterministic argmax against frozen codebooks. Hierarchy depth L and per-stage codebook size K are themselves design axes; their useful settings are corpus-bounded by candidate competition (Section 5.3), and we discuss this in Limitations. 3.4
1 πb1 (B) = |Cℓ (b , 1 )|
1 ρB (x) = |B| ,
and quality top-30% samplers set πb1 (B) ∝ |B|,
B} ρB (x) = 1{q(x)≥τ , ZB
where q(·) is the FineWeb-Edu quality score, τB is the within-sub-bucket 70th-percentile, and ZB normalises within B. We use four main Stage-2 samplers on the HERMES substrate: L1 max-entropy (no sub-bucket structure below L1); L12 maxentropy (within each L1 ancestor, equal-weight its L12 children); L12 quality top-30% (corrected reader); and L123 quality top-30% (corrected reader). Diagnostic variants (L1-local quality top-30% for the granularity arc) and L123 sidereferences (per-L1 L123 local random coverage, the quota-preserving coverage row for Finding B, and quota-flattening global L123 max-entropy) are defined in Appendix G; they appear in the leaderboard (Appendix C). A configuration is (L1 outer family) × (Stage2 sampler). All controlled experiments share
The sampling taxonomy
A sampler is the composition of two stages. For a target granularity level ℓ and a document x with prefix codes b≤ℓ (x) = (b1 , . . . , bℓ ), the sampling probability factorises as P (x | sampler) = wb1 · πb1 b≤ℓ · ρb≤ℓ (x), where wb1 is the Stage-1 outer weight on the L1 ancestor, πb1 (·) a Stage-2 distribution over that ancestor’s active level-ℓ descendants, and ρB (x) a per-document rule inside leaf sub-bucket B. We separate the two stages so the trade-offs between outer-weight family (w) and per-granularity subbucket rule (π, ρ) are unambiguous. 4
the same HERMES-2563 codebook. The withinsub-bucket quality samplers read the FineWebEdu quality score via a per-key bounded heap; a corrected-reader specification and a 70th-percentile sanity number are in Appendix H. All quality top30% rows we discuss in Findings A and B use the corrected reader.
4
Experimental Setup
4.1
Pre-training
4.3
Learned groupings at L1 =256. KMeans-256, MiniBatchKMeans-256, BisectingKMeans-256 (all from scikit-learn (Pedregosa et al., 2011; Lloyd, 1982; Sculley, 2010; Steinbach et al., 2000)), Plain RVQ-2563 (RVQ without LST), and HERMES2563 (ours). All non-HERMES learned groupings share an identical annotation pipeline (fit on a 1.16M-document subsample at K = 256 with cosine assignment, then deterministic full-corpus labeling) so that only the clustering objective differs across baselines; full details in Appendix E. Heuristic groupings. We apply the published WebOrganizer Topic and Format taxonomies (Wettig et al., 2025) to our corpus at their native granularities. For every grouping, sampler axes are varied as in Section 3.4. Sample-wise selectors are out of scope for this baseline set; see Appendix E for the scope statement.
We train a 1B-parameter LLaMA-style decoder (Touvron et al., 2023) for 25B tokens on each sampler configuration. All runs share the same architecture, optimizer, learning-rate schedule, context length, tokenizer, and token budget; only the upstream sampler varies. The pre-training corpus is an internal ∼50M-document collection pre-filtered with public quality classifiers; we treat it as a fixed source distribution and study how to sample from it, not how to clean it. Sampling is with replacement, so the realised sampled set can exceed the source corpus (see Appendix F). Embeddings used by HERMES are produced once by the same frozen encoder for every document and are never updated. We report 20 ranked 1B/25B checkpoints (14 main rows + 6 HERMES capacity-ablation rows) in Appendix C, spanning granularity, Stage-2 sampler, L1-outer-weight, and grouping-method axes; the full leaderboard is sorted by Avg. 4.2
Baselines
5
Results
We use the Results section to rule out two simpler explanations for HERMES’s gain. The gain is not because HERMES is a better L1 grouping method: five 256-way methods sit at a plateau on standard compactness/mass-balance metrics and the two we have paired downstream are numerically tied (Section 5.1). It is not because finer granularity is universally better: at L123 , the within-bucket quality top-30% rule’s L12 advantage over quotapreserving coverage shrinks to a gap too small to interpret under single-seed evaluation, consistent with candidate competition (Section 5.3). The reliable positive result is conditional: within a fixed DoReMi outer-weight family at granularity L12 , switching the Stage-2 sampler from max-entropy coverage to (corrected-reader) quality top-30% improves Avg. by +0.0253 (z +2.09; Section 5.2). The broader outer-weight choice at this rule remains open: we report the highest-scoring configuration in our sweep in Section 5.4, but we have not measured the Uniform + L12 quality top-30% counterpart and therefore cannot claim DoReMi is necessary for this gain. A paired Uniform vs. DoReMi L1 characterisation is reported as an audit in Appendix L.
Downstream evaluation
We evaluate on 16 capability sub-tasks listed in Appendix I, grouped into four ability families: Basic Skills (6 probe sub-tasks, from Gu et al., 2025), Science QA (5; ARC, SciQ, PIQA, LAB-Bench (Clark et al., 2018; Welbl et al., 2017; Bisk et al., 2020; Laurent et al., 2024)), Language Modeling (HellaSwag (Zellers et al., 2019)), and Others (CommonsenseQA (Talmor et al., 2019), Jeopardy, NaturalQuestions (Kwiatkowski et al., 2019), Social IQA (Sap et al., 2019)). Our headline number is Avg., the family-equal macro-mean accuracy X Avg. = 41 accg , g
which prevents the six Basic Skills sub-tasks from dominating. A secondary z-score column standardises each sub-task across the 20 ranked checkpoints and averages within each family. The z column is a unit conversion for legibility, not a separate metric; Avg. and z are reported together throughout.
5.1
At L1 =256, grouping choice is not the source of gains
Before attributing any gain to HERMES, we first ask whether the gains are simply due to a better 5
256-way clustering. The answer is no: at L1 =256 the grouping axis sits on a plateau on selected compactness/mass-balance metrics (Avg cosine, Neff , entropy), and the two grouping families we have paired downstream are numerically tied. Intrinsic clustering metrics. Five 256-bucket grouping methods evaluated on the same 1.16Mrow fit set and 221k-row validation shard (Appendix E); the baselines use the raw 1024-dim embeddings while HERMES applies its Learned Semantic Transform first. They sit tightly together (Table 2): mean cosine to centroid varies by less than 0.003, effective cluster counts span 236–247, and entropies fall in [5.46, 5.51]. Standard clustering criteria do not distinguish these methods at L1 =256.
Figure 3: Granularity arc under fixed HERMES codebook and DoReMi-L1 outer weights. Solid: quotapreserving coverage. Dashed: corrected-reader quality top-30%. Open diamond: global L123 max-entropy side-reference (quota-flattening). The L12 quality gain collapses at L123 , consistent with candidate competition (Section 5.3).
Paired downstream training. Two of the five have been paired against identical Uniform + L1 max-entropy: Uniform · HERMES-2563 (Avg. 0.4155, z +1.048) and Uniform · KMeans-256 (Avg. 0.4153, z +1.059). They differ by 0.0002 on Avg., numerically tied; the other three are an empirical question. With the L1 grouping axis ruled out, the remaining design axes are where HERMES’s contribution must come from.
5.2
Finding A: Stage-2 rule choice matters at L12
With the L1 clustering explanation ruled out (Section 5.1), we next isolate the Stage-2 sampler axis at a fixed granularity. Holding the L1 outer at DoReMi and the granularity at L12 , switching the Stage-2 sampler from max-entropy coverage to quality top-30% (under the corrected FineWeb-Edu quality reader; Appendix H) raises Avg. by +0.0253 (z +2.09). Only the Stage-2 rule changes between rows; the L1 weights, the granularity, and the HERMES-2563 codebook are held fixed. The L12 point is the peak of the granularity arc in Figure 3 (solid = quota-preserving coverage; dashed = quality top-30%; both evaluated at DoReMi-L1 outer weights). The open diamond at L123 = 0.4061 is the global L123 max-entropy side-reference, which is a quota-flattening variant and not a quota-preserving Stage-2 contrast. Quality top-30% is a within-sub-bucket selection rule: inside each L12 ancestor, the sampler concentrates draws on the top-30% of documents ranked by the FineWeb-Edu quality score, on sizeproportional sub-bucket mass. Max-entropy is a coverage rule: each sub-bucket is equalised regardless of natural size, with all documents inside eligible. The two rules therefore differ on two axes simultaneously, sub-bucket mass distribution (equal vs. size-proportional) and per-document eligibility (all docs vs. within-sub-bucket top-30%), so +0.0253 is a combined Stage-2 rule contrast rather than a pure quality-ranking ablation; we do
Auxiliary label-system diagnostics. HERMES labels are moderately but not redundantly related to the WebOrganizer Topic, WebOrganizer Format, and Topic-over-Source taxonomies on a 259,255document intersection; finer HERMES granularity produces buckets that align more tightly with each reference family while remaining a distinct label basis (Appendix M). Qualitative inspection on a 4.8M-document sample confirms this: all 256 L1 buckets are populated, and distinctive ngrams recover coherent topical or stylistic regions (e.g., books and publishing, music releases, recipes, video games, biomedicine, ophthalmology, astronomy), with L12 children refining each L1 region into narrower sub-regions (e.g., a Music parent splits into classic rock, music videos, MP3 pages, hip-hop, and gospel; Appendix N). A sparsity audit on the full 227M-document annotated corpus separately rules out third-stage codebook collapse, verifies that every L1 cell refines into a multi-child L12 subtree, and shows that L12 → L123 refinement is concentrated in the higher-mass tail of L12 cells (Appendix F, extended discussion). 6
Table 2: Intrinsic plateau: five learned grouping methods at L1 =256 are mutually indistinguishable on standard clustering P metrics. Effective cluster count is the perplexity of the bucket-mass distribution, Neff = exp(H) with H = − k pk ln pk and pk the fraction of documents in bucket k. Method KMeans (sklearn) MiniBatchKMeans BisectingKMeans Plain RVQ (c1 ) HERMES (c1 , ours)
Eff. clusters
Entropy
Avg cos→centroid
Recall@10
240.98 238.55 247.13 235.99 240.46
5.485 5.475 5.510 5.464 5.483
0.8740 0.8731 0.8709 0.8719 0.8739
0.356 0.345 0.285 0.348 0.355
Table 3: At fixed L12 granularity under DoReMi L1 outer weights, switching the Stage-2 sampler from max-entropy coverage to quality top-30% (corrected FineWeb-Edu reader) raises Avg. by +0.0253 (z +2.09). Each row is a single 1B/25B training seed; we discuss the single-seed limitation in Section 7. Stage-2 sampler (DoReMi; granularity L12 )
Avg.
z
L12 max-entropy coverage L12 quality top-30% (corrected reader)
0.3969 0.4222
−0.458 +1.628
Figure 4: Cumulative distribution of bucket sizes (logscale x-axis) on the realised L12 and L123 selection sets. Refining to L123 shifts the CDF leftward: the median pool falls from 2,271 to 429 documents, and 36% of L123 buckets contain fewer than 30 documents.
not attempt to isolate the two axes here. At L12 , sub-buckets are large enough (median pool 2,271 documents; Figure 4) that the within-bucket quality ranking is comparatively stable, so the quality top-30% configuration extracts a meaningful signal that the coverage configuration misses. Whether the same Stage-2 advantage survives at finer granularities is the question Section 5.3 addresses. 5.3
should diminish. The leaderboard supplies a direct test. Holding DoReMi-L1 outer weights fixed and pairing both Stage-2 rules in their quota-preserving forms (Stage-1 L1 weights preserved exactly), the +0.0253 L12 advantage of quality top-30% over coverage collapses at L123 : quality top-30% reaches 0.3988 and per-L1 L123 local random coverage reaches 0.3986, a ∆ = +0.0002 that is too small to interpret under single-seed evaluation. The L12 Stage-2 advantage of quality top30% over coverage thus loses its measurable edge at L123 , consistent with the candidate-competition account in which within-bucket quality ranking is no longer measurable over coverage once median pools contract to ∼429 documents. A separate quota-flattening variant (global L123 max-entropy, 0.4061) is a side-reference (Appendix C) that differs from L123 quality top-30% on both the Stage-2 rule and quota preservation, so we do not read it as a fair Stage-2 contrast. We frame this as evidence consistent with candidate competition rather than causal isolation; the full Stage-2 rule × granularity table is in Appendix K, and “only-large-sub-
Finding B: the L12 advantage collapses at L123
HERMES exposes a boundary on the granularity axis we call candidate competition: as sub-buckets shrink, the within-bucket pool contracts, and perbucket top-k is no longer a stable proxy for the global top-k it is meant to approximate. We measure this contraction directly on the sampler’s realised selection set. The median candidate pool contracts 5.3× from L12 (2,271) to L123 (429), and the contraction is robust across percentiles (Figure 4; full statistics in Appendix F). At this scale a per-sub-bucket top-30% rule reduces to ranking inside pools of ∼429 documents, a regime in which the withinsub-bucket score is a less reliable proxy for the global ranking. Max-entropy at L123 has no such dependency. The mechanism is therefore consistent with: at fine granularity, the L12 Stage-2 advantage of quality top-30% over quota-preserving coverage 7
buckets”/“synthetic-shrinkage” controls are noted in Section 7. 5.4
document scoring) into a continuous granularity dial the practitioner sets by prefix length, paying per-document annotation cost once and trading resolution at sample time.
Synthesis: the rank-1 configuration is L12 quality top-30% under DoReMi-L1
7
We report the highest-scoring configuration in our sweep alongside its closest neighbours (full leaderboard in Table 4, Appendix C); we do not claim to have identified the optimal configuration across the full design space. Rank 1 (HERMES + DoReMi-L1 + L12 quality top-30% under the corrected reader, Avg. 0.4222) applies the DoReMi L1 weights and then, within each L12 sub-bucket, concentrates draws on the top-30% of documents ranked by the FineWeb-Edu quality score: the DoReMi L1 quota is preserved, and within-bucket selection extracts a meaningful quality signal at a granularity where pools are still large enough for the ranking to be stable (Section 5.3). Ranks 2–4 sit at the L1 =256 plateau: HERMES, KMeans, and WebOrganizer-Format under Uniform + L1 max-entropy land within ∼0.002 Avg. of one another, consistent with Section 5.1. What we do not claim. We have not run a Uniform + L12 quality top-30% configuration under the corrected reader, so the outer-weight choice at the rank-1 Stage-2 rule cannot be adjudicated from the leaderboard alone; in particular, we cannot conclude that DoReMi is necessary for the rank-1 Avg. The paired Uniform vs. DoReMi L1 characterisation (Appendix L) shows that the choice between our two O(1)-proxy outer weights is not free at L1 , but a fair comparison at L12 quality top-30% would require an additional 1B/25B run.
6
Conclusion
HERMES is a data-derived multi-granularity corpus-labeling substrate built from a Learned Semantic Transform and a 3-stage RVQ: one annotation, prefix-readable granularity from L1 =256 to L123 ∼130k. At L1 =256, HERMES sits at a plateau with the KMeans family on standard clustering metrics and is downstream-matched within 0.0002 Avg., so the contribution is the substrate, not the clusterer. At L12 under DoReMi-L1, switching the Stage-2 sampler from max-entropy coverage to corrected-reader quality top-30% raises Avg. by +0.0253 (z +2.09; Finding A). The advantage collapses at L123 (quality vs. per-L1 local random coverage: 0.3988 vs. 0.3986; Finding B), consistent with candidate competition: a 5.3× median pool contraction (2,271 → 429) leaves the withinbucket ranking signal no longer measurable over coverage. Pre-training data design becomes a labelsystem question: granularity and the Stage-2 sampler are jointly determined axes on the substrate, not independent hyperparameters. Promising next directions are (a) deeper or wider RVQ hierarchies as corpus size grows, with the useful depth bounded by candidate competition; (b) jointly learned outer weights at multiple prefix lengths; and (c) soft within-bucket coverage signals, including intrinsic per-bucket priors such as the RVQ reconstruction error already produced by the substrate, as an alternative to hard top-k selection on external scorers.
Discussion Limitations
The contribution is the substrate, not the clustering algorithm: HERMES sits at a plateau with four 256bucket clusterers on standard compactness/massbalance metrics and is downstream-matched to KMeans within 0.0002 Avg. (Section 5.1). What HERMES adds is a single offline annotation in which granularity becomes a free variable, exposed as a prefix length. In a flat KMeans pipeline a granularity sweep is a sequence of independent clusterings; in HERMES the same hierarchy is read at different prefix lengths, which is what makes the granularity-by-sampler interaction (Section 5.3) cleanly measurable. More structurally, HERMES turns the corpus-labeling axis (previously a binary choice between coarse fixed labels and per-
Scale. All conclusions are at 1B parameters and 25B tokens on one internal corpus; whether the interaction persists at 7B/300B is not validated here. Sub-bucket-size shrinkage is a structural property of residual quantization rather than a small-scale artefact, so we expect the qualitative direction to survive, but the granularity at which the candidate pool collapses may shift with scale (Kaplan et al., 2020). Encoder. A single frozen encoder produces the input geometry; different encoders may shift the realised L123 cardinality and the threshold at which candidate competition becomes dominant. Outer-weight axis is under-sampled. Ap8
pendix L pairs Uniform vs DoReMi at L1 + L1 max-entropy only; our DoReMi weights are learned once at L1 and reused at L12 and L123 without re-learning (the proxy budget makes natively training DoReMi proxies at finer granularity impractical, and regression-style alternatives are infeasible at K = 256–130k). Whether learned outer weights at the granularity actually used by Stage-2 close or invert the gap is open. Outer weight at rank-1 Stage-2 rule is undertested. Our highest-scoring configuration (Section 5.4) uses DoReMi outer weights with L12 quality top-30% under the corrected reader, but we did not run the parallel Uniform + L12 quality top-30%. Without this counterfactual, we cannot claim DoReMi is necessary for the rank-1 Avg., nor that DoReMi is the optimal outer-weight family for this Stage-2 rule. The Stage-2 contrast we report (Section 5.2, within DoReMi) is a controlled within-slice comparison and is not affected by this gap. Within-sub-bucket scoring function. The quality samplers use the FineWeb-Edu quality score field released with the FineWeb corpus (Penedo et al., 2024). Whether the sign of Finding A and the advantage collapse at L123 in Finding B hold under perplexity-based or other classifier-based scoring families is a direct follow-up; scores whose distribution differs sharply from FineWeb-Edu’s right tail may interact with the small-pool regime differently. Mechanism evidence strength. The 5.3× median sub-bucket shrinkage is a direct measurement, and the L12 advantage collapse at L123 under the quota-preserving pair (Section 5.3) is a paired training contrast in the direction consistent with the candidate-competition account. We treat this as evidence consistent with the mechanism rather than causal isolation. Two controls that would tighten the claim, the “only-large-sub-buckets” and “synthetic-shrinkage” variants, are future work. Hierarchy depth and codebook size. We sweep granularity by prefix length on a single L = 3, K = 256 codebook chosen for the ∼50Mdocument corpus. Deeper hierarchies (larger L) or wider codebooks (larger K, possibly nonuniform across stages) are natural extensions. The candidate-competition account suggests diminishing returns once deeper prefixes push the median sub-bucket below the regime where within-bucket ranking is stable (∼429 documents at L123 here); larger corpora should support deeper hierarchies
before this floor is hit. The framework’s extensibility via RVQ-style codebook design, including deeper hierarchies (larger L) and non-uniform perstage K, is, we argue, a positive selling point of the substrate rather than a caveat. Systematic depth-bycorpus-size sweeps are left to future work.
Artifact Release We release the source code implementing the HERMES annotation pipeline (LST training, RVQ training, and inference). An anonymized artifact submission accompanies this paper.
Ethical Considerations The pre-training corpus is an internal ∼50Mdocument collection that has been pre-filtered with public quality classifiers; we treat it as a fixed source distribution and study sampling, not curation. No additional PII filtering is performed by our work: we inherit the upstream classifier’s coverage. The 20 ranked 1B/25B runs reported here total approximately 100 GPU-days on 8 GPUs at FP32; we did not run extra ablations beyond what is reported. The corpus and encoder are internal, which limits direct reproduction; the source code release described above allows others to apply the pipeline to their own corpus and encoder.
References Yonatan Bisk, Rowan Zellers, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7432–7439. Zhuo Chen, Yuxuan Miao, Supryadi, and Deyi Xiong. 2026. Data mixing for large language models pretraining: A survey and outlook. arXiv preprint arXiv:2604.16380. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2023. High fidelity neural audio compression. Transactions on Machine Learning Research. Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan Celine Lin, Jan Kautz, and Pavlo Molchanov. 2025. Nemotron-CLIMB: CLustering-based Iterative data
9
Mixture Bootstrapping for language model pretraining. In Advances in Neural Information Processing Systems.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2025b. RegMix: Data mixture as regression for language model pre-training. In International Conference on Learning Representations.
Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. 2013. Optimized product quantization for approximate nearest neighbor search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2946–2953.
Shicong Liu, Hongtao Lu, and Junru Shao. 2015. Improved residual vector quantization for highdimensional approximate nearest neighbor search. arXiv preprint arXiv:1509.05195.
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. 2025. OLMES: A standard for language model evaluations. In Findings of the Association for Computational Linguistics: NAACL 2025.
Stuart P. Lloyd. 1982. Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2):129–137. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830.
Iris A. M. Huijben, Matthijs Douze, Matthew J. Muckley, Ruud J. G. van Sloun, and Jakob Verbeek. 2024. Residual quantization with implicit neural codebooks. In International Conference on Machine Learning. Hervé Jégou, Matthijs Douze, and Cordelia Schmid. 2011. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2024. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. In Advances in Neural Information Processing Systems.
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. 2024. DataInf: Efficiently estimating data influence in LoRA-tuned LLMs and diffusion models. In International Conference on Learning Representations.
Jiahui Peng, Xinlin Zhuang, Jiantao Qiu, Ren Ma, Jing Yu, He Zhu, and Conghui He. 2025. Topic over source: The key to effective data mixing for language models pre-training. arXiv preprint. Preprint, arXiv:2502.16802.
Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. 2024. LAB-Bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. 2022. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11513–11522.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. SocialIQA: Commonsense reasoning about social interactions. In Proceedings of EMNLP, pages 4463–4473. David Sculley. 2010. Web-scale k-means clustering. In Proceedings of the 19th International Conference on World Wide Web, pages 1177–1178.
Fengze Liu, Weidong Zhou, Binbin Liu, Zhimiao Yu, Yifan Zhang, Haobin Lin, Yifeng Yu, Bingni Zhang, Xiaohuan Zhou, Taifeng Wang, and Yong Cao. 2025a. QuaDMix: Quality-diversity balanced data selection for efficient LLM pretraining. arXiv preprint arXiv:2504.16511.
Michael Steinbach, George Karypis, and Vipin Kumar. 2000. A comparison of document clustering techniques. In KDD Workshop on Text Mining.
10
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of NAACL, pages 4149– 4158.
Wanyun Xie, Francesco Tonin, and Volkan Cevher. 2025. Chameleon: A flexible data-mixing framework for language model pretraining and finetuning. In International Conference on Machine Learning. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. 2025. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In International Conference on Learning Representations.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
Zichun Yu, Spandan Das, and Chenyan Xiong. 2024. MATES: Model-aware data selection for efficient pretraining with data influence models. In Advances in Neural Information Processing Systems.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural discrete representation learning. In Advances in Neural Information Processing Systems.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2022. SoundStream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507.
Phil Wang. 2021. vector-quantize-pytorch. https://github.com/lucidrains/ vector-quantize-pytorch. Software.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy Usergenerated Text, pages 94–106. Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. 2024. QuRating: Selecting high-quality data for training language models. In Proceedings of the 41st International Conference on Machine Learning. Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. 2025. Organize the web: Constructing domains enhances pre-training data curation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 66678–66705. PMLR. Xiangyu Xi, Deyang Kong, Jian Yang, Jiawei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, and Wei Ye. 2025. SampleMix: A sample-wise pre-training data mixing strategy by coordinating data quality and diversity. arXiv preprint arXiv:2503.01506. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. LESS: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. 2023a. DoReMi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. 2023b. Data selection for language models via importance resampling. In Advances in Neural Information Processing Systems.
11
A
Capacity Ablation: L1 Codebook Size
To isolate the role of the coarsest stage, we hold stages 2 and 3 at K = 256 and vary the L1 codebook size K ∈ {32, 64, 128, 256}. All four runs share the same encoder, training recipe, and Uniform + L1 max-entropy sampler. We read this through L1’s role as a capacity gate. Below the gate (K = 32, 64), L1 under-resolves the corpus; stages 2–3 absorb dimensions of variation L1 should have captured, so each L1 bucket is internally heterogeneous and the sampler falls back to a near-uniform draw. At the trough (K = 128), L1 acts non-trivially but is mis-aligned with corpus structure; the sampler can no longer fall back to a uniform draw, but its reweighting points in the wrong direction. At K = 256, the effective cluster count saturates near nominal (240/256), L1 alone carries the partition, and Avg. jumps. The shape is not “more buckets is always better”; it is “L1 must be large enough that bucket-level reweighting and corpus structure are aligned.” This ablation is a capacity dimension within a single grouping family and does not contradict the L1 =256 plateau across grouping methods (Section 5.1).
B
Figure 5: HERMES L1 capacity ablation. Holding stages 2–3 fixed, the L1 capacity sweep of 16-task Avg. accuracy is non-monotone: K = 128 is the trough, while K = 256 is the best L1 setting in this study. Xaxis tick labels show both the nominal codebook size K and the realised effective cluster count Neff at that K (saturating near 240 at K = 256; see Table 2 for the Neff definition).
ing plateau (Section 5.1) and the outer-weight paired plateau (Appendix L): HERMES, KMeans, and WebOrganizer-Format under Uniform + L1 max-entropy are within roughly 0.002 Avg. of one another (ranks 2–4); the same grouping families under DoReMi land 0.9–2.1 z lower (ranks 9, 10, 17), with the WO-Topic pair (ranks 5, 6) being the one too small to interpret under single-seed evaluation. The bottom tier is populated by L123 granularity rows under DoReMi (ranks 13, 14, 15, which sit within 0.002 of one another and below the L1 plateau) and capacity-starved HERMES variants under both Uniform (ranks 11, 12, 16) and DoReMi (ranks 18, 19, 20). The L12 quality top30% advantage at rank 1 collapses to noise at L123 (rank 13 quality top-30% Avg. 0.3988 vs. rank 14 per-L1 L123 local random coverage Avg. 0.3986), consistent with the candidate-competition account in Section 5.3.
Operational Definitions of Granularity
We use three granularity levels, each defined by a prefix of the HERMES code (c1 , c2 , c3 ). L1 saturates its nominal capacity. L12 is essentially fully populated (99.8% of nominal 2562 joint codes are observed). L123 deviates substantially: only 129,955 of 2563 ≈ 16.8M nominal joint codes are realized, a roughly two-orders-of-magnitude gap. We attribute this to natural sparsity of stage-3 RVQ residuals on this corpus: once the first two stages have absorbed the dominant directions of variation, the third-stage residuals concentrate on a thin manifold of the joint code space. L123 is operationally defined as “the prefix length that realizes approximately 130k buckets,” and it is in this regime that candidate competition takes hold.
C
D
HERMES Training Details
Encoder. An internally trained 1024-dim sentence-level encoder, kept frozen throughout HERMES training. Embeddings are read once per document; no fine-tuning at this stage.
Full Leaderboard
The top of the table is occupied by mechanismaligned configurations: at fixed DoReMi L1 outer weights, L12 quality top-30% under the corrected reader (rank 1), and the Uniform + L1 max-entropy family (ranks 2–4). WO-Topic under both outerweight families (ranks 5, 6) and global L123 maxentropy coverage (rank 7) sit just below. The middle tier corresponds jointly to the L1 =256 group-
Optimizer and schedule. AdamW, learning rate 3 × 10−4 , weight decay 1 × 10−4 , 10 epochs, 8GPU distributed data-parallel (DDP), batch size 1024, single-precision (FP32) throughout. Codebook initialization and EMA. EMA codebook updates with dead-code threshold 2 and k12
Table 4: Full leaderboard sorted by Avg. (descending), with z standardised across all 20 ranked 1B/25B checkpoints (14 main rows + 6 HERMES capacity-ablation rows). All rows use a single training seed. “Corrected” marks runs that use the corrected FineWeb-Edu quality reader (Appendix H). Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Configuration
L1 outer
Stage-2 sampler
Avg.
z
HERMES + L12 quality top-30% (corrected) HERMES-2563 + L1 max-entropy (ours) KMeans-256 + L1 max-entropy WebOrganizer-Format WebOrganizer-Topic + Topic DRO WebOrganizer-Topic + Topic maxentropy HERMES + global L123 max-entropy coverage HERMES + L1-local quality top-30% (corrected) WebOrganizer-Format + Format DRO HERMES-2563 + L1 max-entropy (ours) HERMES cap-32 + L1 max-entropy HERMES cap-64 + L1 max-entropy HERMES + L123 quality top-30% (corrected) HERMES + per-L1 L123 local random coverage HERMES + L12 max-entropy coverage HERMES cap-128 + L1 max-entropy KMeans-256 + L1 max-entropy HERMES cap-64 + L1 max-entropy HERMES cap-128 + L1 max-entropy HERMES cap-32 + L1 max-entropy
DoReMi
L12 quality top-30%
0.4222
+1.628
Uniform Uniform Uniform on Format DoReMi on Topic Uniform on Topic
L1 max-entropy L1 max-entropy random within Format random within Topic random within Topic
0.4155 0.4153 0.4134 0.4082 0.4079
+1.048 +1.059 +0.949 +0.508 +0.469
DoReMi
global L123 max-entropy
0.4061
+0.317
DoReMi
L1-local quality top-30%
0.4045
+0.114
DoReMi on Format DoReMi Uniform Uniform DoReMi
random within Format L1 max-entropy L1 max-entropy L1 max-entropy L123 quality top-30%
0.4029 0.4028 0.3994 0.3992 0.3988
+0.128 +0.048 −0.312 −0.362 −0.294
DoReMi
per-L1 L123 local random
0.3986
−0.302
DoReMi Uniform DoReMi DoReMi DoReMi DoReMi
L12 max-entropy L1 max-entropy L1 max-entropy L1 max-entropy L1 max-entropy L1 max-entropy
0.3969 0.3934 0.3889 0.3887 0.3886 0.3798
−0.458 −0.747 −1.147 −1.163 −1.181 −1.885
means initialization (10 iterations); stage k’s codebook is learned in the residual space left by stages 1, . . . , k − 1, so prefix codes form a strict hierarchy.
LST objective is L = λstruct Lstruct + λquant Lquant + λortho Lortho + Lcommit ,
LST loss equations. LST is trained jointly with the RVQ codebooks under three named losses, following the rotate-then-quantize lineage of Ge et al. (2013). Pairwise structure preservation keeps cosine geometry between raw and transformed embeddings; over M = 2048 random pairs (i, j) per minibatch, X 2 1 Lstruct = M cos(ei , ej ) − cos(hi , hj ) .
with λstruct = λquant = 1.0 and λortho = 0.1. The per-stage commitment term is Lcommit = β
2 , 2
with β = 0.25 and sg(·) the stop-gradient operator. Inference. Implementation builds on the opensource vector-quantize-pytorch library (Wang, 2021); we annotate the entire ∼50M-document corpus once and reuse the same annotation across all sampler experiments.
Quantization-aware reconstruction P couples LST to the codebooks. Letting ĥi = 3k=1 qk (hi ) denote the RVQ reconstruction (projected to the unit sphere), over a minibatch of size B, B X
sg(qk ) − rk
k=1
(i,j)
Lquant = B1
L X
E
Annotation Pipeline for Unsupervised Clustering Baselines
All unsupervised clustering baselines reported in Section 5.1 and Table 2 (KMeans, MiniBatchKMeans, BisectingKMeans, Plain RVQ) share an identical two-stage annotation procedure that differs from HERMES only in the clustering objective; HERMES’s own training is described in Appendix D.
∥ĥi − hi ∥22 .
i=1
Orthogonality prevents representation collapse: Lortho = ∥W ⊤ W − I∥2F , combined with a perstep SVD projection onto the orthogonal manifold (writing W = U ΣV ⊤ , set W ← U V ⊤ ). The total 13
Table 5: Per-granularity bucket statistics on realised selection sets. The L12 → L123 median shrinks 5.3×.
Inputs. The same ∼50M-document corpus as HERMES, with the same precomputed 1024-dim sentence-level embeddings ei . No embedding is recomputed.
Granularity L1 L12 L123
Stage 1: Centroid fitting on a held-out subsample. All baselines fit on a fixed subsample of 1,158,563 embeddings (the same set across methods); a separate 221,476-embedding validation shard reports intrinsic metrics for Table 2. Shared settings: K = 256 centroids, embeddings un-normalised at fit time and L2-normalised at assignment, a fixed random seed, and up to 100 update iterations. KMeans, MiniBatchKMeans, and BisectingKMeans all use scikit-learn’s default initialisation and restart policies for each algorithm; the only non-default we set is MiniBatchKMeans’s minibatch size at 8192. Plain RVQ uses EMA codebook updates on the same 1.16M-row sample, without the Learned Semantic Transform; this is what “plain” means in our naming.
Active buckets
Median
Documents
256 65,408 129,955
n/a 2,271 429
∼50M ∼68M ∼68M
interface that a granularity-aware sampler could plug into. A head-to-head against per-sample selectors at smaller regimes is a natural follow-up.
F
Sampling Diagnostics
We summarise the bucket-size statistics that underpin the candidate-competition diagnostic. Pergranularity statistics are computed on the actual sampled set used by each sampler (∼68M selected documents under DoReMi outer weights for the L12 and L123 samplers; sampling is with replacement, so the sampled set exceeds the ∼50M source corpus), not on the raw 50M-document corpus, so they reflect the candidate pool that within-subbucket selection sees. Full bucket-size distributions on log scale show the L12 → L123 bulk shifting roughly an order of magnitude leftward, with the L123 lower tail developing a heavy concentration of sub-buckets containing only a handful of documents: the regime in which any within-sub-bucket ranking criterion reduces to a near-uniform draw. “Active buckets” counts sub-buckets receiving at least one document; for L123 this is 129,955 out of nominal 2563 .
Stage 2: Full-corpus annotation. Saved centroids are L2-normalised once and applied to every document in the full corpus by cosine assignment, arg maxk ẽ⊤ i c̃k (argmin Euclidean is supported but not used). Annotation runs once per method and is reused across every sampler ablation in this paper, mirroring HERMES’s reuse (Appendix D). Why this licenses the plateau claim. The 1.16M-row fit set, 1024-dim raw embedding, K = 256, normalisation convention, iteration budget, and random seed are identical across KMeans, MiniBatchKMeans, BisectingKMeans, and Plain RVQ; the only thing that differs is the clustering objective. This is what licenses the Section 5.1 claim that intrinsic Avg. cosine spreads by less than 0.003 across all five 256-bucket methods (KMeans 0.8740, MiniBatchKMeans 0.8731, BisectingKMeans 0.8709, Plain RVQ-c1 0.8719, HERMES-c1 0.8739): they are not measured on different corpora or under different normalisation conventions.
Further sparsity statistics on the full annotated corpus. On the full 227,447,180-document annotated set, we additionally compute the effective cell count Neff = exp(H), the Gini coefficient over per-cell document counts, the top-k for 50% cumulative mass, and per-cell P10/P90 (Table 6). We also compute, for every parent at L1 and L12 , whether a single child carries more than 90% of the parent’s mass. Per-cell medians are reported in Table 5 above (and Figure 4); we do not duplicate them here.
Sample-wise selectors are out of scope for this baseline set. DSIR (Xie et al., 2023b), QuRating (Wettig et al., 2024), MATES (Yu et al., 2024), LESS (Xia et al., 2024), DataInf (Kwon et al., 2024), SampleMix (Xi et al., 2025), and QuaDMix (Liu et al., 2025a) score documents individually rather than reweighting groups. At the 50Mdocument × 1B-parameter × 25B-token scale we target they are costly to score, tune, and validate, and most do not expose an explicit mixture-control
What this rules out. Effective cell count rises monotonically (240.5 → 40,693 → 46,007): the third RVQ stage adds effective discrimination above L12 even though only 0.77% of nominal L123 cells are populated. The Gini coefficient rises with granularity (0.20 → 0.52 → 0.73): mass becomes Zipfian as a fixed corpus is distributed over 14
Table 6: Per-prefix sparsity statistics on the full 227Mdocument annotated corpus, complementing Table 5. Statistic
L1
L12
H
(a) Field presence. On a 2,000-row sanity scan of the corpus metadata, the nested path containing the FineWeb-Edu quality score was populated in 2000/2000 rows, while a top-level fallback field was populated in 0/2000. The corrected reader prefers the nested path and falls back to the toplevel only for forward compatibility. All quality top-30% rows discussed in Findings A and B (Sections 5.2, 5.3) and listed in the leaderboard (Appendix C) use the corrected reader. (b) Threshold sanity number. A dry run of the corrected reader over 556,976 quality values yields a global 70th-percentile τglobal = 1.374, statistically indistinguishable from the nested-only p70 = 1.380 measured directly on the sanity scan. τglobal is a sanity number only; the within-subbucket quality samplers compute their thresholds per sub-bucket via a bounded heap that keeps the local top-30% within each sub-bucket, so the actual threshold τB varies with B and is, on heavy-tailed sub-buckets, very different from τglobal .
L123
Nominal cells 256 65,536 1.68×107 Neff = exp(H) 240.5 40,693 46,007 Gini over cell mass 0.20 0.52 0.73 Top-k for 50% mass 95 11,215 12,268 P10 per cell 534k 433 2 P90 per cell 1.27M 7,798 5,017 >90%-dominant parents 0/256 57,718/65,408 —
more cells, but no cell at any level carries more than ∼1% of the corpus. Parent-child refinement. Zero of the 256 L1 cells have a single L12 child carrying more than 90% of the parent’s mass: every L1 cell refines into a genuine multi-child subtree. At L12 → L123 , 57,718 of 65,408 L12 cells (88%) have a dominant (>90%) L123 child. This is expected given that the median L12 bucket carries only 2,271 documents (Table 5): most L12 cells do not have enough mass to populate multiple L123 children with meaningful weight. The remaining ∼7,700 L12 cells (the higher-mass tail) account for the bulk of L123 refinement; this pattern supports the Section 5.3 regime, in which sub-buckets containing meaningful refinement coexist with low-mass tails.
G
Quality Field Sanity
I
Capability Sub-task List
We evaluate on 16 capability sub-tasks grouped into four ability families. The six Basic Skills probes are taken from the OLMES capability suite (Gu et al., 2025); the remaining ten are standard public benchmarks. Basic Skills (6 tasks): surface-level reasoning and string manipulation probes from OLMES (Gu et al., 2025): arithmetic, coding, common knowledge, logical reasoning, pattern recognition, and string operations. Science QA (5 tasks): factual/scientific knowledge: ARC-Easy and ARC-Challenge (Clark et al., 2018), SciQ (Welbl et al., 2017), PIQA (Bisk et al., 2020), and the DB-QA subset of LAB-Bench (Laurent et al., 2024). Language Modeling (1 task): sentence-level commonsense continuation: HellaSwag (Zellers et al., 2019). Others (4 tasks): open-domain and social knowledge: CommonsenseQA (Talmor et al., 2019), Jeopardy (a quiz-show open-domain task), Natural Questions (Kwiatkowski et al., 2019), and Social IQA (Sap et al., 2019). All sub-tasks are run in their standard pretrain configuration; per-task metric is accuracy. Avg. is the family-equal macro-mean defined in Section 4.2.
Diagnostic and Side-Reference Stage-2 Samplers
Beyond the four main Stage-2 samplers defined in Section 3.4, we use one diagnostic and two sidereference variants on the HERMES substrate. L1local quality top-30% (diagnostic) selects, within each L1 ancestor, the top-30% by FineWeb-Edu quality on size-proportional L12 mass; this row anchors the L1 point of the granularity arc (Appendix J). Per-L1 L123 local random coverage (side-reference, quota-preserving) draws L123 subbuckets uniformly within each L1 ancestor, preserving the Stage-1 L1 quota exactly; this is the coverage row that pairs with L123 quality top-30% in Finding B (Section 5.3). Global L123 max-entropy (side-reference, quota-flattening) flattens all ∼130k L123 sub-buckets corpus-wide; it inadvertently reshapes the L1 mass towards L1 cells that split into many L123 children, so it is not a quota-preserving Stage-2 contrast. All three variants appear in the leaderboard (Appendix C); only the per-L1 local random row is used as the coverage end of the quota-preserving Finding B pair. 15
Table 7: Granularity arc under DoReMi-L1 outer weights and within-sub-bucket quality top-30% (corrected FineWeb-Edu reader). The arc peaks at L12 ; the L12 → L123 drop is consistent with the candidatecompetition account in Section 5.3. Granularity (DoReMi; quality top30% Stage-2, corrected) L1 (L1-local quality top-30%) L12 (∼65k buckets) L123 (∼130k buckets)
J
Avg.
z
0.4045 0.4222 0.3988
+0.114 +1.628 −0.294
Four grouping families have been paired at L1 + L1 max-entropy, varying only whether the 256 L1 weights come from Uniform or DoReMi. In three of four paired contrasts, replacing Uniform with DoReMi at L1 reduces Avg. by 0.010– 0.026 accuracy points (z swing −0.82 to −2.21); WebOrganizer-Topic is numerically tied. We do not read this as a structural claim about DoReMistyle optimisers in general: this comparison is on a different axis from granularity, and the matched counterparts for L12 quality top-30% (the rank-1 row) have not been run.
Granularity Arc Under Quality Top-30%
Why DoReMi might underperform Uniform at L1 . One natural reading is that group-DRO worst-group loss is misaligned with capability Avg.: DoReMi up-weights groups whose proxymodel loss is high, which at this scale tend to be format-heavy clusters (OCR fragments, code templates) that contribute little to the 16-task capability suite; geometric cluster groupings like KMeans amplify the misalignment (the largest ∆ in the table). Whether this underperformance reverses at finer granularities or under different Stage-2 rules is open.
The candidate-competition mechanism makes a directional suggestion along the granularity axis when Stage-2 is held at per-sub-bucket FineWebEdu quality top-30% under DoReMi-L1 outer weights: the arc should peak at the intermediate granularity where buckets are large enough for the within-bucket quality ranking to be stable, with the L12 → L123 leg falling once the sub-bucket pool contracts. The three corrected-reader rows we have are consistent with this picture: the arc rises from L1 to L12 , then falls back to ∼0.399 at L123 , where it converges with the quota-preserving L123 per-L1 local random coverage row (0.3986; Appendix C). The candidate-competition account is therefore supported by the L12 → L123 drop in the quality top-30% line, not by a quality-vs-coverage sign flip at L123 . The peak at L12 and the drop at L123 are consistent with the candidate-competition account: at L123 , the median candidate pool is 429 documents (Appendix F), below the regime where within-subbucket ranking is stable. A non-decreasing arc, or a coarse-end collapse, would have been inconsistent with the account; neither is observed.
K
Detailed Finding B Numbers
L
Outer-Weight Characterisation at L1
M
Label-System Relationship
We audit how HERMES labels relate to three reference label families on the intersection of documents annotated by all four systems (n = 259,255): WebOrganizer Topic, WebOrganizer Format, and Topic-over-Source (Peng et al., 2025). Each reference family has 24 classes, so the maximum reference entropy is log2 24 ≈ 4.58 bits. For each (HERMES prefix, reference label) pair we report arithmetic-mean normalised mutual information (NMI), the median per-bucket purity (maxclass fraction over the reference label, computed per HERMES bucket and aggregated by median across HERMES buckets), and conditional entropy H(reference | HERMES) in bits. HERMES is not a clone of any reference family. The highest NMI in Table 10 is 0.40 (HERMES L1 vs. WO Topic), which is moderate, not identity. NMI with WO Format is the lowest (0.13 at L1 , 0.24 at L12 /L123 ); ToS sits in between (0.33 across all granularities). HERMES therefore captures a topical-leaning signal that overlaps but is not coextensive with any of the three reference families.
We report the paired L1 comparison between our two O(1)-proxy outer-weight families (Uniform and DoReMi; Section 3.4) as a sanity audit, not as a structural finding. The paper’s main granularity claim (Finding A, Section 5.2) is conditioned on fixed DoReMi outer weights; the comparison below characterises the cost of that choice at L1 , where we have the matched runs. We do not project these results to L123 , where the Uniform counterpart has not been run (see Section 5.4 and Limitations).
Finer granularity sharpens topical alignment. Median per-bucket purity climbs monotonically 16
Table 8: Stage-2 rule × granularity under fixed DoReMi-L1 outer weights and HERMES-2563 codebook, with both rules in their quota-preserving forms (Stage-1 L1 weights preserved exactly). At L12 (median pool 2,271 docs), quality top-30% outperforms coverage by ∆ = +0.0253 (Finding A). At L123 (median pool 429 docs), the L12 advantage collapses: quality top-30% and per-L1 L123 local random coverage are numerically tied (∆ = +0.0002, too small to interpret under single-seed evaluation), consistent with the candidate-competition mechanism (Section 5.3). The global L123 max-entropy row (0.4061) is a quota-flattening side-reference (Appendix C), not a quota-preserving Stage-2 contrast, and is not included here. All quality rows use the corrected FineWeb-Edu reader; single 1B/25B training seed per cell. Granularity
Stage-2 rule
Mass dist.
Per-doc eligibility
Avg.
L12 L12 L123 L123
max-entropy (per-L1) quality top-30% per-L1 local random quality top-30%
equal sub-bucket size-proportional equal sub-bucket size-proportional
all docs top-30% by quality all docs top-30% by quality
0.3969 0.4222 0.3986 0.3988
Table 9: Paired Uniform vs. DoReMi at L1 + L1 max-entropy across four grouping families. We report the comparison descriptively; the sign at finer granularities is not directly testable from this table. Grouping family
Uniform (Avg / z)
DoReMi (Avg / z)
∆ Avg
∆z
HERMES-2563 (c1 , ours) KMeans-256 WebOrganizer-Format WebOrganizer-Topic
0.4155 / +1.048 0.4153 / +1.059 0.4134 / +0.949 0.4079 / +0.469
0.4028 / +0.048 0.3889 / −1.147 0.4029 / +0.128 0.4082 / +0.508
−0.0127 −0.0264 −0.0105 +0.0003
−1.000 −2.206 −0.821 +0.039
Table 10: HERMES label system vs. three reference families on a 259,255-document shared intersection. Each reference family has 24 classes. Purity is the median across HERMES buckets of the max-class fraction over the reference label; H(· | ·) is in bits, against a 4.58-bit reference-label prior. The shared intersection populates 52,536 L12 and 57,512 L123 HERMES buckets, fewer than the full-corpus totals (Table 5), because of the smaller document sample. HERMES Reference
extent than at L1 . The format axis is comparatively underresolved. Even at L123 , median purity under WO Format is only 0.53. HERMES’s geometry-derived hierarchy captures topical content better than document format, consistent with the input embedding being a sentence-level semantic encoder rather than a layout/format model. A format-aware encoder is a natural future direction.
NMI Purity H(ref|HERMES)
L1
WO Topic 0.400 WO Format 0.128 ToS 0.334
0.63 0.28 0.53
1.94 3.10 2.29
L12
WO Topic 0.358 WO Format 0.235 ToS 0.329
0.83 0.50 0.71
0.90 1.62 1.14
L123
WO Topic 0.360 WO Format 0.240 ToS 0.331
0.89 0.53 0.75
0.87 1.57 1.09
What this licenses. HERMES at L12 and L123 can be read as a higher-resolution topical labeling, with the resolution exposed as a prefix length rather than rebuilt as a new clustering or taxonomy. Importantly, the moderate NMI with each of WO Topic, WO Format, and ToS at L1 shows that HERMES does not duplicate any one reference taxonomy; the hierarchy adds resolution rather than re-deriving a fixed label system.
with HERMES granularity against every reference family: WO Topic 0.63 → 0.83 → 0.89, ToS 0.53 → 0.71 → 0.75, WO Format 0.28 → 0.50 → 0.53. Conditional entropy of WO Topic given HERMES drops from 1.94 bits at L1 to 0.87 bits at L123 , an 80% reduction relative to the 4.58bit prior. At L12 /L123 a HERMES bucket therefore behaves as a refined topical cluster: it does not name a single human-readable topic, but the documents inside it share one to a substantially greater
N
Qualitative Label Inspection
This appendix complements the quantitative labelrelationship analysis (Appendix M) and the sparsity audit (Appendix F) with qualitative evidence that HERMES codes carry interpretable semantic content at every prefix level. All n-gram signatures below come from a uniformly spaced sample of 4,800,000 documents (160 aligned text/annotation shards × 30,000 rows per shard); full-corpus doc17
ument counts come from the full 227M-document annotation audit. For each L1 bucket, distinctive unigrams and bigrams were computed by ranking corpus-internal frequency against the corpus background. All 256 L1 buckets are populated in the sample (per-bucket sample size 3,481 to 52,912, median 18,160). N.1
Representative L1 buckets
N.2
Prefix hierarchy examples
To check that prefix length acts as a semantic granularity control rather than a numeric handle, we selected four high-population L1 parents and inspected their largest L12 children, then drilled into selected L123 descendants. Parent and child summaries are author-assigned from the n-gram signatures. L123 drill-down. A selected L123 inspection is consistent with the sparsity audit (Appendix F): where the third RVQ stage is populated, it surfaces local refinements rather than a dense thirdlevel tree. For example, the hip-hop / pop-artist sub-cluster 24_12 splits into 24_12_250 (popcaan, drizzy, drake, ras cal) and 24_12_95 (cudi, chioma, quavo, archuleta), and the art-education sub-cluster 218_199 splits into 218_199_127 (art students, art academy, teaching art) and 218_199_16 (art students, art teachers, scholastic art). Scope and caveat. These examples are qualitative and selected for readability; they support the interpretation of HERMES codes as a prefix-readable hierarchical label substrate, and are consistent with the quantitative purity climb (Appendix M) and the parent-child refinement statistics (Appendix F). They are not a downstream-performance result, and not every bucket is equally clean: e.g., L1 buckets sitting near format/quality boundaries (OCR fragments, navigation pages) are not represented in this selection but appear in the full 256-row table.
18
Table 11: Representative HERMES L1 bucket signatures from a 4.8M-document sample. The full 256-row table is released in the supplementary material. The examples show that the learned labels recover coherent topical and stylistic regions without using hand-written taxonomies; per-bucket interpretable labels are author-assigned summaries of the n-gram signatures, not learned outputs. L1 cell
Interpretable label
Full-corpus docs
Sample docs
179
Books, publishing, fiction
2,525,694
52,912
24
Music releases and reviews
2,102,386
45,359
218
Visual art and exhibitions
1,797,386
37,901
11
Software and developer tooling
1,725,904
28,991
6
Recipes and cooking
1,280,176
26,562
4
Biomedicine, molecular biology
1,255,632
25,967
8
Video games and RPGs
1,050,568
21,781
116
Astronomy and space science
890,156
17,993
10
Macroeconomics and markets
885,835
19,210
97
Wine and winemaking
617,996
12,830
76
Eye care and ophthalmology
431,300
9,033
254
North Korea / geopolitics
157,314
3,481
Distinctive n-grams (top 6) contemporary romance, cover reveal, netgalley, urban fantasy, debut novel, hardcover paperback album review, second album, progressive rock, released album, title track, debut single solo exhibition, museum contemporary, artist statement, painting sculpture, art practice, art fair version control, sqlite, dbforge, configuration file, unit tests, jdbc marinade, cook minutes, finely chopped, pepper taste, cloves garlic, saute gene expression, assays, crispr, gene therapy, kinase, neuronal azeroth, guild wars, npcs, edh, pve, clash royale astronomers, nasa’s, hubble, space agency, black holes, cassini gdp growth, bernanke, bull market, yellen, yield curve, fomc winemaker, winemaking, cabernet sauvignon, tannins, sauvignon blanc, riesling lasik, cornea, optometry, macular, contact lens, cataracts pyongyang, kim jong, korean peninsula, jong-un, kim jong-un, korean leader
Table 12: Representative HERMES prefix hierarchy. For each of four high-population L1 parents we show its five largest L12 children. “Active L12” counts L12 sub-buckets that received any documents under the parent; child summaries are author-assigned from n-gram signatures on the 4.8M-document sample. L1 parent
179: Books / publishing
24: Music
218: Visual art
11: Software / dev tools
Full docs
2,525,694
2,102,386
1,797,386
1,725,904
Active L12
L12 child
Summary
256
179_97 179_176 179_255 179_148 179_133
writing/novel advice children’s books romance sci-fi/fantasy classic/animal literature
256
24_18 24_65 24_150 24_12 24_183
classic rock music videos MP3/download pages hip-hop / pop artists Christian / gospel music
255
218_9 218_199 218_18 218_243 218_120
general art discourse art education modern artists Renaissance / Italian art Islamic / Middle Eastern art
256
11_235 11_150 11_226 11_123 11_142
Azure / ASP.NET / API docs file/PDF tooling data/table tooling algebra/math software unit and functional testing
19