arXiv CS open access

Adaptive Frequency Threshold Byte Pair Encoding for Mixed Natural Language and Structured Symbolic Corpora

Pietro Di Nicco et al. · arxiv_cs

arXiv CS · Papers

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Adaptive Frequency Threshold Byte Pair Encoding for Mixed Natural Language and Structured Symbolic Corpora Pietro Di Nicco, Lucio La Cava, Andrea Tagarelli DIMES Department, University of Calabria, Italy {lucio.lacava, tagarelli}@dimes.unical.it

Abstract

arXiv:2610.32921v1 [cs.CL] 23 Oct 2026

Modern language models rely on subwording tokenization to define the discrete units of input processing, typically using methods like Byte Pair Encoding (BPE) that treat whitespace as a fixed semantic boundary. This design is inherently limited, as many meaningful concepts—including multiword expressions and the structured environments of formal languages like LATEX—frequently span across whitespaces. While recent superword tokenization methods attempt to address this by relaxing whitespace constraints, they often rely on greedy frequency-driven merging that fails to account for token length or the specific needs of underrepresented domain-specific languages. In this paper, we introduce Adaptive Frequency Threshold Byte Pair Encoding (AdaBPE), a new tokenization method that leverages an adaptive frequent-pattern mining approach. AdaBPE features a twofold dynamicity: it adapts minimum frequency thresholds based on both the length of a candidate token, to prevent favoring frequent short patterns, and its type, to distinguish between general and formal language patterns, using LATEX as case in point. This allows the model to efficiently allocate vocabulary capacity to domain-specific notations, such as LATEX, even when they represent a small fraction of the training data. We evaluate AdaBPE across five datasets spanning LATEX and natural language, showing consistent gains in encoding efficiency and downstream language-model performance on domain-specific content without compromising general-language performance.

Introduction

In modern language models, the goal of tokenization is to construct a vocabulary of atomic units, called tokens, from a large text corpus. This vocabulary defines the discrete symbols that the model can process as input. The main approach for learning such a vocabulary is subword tokenization. Widely used algorithms, e.g., Byte Pair Encoding (BPE)

(Gage, 1994), WordPiece (Schuster and Nakajima, 2012), and Unigram (Kudo, 2018), learn a vocabulary of subword units from the training corpus. These methods typically treat whitespace as a semantic boundary between words, using it to constrain how text is segmented into subword tokens. This design however has an inherent limitation: whitespace is not, in general, a reliable indicator of semantic boundaries. Many concepts are expressed as multiword expressions, such as “Artificial Intelligence” or “state of the art”, where semantically meaningful units often span multiple whitespace-separated tokens. Ideally, such expressions should be represented as a single lexical unit during tokenization. A similar issue arises in formal languages including programming languages and markup, such as HTML, Markdown, or LATEX. For example, the expression \begin{equation} . . . \end{equation} denotes a single mathematical environment, yet conventional subword tokenizers process each command, delimiter, and surrounding text separately. Related work. Recent studies have revisited the objectives of conventional subword tokenization approaches, highlighting their inherent limitations. Lim et al. (2025) frame subword tokenization as an NP-hard problem, and propose a greedy alternative called GreedTok. Similarly, Asgari et al. (2026) propose MorphBPE to constrain BPE merge operations to respect morphene boundaries, improving morphological consistency. Phan et al. (2024) relate subword tokenization to a bias phenomenon, while (Zhang et al., 2025) investigated negative impacts on the LLM performance on reasoning tasks. A complementary body of research has focused on vocabulary design. Tao et al. (2024) analyze the compute-optimal vocabulary size growing with the model scale, whereas Lai et al. (2021); Kumar and Thawani (2022); Zhang et al. (2021) explore incorporating larger linguistic units into the

vocabulary; however, these approaches are limited to specific tasks (e.g., machine translation) or languages. More recently, a new trend of “superword” tokenization has emerged. Particularly, SuperBPE (Liu et al., 2025) and BoundlessBPE (Schmidt et al., 2025) adopt a variant of BPE that, after initialization like standard BPE, relaxes the pre-tokenization constraint allowing merge candidates that cross whitespace boundaries. The obtained results highlight the benefits of multiword tokens in improving compression, reducing training costs, and accelerating inference. Motivations of this study. However, two further issues remain largely unaddressed, which are discussed as follows. (I1) Although methods like SuperBPE relaxes the constraint that token merges must respect whitespace boundaries, their vocabulary induction procedure remains based on greedy frequency-driven merging. This greedy merge criterion is based primarily on the frequency of adjacent token pairs rather than directly optimizing the reduction in encoding length. Consequently, it does not explicitly account for the length of candidate tokens or the compression benefit obtained from introducing longer tokens, potentially favoring frequent short patterns over less frequent candidates that could yield larger reductions in sequence length. (I2) The above issue is further exacerbated for domain-specific languages that, in the training data of modern LLMs, is relatively less represented than other sublanguages or formal languages. A particularly relevant example is the representation of mathematical notation through LATEX, which has been shown to improve the reasoning ability of LLMs on mathematical tasks (Taylor et al., 2022), (Lewkowycz et al., 2022), (Ahn et al., 2024). Unlike lightweight markup such as Markdown, LATEX encodes mathematical content through a rich hierarchy of commands and environments which preserves syntactic and semantic structure that is essential for mathematical reasoning, suggesting that LATEX constitutes a more informative training signal for mathematical language modeling than markup languages designed primarily for document formatting. LATEX commands and environments define structured semantic units that are largely independent of whitespace, highlighting the mismatch between whitespace-based tokenization and the underlying semantic structure of the document. Contributions. In this paper, we aim to address the above issues (I1-I2) by introducing a new tokeniza-

tion method, named AdaBPE, which leverages an adaptive frequent-pattern mining approach to identify multiword expressions and build an enhanced BPE-based vocabulary. AdaBPE is designed to explicitly distinguish general language patterns from formal language patterns, with emphasis on LATEX, in inducing supertokens. The key feature of AdaBPE is a twofold dynamicity in determining token merges, since it adapts the minimum frequency threshold of a token pattern to both its length (cf. I1) and its type—thus differently treating frequent multiword expressions belonging to either LATEX or general language (cf. I2). Yet, AdaBPE provides a single vocabulary-construction procedure, which allocates capacity differently to general natural language and to domain-specific notation, without requiring separate vocabularies. This leads to an efficient mechanism for capturing domain-specific languages like LATEX even if these languages account for only a small fraction of the training data. Across five datasets spanning both LATEX and general-language domains, we compared AdaBPE against SuperBPE and BPE along four complementary axes—encoding efficiency, supertoken contribution, token-distribution statistics—showing that AdaBPE achieves consistent compression gains on domain-specific content without degrading performance on general language. Moreover, we trained small-scale language models using AdaBPE and SuperBPE tokenizers, and evaluated these models on five downstream benchmarks to assess the impact of AdaBPE’s tokenization strategy on downstream performance.

Methodology

2.1

Preliminaries

BPE (Gage, 1994) constructs a vocabulary of subword units by iteratively merging the most frequent pair of adjacent tokens in a training corpus. Canonically, BPE is applied after whitespace pre-tokenization, restricting merges to occur only within whitespace-delimited words. This design prevents learned units from spanning word boundaries, making BPE effective for capturing morphological structure while limiting its ability to represent frequent multiword expressions as single lexical or semantic units. SuperBPE (Liu et al., 2025) addresses the BPE limitation by introducing a two-stage training procedure: Stage 1 (Subword Learning): The tokenizer performs standard BPE training up to a transition

Algorithm 1 Adaptive Frequency Threshold BPE (AdaBPE) 1: Input: corpus C, BPE vocabulary V over C, global threshmin old min_thr ∈ [0, 1], pattern-specific params. θgeneral min and θlatex , and max iterations I 2: i ← 0 3: while V has changed and i < I do 4: Ci ← tokenize(C, V ) 5: B ← extract_bigrams(Ci ) 6: for all b ∈ B do 7: if b contains LaTeX or alphanumeric patterns then min log2 (nchars(b)) 8: minlatex ← min_thr · (θlatex ) b latex 9: if minb ≤ sup(b, Ci ) then 10: V ← V ∪ {b} 11: end if {general language} 12: else min 13: mingeneral ← min_thr · (θgeneral )nspaces(b)+1 b general 14: if minb ≤ sup(b, Ci ) then 15: V ← V ∪ {b} 16: end if 17: end if 18: end for 19: i←i+1 20: end while

point, using whitespace pre-tokenization to establish a base vocabulary of subwords. Stage 2 (Supertoken Learning): Training resumes from the vocabulary learned in Stage 1, but with whitespace pre-tokenization disabled. By allowing merges to bridge whitespace boundaries, the algorithm encodes common multiword sequences, or “supertokens” into the final vocabulary, enhancing tokenization efficiency for frequently occurring patterns. 2.2

Adaptive Frequency Threshold BPE

Algorithm 1 provides an outline of AdaBPE, which takes as input a training corpus C, a BPE vocabulary V , a global threshold min_thr, and a maximum number of iterations I. The key hyperparameter is the threshold min_thr, which sets the baseline token-selectivity of AdaBPE, and is in turn modulated by two additional, pattern-level languagemin dependent hyperparameters, dubbed θlatex and min θgeneral , which serve to adapt the effective threshold to the type of pattern under consideration during the tokenization process. At each iteration, the corpus C is tokenized based on whitespace delimitation of currently learned (super)tokens (Line 4). The vocabulary V is initialized with the BPE vocabulary and progressively expanded by adding bigrams identified in previous iterations through frequent-pattern mining. For each candidate pattern (bigram) b (Lines 5-6), the algorithm decides whether b should be

added to V by comparing its support sup(b, Ci ) = count(b, Ci )/|Ci | against a frequency threshold, where count(b, Ci ) is the number of occurrences of b in Ci , and |Ci | is the total number of patterns (i.e., tokens) in the corpus tokenized at iteration i. If b contains LATEX content (Lines 7-11), it is added to the vocabulary whenever sup(b) ≥ minlatex , b latex where minb is obtained by scaling min_thr min . Otherwith the LATEX-specific parameter θlatex wise, if b corresponds to general language (Lines 12-18), its support is compared to a threshold general minb , which is computed using a generalmin language parameter θgeneral . general

In determining both minb and minlatex , b the length of pattern b is taken into account to conmin min , trol an exponential scaling of θgeneral and θlatex respectively; for natural language, we consider the number of whitespace-separated words as the length of the pattern, whereas for LATEX, we consider the base-2 logarithm of the number of characters in the pattern. Remarks on frequent pattern mining. Since V is updated at every iteration, AdaBPE induces an exponential supertoken composition mechanism: at the first iteration, only bigrams derived from BPE tokens are considered; at the second iteration, the bigrams produced in the previous step(s) may themselves be combined, yielding 3-grams and 4-grams w.r.t. the original BPE vocabulary; and so on for longer patterns at subsequent iterations. In practice, a very small number of iterations is required to converge—three iterations are sufficient to reach patterns containing up to eight BPE tokens. It should also be noted that, fFor any candidate pattern b at iteration i − 1 that still is at the current iteration i, both count(b, Ci ) and its associated minb threshold do not change, however its support is non-decreased since during AdaBPE training, the number of tokens in the corpus decreases as merges are applied. Consequently, a pattern b could pass the frequency check at subsequent iterations than its first identification. Nonetheless, this does not impact the overall time cost of AdaBPE which is bounded by O(I · |C0 |), i.e., AdaBPE scales linearly with the size of the training corpus expressed in terms of total number of tokens in the corpus when represented using the initial BPE vocabulary. 2.2.1

Setting min_thr.

To determine the value of the initial threshold min_thr, we reason about the size of the final vo-

Figure 1: Bigram frequency distribution estimated from a random sample of size 1.5 · 108 .

cabulary. We set the final vocabulary size to around 200k tokens, so as to match the one of SuperBPE. We adopt this value primarily to ensure a controlled comparison across BPE, SuperBPE and AdaBPE, thus supporting our goal of isolating the effect of the tokenization method. Like in SuperBPE, we set the size of the starting BPE vocabulary to 160k tokens. Since our target final vocabulary is of the order of 200k, AdaBPE should add on the order of 104 tokens at each iteration. During the first iteration, the algorithm explores the bigram distribution, which is known to follow Zipf’s law (Kluckhohn, 1950), (Le et al., 2009). We set min_thr by reasoning about this first iteration, (Empirically, we have observed that the number of patterns added at each subsequent iteration is of the same order of magnitude as the first one). Crucially, min_thr represents an upper bound on the threshold effectively applied to each pattern, since the language-dependent min min (ranging within (0, 1)) parameters θgeneral , θlatex further reduce it. We hence choose min_thr so that only about 103 bigrams have relative support above min_thr in the first iteration; the subsequent downward scaling by the language-dependent parameters then raises the number of admitted patterns to the order of 104 . Figure 1 shows the empirical bigram frequency distribution, estimated from a random sample of 1.5 · 108 bigrams drawn from the training data (see Appendix B). As shown, min_thr = 10−4 is an appropriate choice, since the number of patterns considered in the first iteration remains on the order of 103 , before the scaling introduced by the language-dependent parameters. 2.2.2

min min . Setting θgeneral and θlatex

min min are As previously mentioned, θgeneral and θlatex language-dependent parameters used to scale the frequency threshold according to the type of pattern. Since LATEX is typically relatively less represented in the training data, the scaling to LATEX patterns

should be more aggressive, therefore we set the min min . Following the same constraint θgeneral > θlatex approach as for the global threshold, we examined the frequency distributions of natural-language bigrams and of LATEX bigrams (see Figure 2). To match the conditions set for the final vocabulary size, in the first iteration the two parameters are bounded based on the number of patterns admitted in the first iteration, i.e., growing the starting 160k BPE vocabulary to the target size of 200k requires adding 200k − 160k = 4 · 104 tokens in total. We therefore take 4 · 104 as the upper bound on the number of patterns admitted for a given language type in the first iteration—admitting more would already exceed, the entire vocabulary budget—and set the lower bound one order of magnitude smaller, at 4 · 103 , so that each language regime contributes a non-negligible number of patterns while leaving headroom for the subsequent iterations: sup b4·104

min 2 < min_thr · θgeneral < sup b4·103

min nchars(b) sup b4·104 < min_thr· θlatex < sup b4·103

where sup(bx ) denotes the support of the pattern with rank x in the general-language, resp. LATEX, distribution. Since min_thr = 10−4 and spotting sup(bx ) from the two empirical distributions (Figure 2), we derive a lower bound and an upper bound for each language-dependent parameter: min 2 < 3 · 10−5 3 · 10−6 < 10−4 · θgeneral

min 1.8 4 · 10−8 < 10−4 · θlatex < 5 · 10−7

where the exponent 1.8 for LATEX patterns is estimated as log2 of the average pattern length in the LATEX portion of the training distribution. Solving these inequalities yields min 0.2 < θgeneral < 0.5,

min 0.02 < θlatex < 0.06.

Remarks on hyperparameter tuning. We emphasize that the above settings are not meant as hard constraints but as a principled way to narrow the search to the correct order of magnitude: they identify the region in which the number of admitted patterns is compatible with the target vocabulary size. Moreover, while our AdaBPE hyperparameter thresholds have been determined through a data-driven procedure on a representative training sample, these capture general distributional patterns rather than corpus-specific heuristics. This makes the proposed approach robust, transferable, and applicable across diverse datasets.

pretraining corpus (olmo-mix) (OLMo et al., 2024), more details are given in Appendix B. To match the vocabulary size of BPE and SuperBPE (200k), we trained AdaBPE for three iterations (N = 3), with min min = 0.08. min_thr = 10−4 , θgeneral = 0.3, θlatex Starting from a BPE vocabulary of 160k tokens, AdaBPE reaches a final size of 204,601 tokens. Figure 2: Frequency distribution of natural-language bigrams (right) and of LATEX bigrams (left).

2.3

LATEX and Math Parsing

To decide whether a candidate pattern should be treated as LATEX/mathematical content, we test it against the following regular expression: (\\[a-zA-Z]+|[\^_{}]|[a-zA-Z]+[\d_]+|\d+ [a-zA-Z_]+). The expression is a disjunction of four cases, each capturing a syntactic building block of LATEX and mathematical notation. The first alternative, \\[a-zA-Z]+, matches LATEX control sequences, i.e. a backslash followed by a command name (e.g. \frac, \begin, \leftarrow). The second, [ˆ_{}], matches the structural operators of math mode: superscript (^), subscript (_), and the grouping braces ({, }). The last two, [a-zA-Z]+[\d_]+ and \d+[a-zA-Z_]+, match mixed alphanumeric tokens—letters followed by digits or underscores and vice versa—which are characteristic of indexed variables (e.g. x_, theta_), coefficients, and identifiers found in mathematical/programming notation. We deliberately favor this lightweight, purely lexical test over a full LATEX parser: the check is applied to every candidate bigram during the mining phase, which runs over a very large corpus, so it must be cheap to evaluate. The expression is not meant to validate syntactically correct LATEX, but only to flag—at scale—the presence of the structural markers that distinguish domain-specific notation from general language. Following prior work, we tokenize digits from right to left based on the regular expression (?=(\d{3})+(?!\d)), which has been shown to be a suited choice for supporting the arithmetic abilities of LLMs (Singh and Strouse, 2024).

Evaluation Methodology

3.1

Experimental Setup

We evaluated AdaBPE against two popular approaches: BPE and SuperBPE. To ensure a fair comparison, all three tokenizers are trained on the same data, which correspond to the SuperBPE training data: the 10 GB subset of the OLMo-2 stage-2

3.2

Evaluation Data

Our evaluation spans five datasets covering the two settings targeted by our method, namely the technical and general language. latex-formulas (Oleehy, 2024) and Image2Struct (Roberts et al., 2024) are selected to provide a focused test of whether our approach yields the expected benefits on LATEX based technical documents; specifically, latex-formulas consists of mathematical expressions written in LATEX, and Image2Struct contains LATEX code extracted from arXiv papers. Other three datasets—DROP (Dua et al., 2019), SQuAD (Rajpurkar et al., 2016), and CommonsenseQA (CSQA) (Talmor et al., 2019)—are general naturallanguage benchmarks, included to verify that this specialization does not come at the cost of compression on text-heavy documents. Summary statistics are reported in Appendix 7. 3.3 Evaluation Goals and Assessment Criteria Our evaluation is guided by a single overarching question: does AdaBPE achieve its design goal of better representing under-represented domainspecific languages— LATEX—without degrading its treatment of general natural language? We address this question through an extensive evaluation that combines qualitative and quantitative analyses, comparing AdaBPE against BPE and SuperBPE. Qualitative analysis. We start by qualitatively examining the merge patterns learned by each tokenizer. We carry out two complementary analyses: a side-by-side inspection of how the tokenizers segment representative sentences, and a comparison of the most frequent tokens that are exclusive to each tokenizer on every evaluation dataset. Our quantitative analysis is based on three complementary dimensions: (i) encoding efficiency, (ii) contribution of supertokens, and (iii) statistical properties of the token distribution. We next provide intuitive explanations of these criteria; their formal definitions are deferred to Appendix A. Encoding efficiency. The most direct measure of tokenizer quality is how compactly it encodes text.

We assess this using the Average Encoding Length (AEL), defined as the mean number of tokens per document in the corpus. Lower AEL values indicate shorter tokenized sequences, reducing the computational and inference costs of LLMs.

Dataset

Supertoken contribution. To characterize the role played by the supertokens, we introduce three measures: the Supertoken Share, i.e., the fraction of emitted token stream made up of supertokens; the Mean Savings, i.e., the expected number of base BPE tokens collapsed into a single supertoken; and the Total Savings, which combines the two into an overall measure of the compression attributable to supertokens. Together these metrics indicate how heavily each tokenizer relies on supertokens.

avg decr. w.r.t. BPE

Token distribution. Beyond raw compression, the statistical shape of the token distribution is known to correlate with the quality of a tokenizer for downstream language modeling (Zouhar et al., 2023). Thus, we measure two criteria: N-gram diversity, i.e., the fraction of unique n-grams in the encoded text, and Rényi efficiency (Zouhar et al., 2023), which summarizes how balanced the tokenfrequency distribution is, where higher values indicate a more balanced token-frequency distribution. Downstream task validation. We also assessed the impact of AdaBPE vs. SuperBPE downstream behavior. To this aim, we trained small-scale models on top of AdaBPE and SuperBPE tokenizers, using the full 100GB subset released by (Liu et al., 2025) as the training data (cf. Appendix B) and OLMo-2 (OLMo et al., 2024) as the decoder architecture. This choice is motivated since OLMo-2 is a fully open-source model with publicly released training data and code, making it well suited for controlled tokenizer comparisons. The two resulting OLMo-2 models, dubbed OLMo2-AdaBPE and OLMo2-SuperBPE were trained under identical architecture and budget, differing only in the tokenizer (cf. Appendix E for hyperparameters). We resorted to five evaluation benchmarks: MATH500 (Hendrycks et al., 2021), BoolQ (Clark et al., 2019), Winogrande (ai2, 2019), CSQA (Talmor et al., 2019) and GSM8K (Cobbe et al., 2021). We measured the benchmark accuracy and the average bits-per-byte (BPB) (Gao et al., 2020); unlike perplexity or other token-level measures, BPB normalizes the model’s cross-entropy loss by the number of bytes of the original text, making it directly comparable across tokenizers with different vocabularies and token lengths (see Appendix A).

CSQA latex-formulas SQuAD DROP Image2Struct

BPE

SuperBPE

AdaBPE

16.63 12.43 (25.3%) 12.24 (26.4%) 68.58 59.49 (13.3%) 54.88 (20.0%) 163.63 125.11 (23.5%) 126.03 (23.0%) 264.26 206.78 (21.7%) 210.43 (20.4%) 247.32 212.43 (14.1%) 207.96 (15.9%) –

19.58%

21.14%

Table 1: Average Encoding Length (AEL) comparison between BPE, SuperBPE, and AdaBPE. Numbers in parentheses indicate the relative reduction with respect to the BPE baseline. Best results are bolded.

Results

4.1

Qualitative Analysis

Here, we summarize the main observations and representative examples regarding our qualitative analysis on (i) how each tokenizer segments sentences containing common multiword expressions and (ii) the sets of frequent tokens that are exclusive to each tokenizer across the evaluation datasets. Segmentation examples are reported in Appendix C and the exclusive-token tables in Appendix D. Two consistent patterns emerge across the analysis. First, AdaBPE preserves LATEX command structures more effectively than SuperBPE. For instance, the expressions \frac and \begin{equation} are split by SuperBPE into two tokens while AdaBPE represents them as single tokens, and so it does even with more complex expressions such as \sum_{i=. Our exclusive-token analysis (results reported in Appendix D, Tables 8–12) confirms this at the vocabulary level: AdaBPE’s most frequent tokens on latex-formulas and Image2Struct include full command structures such as \begin{align*}, \end{align*}, \begin{equation*}, whereas SuperBPE’s exclusive tokens on the same datasets are dominated by short, isolated symbol pairs (left(, \right, (t, (z). Second, on natural-language text, AdaBPE reveals to be much more effective than SuperBPE in preserving multiword expressions as single, indivisible units, retaining the semantic and functional integrity of these expressions. Surprisingly, this holds not only for relatively frequent and highly lexicalized terms, like "state of the art" or "artificial intelligence", but also for rarer, more compositional-looking or extended prepositional phrases, such as "late at night", "in the ground", "at the end of", "on the side of the road", which are not captured by SuperBPE. The exclusive-token results (Appendix D) show that this is a systematic, dataset-wide

Table 2: Average tokenization time (in seconds).

phenomenon. Thus, AdaBPE improves upon SuperBPE in avoiding the loss of idiomatic or collocational meaning that arises from purely compositional tokenization, yielding more efficient, semantically coherent input units, with potential advantages for downstream language understanding and generation tasks (cf. Section 6). 4.2

Average Encoding Length

As shown in Table 1, AdaBPE consistently achieves the lowest AEL on latex-domain datasets (i.e., latex-formulas and Image2Struct), with a relative length reduction up to 7% vs. SuperBPE. Also, on natural language benchmarks, AdaBPE performs competitively with SuperBPE, showing that it successfully retains the compression advantages of the SuperBPE baseline while offering superior specialization for mathematical notation. Time Efficiency. Table 2 reports the average tokenization time per instance for the tokenizers across datasets.1 SuperBPE and AdaBPE consistently achieve lower tokenization times than BPE, indicating that the learned supertokens reduce the number of tokenization operations required at inference time. The largest absolute improvements are on the longest and most token-intensive datasets, i.e., latex-formulas and SQuAD, where tokenization time is reduced by roughly 25% and 8%, respectively. The differences between SuperBPE and AdaBPE are negligible overall, with AdaBPE exhibiting a slight advantage on the technical datasets and SuperBPE performing marginally better on the general-language datasets. This comparable efficiency confirms that the AdaBPE’s dynamicfrequency-aware tokenization approach affects the learned vocabulary but substantially not the computational complexity of the tokenization algorithm. 4.3

Supertoken contribution

Table 3 compares the token savings achieved by SuperBPE and AdaBPE across the evaluation datasets. AdaBPE consistently outperforms SuperBPE on all 1 Measured on a machine with Ubuntu, Intel(R) Xeon(R) CPU @ 2.20GHz, 30GB RAM. AdaBPE is implemented in Python using the PySpark library.

Tokenizer Super Share Mean Savings Total Savings latex

1.77 25.49 68.77 8.32 0.42

SuperBPE AdaBPE ∆ (%)

0.133 0.272 +104.5%

1.169 2.094 +79.1%

0.156 0.569 +264.7%

I2S

AdaBPE

1.68 25.52 68.54 8.27 0.44

SuperBPE AdaBPE ∆ (%)

0.147 0.366 +149.0%

1.134 2.049 +80.7%

0.167 0.750 +349.1%

DROP

SuperBPE

1.81 34.22 74.41 10.88 0.63

SuperBPE AdaBPE ∆ (%)

0.229 0.715 +212.2%

1.226 2.836 +131.3%

0.281 2.029 +622.1%

SQuAD

BPE

CSQA latex-formulas SQuAD DROP Image2Struct

SuperBPE AdaBPE ∆ (%)

0.257 0.822 +219.8%

1.209 2.947 +143.8%

0.311 2.424 +679.4%

CSQA

Dataset

SuperBPE AdaBPE ∆ (%)

0.281 0.836 +197.5%

1.206 3.150 +161.2%

0.339 2.634 +677.0%

Table 3: Comparison of AdaBPE and SuperBPE in terms of Supertoken Share, Mean Savings, and Total Savings. Best results per dataset are in bold.

three criteria, demonstrating the effectiveness of our proposed dynamic-frequency-aware tokenization strategy. In particular, AdaBPE substantially increases the share of text represented by supertokens, with improvements ranging from about 105% on latex-formulas to over 200% on DROP and SQuAD. This larger supertoken utilization translates into considerably higher average and total token savings. The improvements are especially pronounced for the most token-intensive datasets, where AdaBPE achieves up to 161% higher mean savings and over 670% higher total savings compared with SuperBPE. These results indicate that AdaBPE is able to identify and exploit more effective multi-token patterns, leading to a more compact representation of the input text. Notably, the gains are consistent across both technical datasets (e.g., latex-formulas and Image2Struct) and general-language benchmarks (e.g., DROP, SQuAD, and CSQA), suggesting that AdaBPE provides a general improvement over existing supertokenization approach. 4.4

Token distribution analysis

Table 4 shows that AdaBPE achieves significantly higher lexical diversity than the BPE baseline, while remaining comparable to SuperBPE across all datasets. Moreover, both SuperBPE and AdaBPE improve over BPE in distributional balance, with SuperBPE retaining a small but consistent edge over AdaBPE. These results are in relation with those from the supertoken analysis of Table 3: AdaBPE exhibits a much higher supertoken share on these datasets, meaning it encodes a larger portion of a text with frequently recurring superto-

Dataset

BPE SuperBPE AdaBPE NDiv RE1 RE2 NDiv RE1 RE2 NDiv RE1 RE2

CSQA latex SQuAD DROP I2S

2.05 0.48 0.49 0.17 0.98

0.70 0.50 0.59 0.48 0.66 0.41 0.67 0.45 0.67 0.51

2.72 0.72 0.67 0.23 1.26

0.80 0.51 0.63 0.52 0.80 0.57 0.78 0.59 0.72 0.57

2.67 0.74 0.66 0.22 1.23

0.80 0.50 0.65 0.52 0.80 0.56 0.77 0.57 0.72 0.53

Table 4: Comparison over N-gram diversity (NDiv) and Rényi efficiency (REα , with α = {1,2}).

kens. Because these units are, by construction, high-frequency patterns, they recur more often in the token stream, which marginally lowers the proportion of unique n-grams. In other words, AdaBPE trades a small amount of n-gram diversity and token-frequency balance for a substantially higher compression of recurrent expressions.

Sensitivity Analysis

We study the sensitivity of AdaBPE to the min two language-dependent parameters θgeneral and min θlatex , as shown in Table 13 in Appendix E. Several observations follow. First, the parameters trade off vocabulary size against compression: as the thresholds decrease, more supertokens are added and the AEL drops, but the vocabulary grows rapidly — from 168k tokens in the most restrictive setting to more than 1.6M tokens in the most permissive one. Second, the AEL on natural-language datasets (DROP, SQuAD, CSQA) is governed almin most exclusively by θgeneral : rows with the same min min yield value of θgeneral but different values of θlatex identical AEL on these three datasets. Symmetmin rically, θlatex controls the AEL on LATEX-heavy datasets (latex-formulas, Image2Struct). This confirms the design intent of the algorithm, which is to decouple the two language regimes. Finally, very aggressive settings (last row) achieve large AEL reductions but require vocabularies an order of magnitude larger than the baseline, which is unlikely to be practical in real training scenarios.

Downstream validation

Table 5 summarizes our main results on the impact of AdaBPE and SuperBPE on downstream performance. Looking at the general-language benchmarks (i.e., BoolQ, CSQA and Winogrande), OLMo2-AdaBPE always achieves higher benchmark-accuracy than OLMo2-SuperBPE, while incurring a slight BPB penalty on two of the three datasets. Results indicate that AdaBPE’s tokenization does not come at the cost of downstream task performance on general natural language. On

Dataset

OLMo2-AdaBPE Acc. BPB

GSM8K MATH500 BoolQ CSQA Winogrande

na na 0.6171 0.1966 0.5043

1.51 1.21 1.84 8.27 0.56

OLMo2-SuperBPE Acc. BPB na na 0.6150 0.1949 0.4972

1.56 1.27 1.44 7.30 0.58

Table 5: Downstream task results.

GSM8K and MATH500, both models exhibit a floor effect in accuracy due to their limited scale, but AdaBPE achieves a clear BPB advantage. Note that downstream performance was evaluated with greedy decoding, so results are not influenced by stochastic sampling at inference time. It should also be emphasized that, for OLMo2AdaBPE, we adopt the AdaBPE configuration that was selected for its encoding efficiency as found min = 0.05, in our sensitivity analysis (i.e., θlatex min θgeneral = 0.25, along with min_thr = 10−4 and I = 3); however, this setting is not necessarily optimal for downstream performance. Nonetheless, the presented downstream validation analysis has unveiled that the encoding efficiency advantages brought from AdaBPE translate to comparable or improved downstream behavior w.r.t. SuperBPE.

Conclusions

We present AdaBPE, an adaptive tokenization method that extends BPE through frequent-pattern mining to identify multiword expressions and construct an enhanced vocabulary. AdaBPE dynamically adjusts token merge criteria based on both pattern length and type, explicitly distinguishing general-language patterns from formal-language structures, with a focus on LATEX expressions. Our qualitative and quantitative evaluations, including comparison with SuperBPE, show that AdaBPE more effectively captures both LATEX commands and general-language multiword expressions as coherent units, preserving their semantic and functional integrity. It achieves up to 7% relative encoding reduction on mathematical LATEX content while maintaining competitive natural-language performance. Experiments with small decoder models further demonstrate improved accuracy and lower bits-per-byte on mathematical benchmarks, with comparable results on general-language tasks. We aim to investigate the extension of AdaBPE beyond LATEX to other formal (markup) languages, task-oriented hyperparameter optimization to improve downstream performance, and scalability of AdaBPE on larger models.

References 2019. Winogrande: An adversarial winograd schema challenge at scale. Janice Ahn, Rishu Verma, Renze Lou, Dingxiao Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. ArXiv, abs/2402.00157. Ehsaneddin Asgari, Yassine El Kheir, MohammadAli SadraeiJavaheri, and Ali Nazari. 2026. MorphBPE: Morphology-aware tokenization for efficient LLM training. In Findings of the Association for Computational Linguistics: ACL 2026, pages 41610–41621, San Diego, California, United States. Association for Computational Linguistics. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal archive, 12:23–38. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS. Clyde Kluckhohn. 1950. Human behavior and the principle of least effort. george kingsley zipf. American Anthropologist, 52(2):268–270. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. Dipesh Kumar and Avijit Thawani. 2022. BPE beyond word boundary: How NOT to use multi word expressions in neural machine translation. In Proceedings

of the Third Workshop on Insights from Negative Results in NLP, pages 172–179, Dublin, Ireland. Association for Computational Linguistics. Yuxuan Lai, Yijia Liu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2021. Lattice-BERT: Leveraging multi-granularity representations in Chinese pretrained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1716–1731, Online. Association for Computational Linguistics. Quan Ha Le, Philip Hanna, Ji Ming, and F. Smith. 2009. Extending zipf’s law to n-grams for large corpora. Artificial Intelligence Review, 32:101–113. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with language models. ArXiv, abs/2206.14858. Jia Peng Lim, Davin Choo, Shawn Tan, and Hady W. Lauw. 2025. A partition cover approach for tokenization. In Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025). Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A Smith, and Yejin Choi. 2025. SuperBPE: Space travel for language models. In Second Conference on Language Modeling. Oleehy. 2024. latex-formulas. https://huggingface. co/datasets/OleehyO/latex-formulas. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2024. 2 olmo 2 furious. ArXiv, abs/2501.00656. Buu Phan, Marton Havasi, Matthew Muckley, and Karen Ullrich. 2024. Understanding and mitigating tokenization bias in language models. ArXiv, abs/2406.16829. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. Josselin Somerville Roberts, Tony Lee, Chi Heem Wong, Michihiro Yasunaga, Yifan Mai, and Percy Liang. 2024. Image2struct: A benchmark for evaluating vision-language models in extracting structured information from images. Preprint, arXiv:TBD.

Craig W. Schmidt, Varshini Reddy, Chris Tanner, and Yuval Pinter. 2025. Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier. In Second Conference on Language Modeling. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE. Aaditya K. Singh and DJ Strouse. 2024. Tokenization counts: the impact of tokenization on arithmetic in frontier llms. ArXiv, abs/2402.14903. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. 2024. Scaling laws with vocabulary: Larger models deserve larger vocabularies. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony S. Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. ArXiv, abs/2211.09085. Xiang Zhang, Juntai Cao, Jiaqi Wei, Yiwei Xu, and Chenyu You. 2025. Tokenization constraints in llms: A study of symbolic and arithmetic reasoning limits. ArXiv, abs/2505.14178. Xinsong Zhang, Pengshuai Li, and Hang Li. 2021. AMBERT: A pre-trained language model with multigrained tokenization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 421–435, Online. Association for Computational Linguistics. Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for Computational Linguistics.

Appendix A: Evaluation Datasets We provide a brief description of the datasets used for the evaluation of AdaBPE and competing tokenizers, along with boxes illustrating example instances. Table 6 summarizes main statistics of the datasets. DROP (Dua et al., 2019) contains questions over passages that potentially require discrete reasoning over multiple pieces of information in the passage. DROP Example Instance Passage: To start the season, the Lions traveled south to Tampa, Florida to take on the Tampa Bay Buccaneers. The Lions scored first in the first quarter with a 23-yard field goal by Jason Hanson. The Buccaneers tied it up with a 38-yard field goal by Connor Barth, then took the lead when Aqib Talib intercepted a pass from Matthew Stafford and ran it in 28 yards. The Lions responded with a 28-yard field goal. In the second quarter, Detroit took the lead with a 36-yard touchdown catch by Calvin Johnson, and later added more points when Tony Scheffler caught an 11-yard TD pass. (Cont.) Question: How many points did the Buccaneers need to tie in the first? Answer spans: { }

"spans": ["3"], "types": ["number"]

Question: What sits on top of the Main Building at Notre Dame? Gold answer: “a golden statue of the Virgin Mary” answer_start: 92

Image2Struct (Roberts et al., 2024) contains LATEX code extracted from arXiv papers. For our experiments, we use the equation and algorithm subsets. Image2Struct Example Instance Text: \begin{algorithmic} \State $PA\_set$ \For { OC\_block in OC\_blocks} \State $PA\_block$ \For {$B_i \in$ OC\_block} ... (Cont.) ... \mathcal{F}_{a}^{PA}$ \EndFor \State $PA\_set \gets PA\_block$ \EndFor \end{algorithmic}

CommonsenseQA (CSQA) (Talmor et al., 2019) contains 5-way multiple-choice questions that require commonsense knowledge to answer. CSQA Example Instance

latex-formulas (Oleehy, 2024) consists of mathematical expressions written in LATEX. latex-formulas Example Instance \begin{align*} a^{\dagger}_m=\sqrt{\frac{2}{\zeta}}z_m\,, \quad a_m=\sqrt{\frac{2}{\zeta}}\bar{z}_m \quad\textrm{for}\quad m=1,\,3\,, \end{align*}

Question: The sanctions against the school were a punishing blow, and they seemed to what the efforts the school had made to change? Choices: A: ignore B: enforce C: authoritarian D: yell at E: avoid Answer: A

SQuAD (Rajpurkar et al., 2016) contains passages paired with questions about their content; the answer is always a contiguous span of the corresponding passage. SQuAD Example Instance Context: Architecturally, the school has a Catholic character. Atop the Main Building’s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend “Venite Ad Me Omnes”. Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. (Cont.)

Appendix B: Assessment Criteria Average Encoding Length. Given a corpus C and a tokenizer T , the Average Encoding Length (AEL) is the average number of tokens produced by T on instances of C: AEL(C, T ) =

1 X |T (s)|, |C|

(1)

s∈C

where T (s) denotes the set of tokens of text s. Supertoken contribution criteria. Let V be the vocabulary created by a tokenizer (i.e., AdaBPE

Statistic

DROP Image2Struct CSQA latex-forms. SQuAD

# Instances 9,535 Characters per instance Min 290 Max 5,578 Avg 1,111.27 Std 598.84 BPE tokens per instance Min 75 Max 1,290 Avg 264.26 Std 125.25 Words per instance(∗∗) Min 46 Max 957 Avg 186.38 Std 100.22

occurrence,

600 9,741

100,000(∗) 87,599

39 20 3,869 405 632.50 78.84 634.03 30.43

29 179 440 26,203 145.05 814.94 74.97 320.16

14 5 1,720 73 247.32 16.63 238.77 6.72

10 31 294 913 68.58 163.63 36.58 63.14

3. Total Savings (Savingstotal ): the average number of BPE tokens saved per emitted token over the entire corpus, which quantifies the overall compression gain,

2 4 536 66 64.42 14.60 68.10 5.61

1 25 136 666 9.72 129.82 8.76 49.55

Savingstotal (C, T ) = Sharesuper (C, T )·µsavings (C, T ). (4)

Alphanumeric BPE tokens per instance(∗∗∗) Min 3 0 0 Max 90 145 3 Avg 25.74 11.11 0.01 Std 14.86 15.61 0.11 LATEX BPE tokens per instance Min — 14 — Max — 1,720 — Avg — 247.32 — Std — 238.77 —

0 94 5.18 6.04

0 97 5.81 7.07

10 294 68.58 36.58

— — — —

Table 6: Summary statistics of the evaluation datasets. The full dataset contains approximately one million instances; for evaluation we selected the first 100,000. (∗∗) Words are regarded as sequences of characters separated by whitespace. (∗∗∗) BPE tokens containing at least one digit.

P µsavings (C, T ) =

t∈Vsuper count(t, T (C)) · (depth(t) − 1)

1. Supertoken Share (Sharesuper ): the fraction of the encoded token stream that consists of supertokens,

t∈Vsuper count(t, T (C))

(3)

N-gram Diversity Given a corpus C, a tokenizer T , and a maximum n-gram order N , the N-gram diversity is defined as the sum, over all n-gram orders from 1 to N , of the fraction of distinct n-grams among all n-grams extracted from the tokenized corpus:

(∗)

or SuperBPE, in our setting) built on top of an existing BPE vocabulary. We can partition V into the set of BPE tokens VBPE , which are the tokens inherited from the underlying BPE vocabulary, and the set of supertokens Vsuper . For any token t ∈ V and a corpus C, let count(t, T (C)) denote the number of occurrences of t in the encoded corpus T (C), and depth(t) the number of base BPE tokens constituting t. Note that any token t is a supertoken iff depth(t) > 1; also, a supertoken may itself be formed by merging two existing supertokens—as happens at later iterations of AdaBPE—but this does not affect the definition, which depends only on the full decomposition of t into BPE tokens. We define three metrics to evaluate the behavior of supertokens:

N gD(C, T, N ) =

N X #unique-n-grams ∈ T (C) n=1

(5)

#n-grams ∈ T (C)

In our experiments, N was set to 4. (C)) Rényi Efficiency Let p(t, T, C) = count(t,T be |T (C)| the relative frequency of token t in the tokenized corpus. The Rényi efficiency is defined as:

X ′ 1 RE(C, T, α) = lim ln p(t, T, C)α ′ ′ α →α (1 − α ) t

! , (6)

where α > 0 controls the sensitivity to the tail of the token-frequency distribution, and how strongly frequent tokens dominate the score. In particular: • α = 1: Taking the limit to 1 recovers theP standard Shannon-entropy-based measure, − t p(t, T, C) ln p(t, T, C) • α = 2: This gives more weight to the most frequent tokens via squared probabilities. Bits-per-byte (BPB) Given a tokenizer T and an input text sequence s, Bits-per-byte (BPB) is a standard criterion for evaluating language models and compressors, which is defined as: BP B(s, T ) =

LCE (T (s)) , ln(2) · nbytes (T (s))

(7)

P Sharesuper (C, T ) =

count(t, T (C)) t∈V P super . (2) count(t, T (C)) t∈V

2. Mean Savings (µsavings ): the expected number of base BPE tokens saved per supertoken

where T (s) is the set of tokens obtained by tokenizing the input text s, nbytes (T (s)) is the number of bytes in the original text represented by the token sequence, and LCE is the cross-entropy loss of T (s).

Appendix C: Training Data for the Tokenizers All tokenizers were trained on a 10GB subset of the OLMo2 stage-2 pretraining corpus (olmo-mix) (OLMo et al., 2024) publicly released by Liu et al. (2025). This 10GB subset is drawn from the full released data, which totals approximately 100GB; we used the smaller 10GB portion for tokenizer training as-is, without further filtering. As reported by Liu et al. (2025), a small amount of tokenizer training data is sufficient—enlarging it beyond this scale does not measurably affect the encoding efficiency of the resulting tokenizer—and the subset already incorporates their document-truncation preprocessing, which limits the influence of a few extremely long, internally duplicated documents. Using the same released data for BPE, SuperBPE, and AdaBPE ensures that any difference in the resulting tokenizers is due to the tokenization algorithm rather than from the training data, and makes our setup directly reproducible. min , and θ min , we To determine min_thr, θgeneral latex estimated the bigram frequency distributions of Figures 1 and 2 from a random sample of this 10GB corpus. Sampling was performed at the file level: files were drawn uniformly at random from the 10GB subset, and processed in this random order until the cumulative number of extracted bigrams reached 1.5 · 108 . Remarks on the selection of hyperparameters. The hyperparameters of our AdaBPE were determined through a data-driven procedure using a sample drawn from the same training distribution employed for SuperBPE, rather than being manually tuned for a specific downstream setting. This design choice ensures that the selected frequency thresholds reflect the statistical properties of the data while avoiding overfitting to the full training corpus. Since the procedure relies on identifying recurring patterns and adapting the thresholds accordingly, rather than on corpus-specific heuristics, we expect the resulting methodology to generalize to other datasets with different distributions and domains. In this sense, the proposed approach provides a robust and transferable strategy for configuring pattern-aware BPE variants.

known to be challenging for whitespace-based subword tokenization. The generated sentences were then encoded using the actual trained SuperBPE and AdaBPE tokenizers, along with basic BPE. The resulting token sequences are reported verbatim in the following listings, for each of the tokenizers. Underlined terms correspond to compound words or named entities that are expected to be encoded as single tokens. E XAMPLE 1 — \frac{n \sum_{i=1}^n i}{N} \rightarrow x BPE

Ġi }{\ N } Ġ\ rightarrow Ġx SuperBPE

AdaBPE

frac{

n n

sum

_{i

Ġi

}{\

rightarrow

Ġx

\frac{

\sum_{i=

}{\

n N

= }Ġ\

Ġi

Ġx

}Ġ\rightarrow

E XAMPLE 2 — \begin{equation}y = ax + b \end{equation} BPE

\ begin { equation } Ġy Ġ= Ġax Ġ+ Ġb Ġ\ end { equation }

superBPE

\begin{equation Ġax

Ġ+

equation AdaBPE

Ġb

}

Ġy

Ġ=

Ġb

Ġ\

end{

Ġ=

Ġax

}

\begin{equation} Ġ+

Ġy

Ġ\end{equation}

E XAMPLE 3 — The development of Artificial Intelligence requires vast amounts of data. BPE

The

Ġdevelopment

ĠIntelligence

Ġof

ĠArtificial

Ġrequires

Ġvast

Ġamounts Ġof Ġdata . SuperBPE

Appendix D: Qualitative Comparison To illustrate the segmentation behavior of AdaBPE on representative multiword expressions, we generated a set of example sentences targeting constructs

\ frac { n \ sum _{ i = 1 }^ n

AdaBPE

The

ĠdevelopmentĠof

ĠArtificial

ĠIntelligence

Ġrequires

ĠamountsĠof

Ġdata.

The

ĠdevelopmentĠof

ĠArtificialĠIntelligence Ġvast

Ġvast

ĠamountsĠof

Ġrequires Ġdata.

E XAMPLE 4 — The new software uses state of the art machine learning.

BPE

The Ġnew Ġsoftware Ġuses Ġstate

E XAMPLE 7 — At the end of the long march, the soldiers were still standing in formation on the side of the road. At

BPE

Ġmarch

Ġof Ġthe Ġart Ġmachine Ġlearning . SuperBPE

TheĠnew ĠstateĠofĠthe

ĠmachineĠlearning AdaBPE

Ġstill

Ġsoftware

Ġuses

Ġon .

Ġart

SuperBPE

TheĠnew

Ġsoftware

Ġuses

ĠstateĠofĠtheĠart

ĠmachineĠlearning

Ġthe

Ġend ,

Ġthe

BPE

The

Ġhigh

Ġschool

Ġice

Ġcream

Ġstudent

Ġate

Ġsitting

Ġon

Ġwhile

Ġthe

Ġlong

Ġsoldiers

Ġwere

Ġstanding Ġthe

Ġside

Ġin Ġof

Ġformation Ġthe

AtĠthe

ĠendĠofĠthe

Ġmarch

,Ġthe

Ġlong Ġsoldiers Ġstanding

Ġin

ĠonĠthe

Ġformation Ġroad

AtĠtheĠendĠof

AdaBPE

Ġroad

ĠwereĠstill

ĠsideĠofĠthe E XAMPLE 5 — The high school student ate ice cream while sitting on the living room couch.

Ġof

. ĠtheĠlong

Ġmarch

,Ġthe

Ġsoldiers

ĠwereĠstill

ĠstandingĠin

Ġformation

ĠonĠtheĠsideĠofĠtheĠroad

Ġthe Ġliving Ġroom Ġcouch . SuperBPE

The

ĠhighĠschool

Ġstudent

Ġate

ĠiceĠcream

Ġwhile

ĠsittingĠonĠthe

AdaBPE

E XAMPLE 8 — Then, late at night, he buried the old box in the ground beside the road.

ĠlivingĠroom

Ġcouch

The

ĠhighĠschool

Ġstudent

Ġate

ĠiceĠcream

Ġwhile

BPE

ĠsittingĠonĠthe

ĠlivingĠroom

SuperBPE

AdaBPE Ġcloud

Ġgathered

Ġthe

Ġvalley

Ġand

Ġstorm

Ġcloud

Ġover Ġanother

Ġloomed

Ġon

Ġlate

Ġat

Ġnight

Ġhe

Ġground Ġbeside Ġthe Ġroad .

E XAMPLE 6 — Storm cloud gathered over the valley, and another storm cloud loomed on the horizon

Storm

Ġburied Ġthe Ġold Ġbox Ġin Ġthe

Ġcouch

BPE

Then

Ġlate

,Ġhe

Ġburied

ĠatĠnight

ĠtheĠold

Ġbox

ĠinĠthe

Ġground

ĠtheĠroad

Then

ĠlateĠatĠnight

,Ġhe

ĠtheĠold

Ġbox

Ġburied

ĠinĠtheĠground ĠtheĠroad

Ġbeside

Ġthe Ġhorizon SuperBPE

Storm

Ġcloud

Ġgathered

ĠoverĠthe

Ġvalley

,ĠandĠanother

AdaBPE

Ġstorm

Ġloomed

ĠonĠthe

Storm

Ġcloud

ĠoverĠtheĠvalley Ġstorm

Ġcloud

ĠonĠtheĠhorizon

Ġcloud

Ġhorizon Ġgathered ,ĠandĠanother Ġloomed

The examples above provide evidence of the recurring differences between AdaBPE and SuperBPE in their treatment of LATEX syntax and of general natural language. For LATEX examples, AdaBPE tends to merge complete syntactic units into single tokens, whereas SuperBPE often performs only partial merges. Representative examples include \begin{equation}, \end{equation}, \sum_{i=, and \rightarrow.

For general-language examples, the two tokenizers typically agree on short, high-frequency multiword expressions. An exception is Example 3, where AdaBPE merges Artificial Intelligence into a single supertoken, while SuperBPE keeps the two words separate. The difference becomes more pronounced for longer expressions. In Example 4, AdaBPE merges the entire phrase state of the art into a single supertoken, whereas SuperBPE represents it with two tokens. Likewise, in Examples 7–8, AdaBPE learns longer expressions such as At the end of, on the side of the road, late at night, and in the ground, as well as phrasal verbs such as standing in, which SuperBPE tokenizes into multiple tokens.

Appendix E: Exclusive Tokens To complement our qualitative analysis, we report in Tables 8–12 the top-frequent tokens that are exclusive to each tokenizer’s vocabulary, on each of the full-size evaluation dataset. Two consistent patterns emerge across datasets. On the LATEX-heavy datasets (latex-formulas, Table 8; Image2Struct, Table 12), AdaBPE’s exclusive tokens correspond to well-formed LATEX constructs: environment delimiters such as begin{align*}, end{align*}, indexed-variable patterns such as _{n, _{k, _{j, and command prefixes such as }\frac{ and sum_{i= are all represented as single tokens by AdaBPE, whereas SuperBPE often splits them into multiple tokens. On the general-language datasets (CSQA, Table 9; SQuAD, Table 10; DROP, Table 11), SuperBPE’s unique tokens on these datasets often correspond to dataset-specific named entities or frequent lexicalized expressions, which capture recurring surface forms but do not necessarily reflect productive linguistic patterns (e.g. UnitedĠStates, AfricanĠAmerican, citationĠneeded). AdaBPE’s exclusive tokens are also frequent surface forms, but they arguably reflect more compositional and syntactic structure than SuperBPE’s dataset-specific named entities, such as determinerled noun phrases and multiword expressions—e.g. ĠtheĠregion, ĠtheĠSovietĠUnion, ĠearlyĠin— consistent with the segmentation examples in Appendix C.

Appendix F: Details on Model Training for Downstream Evaluation Table 7 reports the main settings used to train OLMo2-AdaBPE and OLMo2-SuperBPE. Both

models were trained on the full 100GB subset released by Liu et al. (2025), and share an identical architecture and training budget (e.g., number of epochs, batch size), so that they differ only in the tokenizer used to preprocess the training corpus (AdaBPE vs. SuperBPE). Keeping every other architectural and optimization choice fixed isolates the effect of the tokenizer on the downstream results reported in Table 5 ensuring that observed differences in accuracy and BPB are attributable to the tokenization strategy rather than to confounding factors in model capacity or training regime.

Architecture Hidden size Position embedding Attention heads Activation function Layers Epochs Batch size Optimizer Tokenizer

OLMo-AdaBPE

OLMo-SuperBPE

OLMo-2 2048 RoPE 16 SiLU 16 3 8 AdamW AdaBPE

OLMo-2 2048 RoPE 16 SiLU 16 3 8 AdamW SuperBPE

Table 7: Main hyperparameters used during model training

Appendix G: Details on Sensitivity Analysis Table 13 reports the AEL obtained on the evaluation data when each parameter is varied within the bounds from the previous section, while keeping all other settings fixed. The first row corresponds to the configuration used in the main experiments min min (θgeneral = 0.3, θlatex = 0.08); the remaining rows progressively relax the thresholds.

AdaBPE only Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Token end{align*} begin{align*} \begin{align _{n _{k _{j Ġ\sum_{ }\left( )Ġ=Ġ\ sum_{i= tilde{ int_ int_{ }\left t_ t, bar{ }\frac{ }ĠĠx_

SuperBPE only

Frequency

Rank

Token

98,652 58,772 40,418 10,119 6,384 6,235 4,486 4,185 4,123 4,116 4,098 4,077 4,013 3,737 3,671 3,449 3,436 3,315 3,244 3,190

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

left( (t \right ,Ġ\ left(\ \in nĠ+Ġ\ }.\ Ġ\in Ġ\right (z Ġ&Ġ Ġ&Ġ\ cases i, (n ,y (y (a

Frequency 15,092 12,354 11,718 9,964 9,765 9,166 8,080 7,576 7,379 6,097 5,695 5,330 5,001 4,905 4,492 3,723 3,643 3,505 3,235 2,778

Table 8: Top exclusive tokens by frequency on latex-formulas. Left: tokens used by AdaBPE but absent from SuperBPE. Right: tokens used by SuperBPE but absent from AdaBPE.

AdaBPE only Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Token ĠdoĠwhat Ġdo? ,ĠwhatĠwas Ġdidn ĠwhatĠto ĠyouĠbuy ĠdoingĠwhat Ġdidn’ ĠbeĠfound WhatĠkindĠof ĠstandingĠin tĠlike ĠdoĠif ĠwhatĠdo ĠdoĠto ĠhappensĠto ,ĠsoĠwhat ,ĠwhatĠdo ĠtoĠbed ĠwhatĠwill

SuperBPE only

Frequency

Rank

201 178 81 72 68 60 56 52 48 44 43 42 40 35 34 32 31 30 28 25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Token WhatĠhappens ĠwasĠhe TheĠman WhatĠcan Ġficus WhyĠwould ĠdidĠshe ĠwereĠthey ĠcouldĠhappen Ġit. ĠtoĠit Ġdidn’tĠwantĠto ĠisĠknown ĠfreshĠair ĠwantedĠa WhyĠdoes What’s Ġdidn’tĠhave ĠfastĠfood Ġdidn’tĠknow

Table 9: Top exclusive tokens by frequency on CSQA.

Frequency 80 67 65 54 53 51 48 43 37 31 30 22 21 20 17 15 14 13 13 12

AdaBPE only Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Token :Ġ .ĠWhatĠwasĠthe ĠtheĠlateĠ :Ġthe .ĠWhatĠtypeĠof ĠtheĠregion Ġ(e ĠtheĠisland thĠand .ĠWhatĠare ĠtheĠperiod ĠHowĠmany ĠtheĠRoman ĠtheĠnational ĠtheĠSovietĠUnion ĠtheĠmidĠtheĠuniversity ĠtheĠNorth ĠtheĠChinese ĠannouncedĠthat

SuperBPE only Frequency

Rank

1,735 1,664 1,239 1,121 1,061 1,044 990 986 973 922 887 874 854 841 833 827 784 758 751 749

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Token citationĠneeded ?Ġ ĠÂ°C ĠuntilĠ Ġcity’s ĠapproximatelyĠ .ĠByĠ thĠcentury, ĠMiddleĠAges th-century OnĠ thĠcenturies Ġworld’s -speaking ĠUS$ Ġ(including ĠPeople’s ĠRomanĠEmpire Ġd’ thĠandĠ

Frequency 3,488 2,407 1,655 1,564 1,214 1,188 987 981 981 969 924 912 880 795 786 769 754 751 719 711

Table 10: Top exclusive tokens by frequency on SQuAD.

AdaBPE only Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Token S. ĠtheĠfourth ĠtheĠlead ĠunderĠtheĠageĠofĠ ĠThereĠwere ĠpeopleĠ( ĠtheĠperiod InĠtheĠfirst ĠtheĠloss ĠHowĠmany ĠofĠindividuals .ĠInĠtheĠsecond ĠaĠfemale Ġtogether, ĠtheĠlongest ĠearlyĠin ĠfirstĠhalf ĠtheĠsecondĠhalf ĠtheĠSpanish ĠtheirĠhome

SuperBPE only Frequency

Rank

3,874 1,637 1,260 1,086 640 633 632 599 592 516 450 437 415 402 401 394 366 330 308 300

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Token ),Ġ UnitedĠStates U.S. ĠonĠaĠ ĠfromĠtwo ĠwereĠ ĠageĠ ĠandĠaĠ ĠforĠaĠ ĠeveryĠ ĠAfricanĠAmerican ’Ġ ĠanĠ ).ĠThere Ġyears,Ġ ĠNativeĠAmerican Ġendzone -point -winning ĠNewĠEngland

Table 11: Top exclusive tokens by frequency on DROP.

Frequency 5,274 4,956 3,864 1,890 1,238 1,147 1,054 1,020 996 982 853 775 663 610 583 561 462 401 358 314

AdaBPE only Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Token

SuperBPE only Frequency

Rank

263 219 205 202 198 185 133 132 126 126 126 122 103 101 89 84 82 82 81 79

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

_{t Ġ\textbf{ \end{equation*} hat{\ \begin{equation*} },Ġ\ Ġ\text{ \caption{ hat{ begin{align*} end{align*} _{k boldsymbol{\ {x mathbf{x tilde{ _{\text{ }Ġ= }Ġ=Ġ\ theta_

Token

Frequency

}Ċ $Ċ *}Ċ i, ]Ċ ,Ġ\ align equation Ġ\in \in \begin{equation \item .Ċ $.Ċ Ġ\\Ċ left( (t Ġ+Ġ\ \\Ċ left(\

1,483 498 409 294 288 273 256 238 231 208 198 175 164 140 134 121 116 115 112 101

Table 12: Top exclusive tokens by frequency on Image2Struct.

|V |

DROP

SQuAD

CSQA

I2S

baseline

204,601

210.43

126.03

12.24

54.88

207.96

min = 0.05, θlatex min = 0.5 θgeneral

181,752

222.60 (-5.78%)

136.02 (-7.92%)

13.44 52.93 202.47 (-9.80%) (+3.55%) (+2.64%)

min = 0.08, θlatex min = 0.5 θgeneral

168,292

222.60 (-5.78%)

136.02 (-7.92%)

13.44 (-9.80%)

55.75 211.63 (-1.58%) (-1.76%)

min = 0.05, θlatex min = 0.3 θgeneral

208,043

210.43 (+0.0%)

126.03 (+0.0%)

12.24 (+0.0%)

52.57 200.41 (+4.20%) (+3.63%)

min = 0.08, 247,260 θlatex min θgeneral = 0.25

208.21 (+1.05%)

124.25 (+1.41%)

12.04 54.86 207.90 (+1.63%) (+0.03%) (+0.02%)

min = 0.05, θlatex 261,512 min θgeneral = 0.25

206.33 124.24 12.04 52.11 198.89 (+1.94%) (+1.42%) (+1.63%) (+5.04%) (+4.36%)

min = 0.05, θlatex 317,155 min θgeneral = 0.23

205.08 (+2.60%)

122.57 (+2.74%)

11.86 (+3.10%)

min = 0.05, θlatex min θgeneral = 0.2

201.14 (+4.41%)

119.65 (+5.06%)

11.51 51.13 196.25 (+5.96%) (+6.83%) (+5.63%)

Parameters

646,506

51.86 197.48 (-5.50%) (+5.03%)

min = 0.08, 1,685,088 θlatex 183.60 107.57 10.09 54.93 204.55 min θgeneral = 0.1 (+12.75%) (+14.64%) (+17.56%) (-0.09%) (+1.63%)

min min Table 13: Sensitivity analysis of AdaBPE: AEL scores for different settings of θgeneral and θlatex . Values in parentheses indicate the percentage variation relative to the baseline.

Adaptive Frequency Threshold Byte Pair Encoding for Mixed Natural Language and Structured Symbolic Corpora

Analyze & explore related research in Conceptio

Related documents