arXiv (All)
open access
Frequency-Ordered Tokenization for Better Text Compression
We present frequency-ordered tokenization (FOT), a simple preprocessing technique that improves lossless text compression by exploiting the power-law frequency distribution of natural language (Zipf's law). FOT segments text into the vocabulary of a pretrained Byte Pair Encoding (BPE) tokenizer, ren
Conceptio Open Archive
Analyze & explore related research in Conceptio
Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.
This document is indexed with metadata only: full text is not available in the archive for this record. Open the official source ↗
Related documents
- zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss#1297957
- A Non-constant Lower Bound for Grammar-Based Compression with Greedy#1000075
- InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing#820206
- SumRank: Aligning Summarization Models for Long-Document Listwise Reranking#998158
- RenderRank: Learning to Rerank Text with Compressed Visual Tokens#1129671
Record · ID 1325106
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Bring this data into your agent: Connect via MCP or API · 20 free credits to start · no card required
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive