arXiv (All)
open access
Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining
Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and often requiring large vocabularies. We introduce \textbf{Phonemic Tokenizer}, a linguistically motivated tokenizer for Vietnamese and Chinese that
Conceptio Open Archive
Analyze & explore related research in Conceptio
Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.
This document is indexed with metadata only: full text is not available in the archive for this record. Open the official source ↗
Related documents
- The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese#620371
- The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinese#618692
- Advancing LLM-based phoneme-to-grapheme for multilingual speech recognition#973108
- Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages#652418
- Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages#650965
Record · ID 1035768
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Bring this data into your agent: Connect via MCP or API · 20 free credits to start · no card required
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive