Latent Core Tokenizer: Compress, but Meaningfully
arXiv cs.CL · 9 Oct 2026, 03:32 AEST
The Latent Core Tokenizer (LCT) offers a language-agnostic method for optimizing tokenization by separating structural discovery from vocabulary construction. It demonstrates improved performance across multiple languages compared to traditional methods like BPE and Unigram.
Key facts
- Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages.
- LCT uses Minimum Description Length, entropy-based boundary signals, and morphotactic constraints to identify reusable linguistic units before constructing a shared vocabulary.
- Across 104 languages with a 200K-token vocabulary, LCT achieves lower fertility and higher MorphScore than BPE, Unigram, and parity-aware BPE, while maintaining comparable cross-lingual disparity in tokenization cost.
- Across four multilingual downstream benchmarks, LCT improves aggregate score by 1.48, 1.83, and 2.00 points over BPE, Unigram, and parity-aware BPE, respectively.
- Our findings show that compression alone does not predict representation quality and highlight the importance of morphology-driven structural discovery.
Read aloud
The Latent Core Tokenizer LCT offers a language-agnostic method for optimizing tokenization by separating structural discovery from vocabulary construction. It demonstrates improved performance across multiple languages compared to traditional methods like BPE and Unigram. Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages. LCT uses Minimum Description Length, entropy-based boundary signals, and morphotactic constraints to identify reusable linguistic units before constructing a shared vocabulary. Across one hundred four languages with a 200K-token vocabulary, LCT achieves lower fertility and higher MorphScore than BPE, Unigram, and parity-aware BPE, while maintaining comparable cross-lingual disparity in tokenization cost. Across four multilingual downstream benchmarks, LCT improves aggregate score by one point four eight, one point eight three, and two point zero zero points over BPE, Unigram, and parity-aware BPE, respectively. Our findings show that compression alone does not predict representation quality and highlight the importance of morphology-driven structural discovery.
Sources in this cluster
- Latent Core Tokenizer: Compress, but Meaningfully
arXiv cs.CL · 9 Oct 2026, 03:32 AEST