AI TrackerFrontier desk · AEST
Back to news
normalaitech

FIDEL: Abugida-Aware Pre-tokenization for Ethiopic Languages

arXiv cs.CL · 16 Nov 2026, 05:26 AEST

FIDEL introduces a family of reversible pre-tokenization schemes tailored for Ethiopic languages. The study evaluates these methods on various NLP tasks, revealing improvements in certain areas while noting declines in sentiment analysis performance.

Key facts

Read aloud

FIDEL introduces a family of reversible pre-tokenization schemes tailored for Ethiopic languages. The study evaluates these methods on various N L P tasks, revealing improvements in certain areas while noting declines in sentiment analysis performance. Subword tokenizers assume that meaningful linguistic units can be recovered by splitting text at learned boundaries, an assumption better suited to concatenative morphology than to Semitic root-and-pattern systems. The challenge is acute for the Ethiopic languages, whose abugida script encodes consonant-vowel units as single characters and hides consonantal structure. We report intrinsic tokenizer statistics for BPE and UnigramLM across vocabulary sizes alongside downstream N L P tasks, including transfer scenarios. We find that FIDEL reduction improves news classification consistently and named-entity recognition at larger capacity, while sentiment analysis performance declines under the more aggressive schemes, and extractive question answering performance is preserved.

Sources in this cluster

FIDEL: Abugida-Aware Pre-tokenization for Ethiopic Languages · AI Tracker