Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers
arXiv cs.CL · 16 Nov 2026, 04:44 AEST
Typed Decision Models can replace fine-tuned classifiers in many NLP tasks without the need for training. However, fine-tuning still proves to be the most accurate method overall.
Key facts
- Typed Decision Models such as TypeSafe's Jev, also called System One or System 1 models, answer typed questions about an input with a probability for each option, and they need no training.
- We ask how many of these fine-tuned classifiers a Typed Decision Model can replace under each paper's own evaluation, and what is needed where it cannot.
- With a zero-shot prompt, Jev directly replaced the paper's predictor on 12 of the 46 test sets, mostly single judgments with plain labels, but rarely where the paper's method was trained on its labels.
- Fine-tuning remains the most accurate option, but a zero-shot prompt is often enough for plain labels, and prompt optimization closes most of the remaining gap.
Read aloud
Typed Decision Models can replace fine-tuned classifiers in many N L P tasks without the need for training. However, fine-tuning still proves to be the most accurate method overall. Typed Decision Models such as TypeSafe's Jev, also called System One or System one models, answer typed questions about an input with a probability for each option, and they need no training. We ask how many of these fine-tuned classifiers a Typed Decision Model can replace under each paper's own evaluation, and what is needed where it cannot. With a zero-shot prompt, Jev directly replaced the paper's predictor on twelve of the forty six test sets, mostly single judgments with plain labels, but rarely where the paper's method was trained on its labels. Fine-tuning remains the most accurate option, but a zero-shot prompt is often enough for plain labels, and prompt optimization closes most of the remaining gap.
Sources in this cluster
- Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers
arXiv cs.CL · 16 Nov 2026, 04:44 AEST