Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
arXiv cs.CL · 2 Oct 2026, 03:46 AEST
Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics.
Key facts
- Keyword-matching benchmarks can credit small models for tool use they never perform.
- We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics.
- A 661.6M parameter model (approx.
- 65% code/technical text; no dedicated SFT) and a 1,109M model (web-heavy multi-phase curriculum; 6B-token tool-SFT) share decoder, tokenizer, and special tokens, scoring almost identically on lenient tool-use metrics.
- Verbatim-reproduction checks on training examples separate them completely: the 600M emits valid tool calls with generalized arguments on 6/6 examples; the 1B does so on 0/6 across checkpoints.
Read aloud
Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. A six hundred sixty one.6M parameter model approx. sixty five percent code/technical text; no dedicated SFT and a one109M model web-heavy multi-phase curriculum; 6B-token tool-SFT share decoder, tokenizer, and special tokens, scoring almost identically on lenient tool-use metrics. Verbatim-reproduction checks on training examples separate them completely: the 600M emits valid tool calls with generalized arguments on six/six examples; the 1B does so on zero/six across checkpoints.
Sources in this cluster
- Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
arXiv cs.CL · 2 Oct 2026, 03:46 AEST