AI TrackerFrontier desk · AEST
Back to news
normalaitech

Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models

arXiv cs.CL · 2 Oct 2026, 03:46 AEST

Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics.

Key facts

Read aloud

Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. A six hundred sixty one.6M parameter model approx. sixty five percent code/technical text; no dedicated SFT and a one109M model web-heavy multi-phase curriculum; 6B-token tool-SFT share decoder, tokenizer, and special tokens, scoring almost identically on lenient tool-use metrics. Verbatim-reproduction checks on training examples separate them completely: the 600M emits valid tool calls with generalized arguments on six/six examples; the 1B does so on zero/six across checkpoints.

Sources in this cluster