AI TrackerFrontier desk · AEST
Back to news
normalaitech

Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination

arXiv cs.AI · 23 Oct 2026, 03:23 AEST

A study examines the performance of open language models on the Spanish MIR residency examination. The findings highlight issues of overconfidence and reasoning in medical contexts.

Key facts

Read aloud

A study examines the performance of open language models on the Spanish MIR residency examination. The findings highlight issues of overconfidence and reasoning in medical contexts. Accuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more. We examine these questions for four open models of eight to twenty seven billion parameters on the Spanish MIR residency examination, scoring one thousand five hundred fifty two questions from eight editions under the exam's own rules. Answering every question, the three larger models would rank among roughly the first six hundred to three thousand candidates, with forty one to forty three specialties still open to them. Medical fine-tuning reduced overconfidence at the cost of accuracy, without improving how well confidence separates right from wrong answers. Moreover, thirteen-thirty eight percent of correct answers rejected a distractor for a clinically unsound reason that confidence did not reflect.

Sources in this cluster

Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination · AI Tracker