MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection
arXiv cs.AI · 4 Nov 2026, 04:40 AEST
MedPhysBench is a benchmark designed to evaluate language models on medical physics calculations and error detection. It includes ten seeded generators of calculation items across various medical physics domains and assesses model performance on both calculations and error detection.
Key facts
- Medical physicists turn a prescription into a delivered dose and check one another's calculations, because one error in a beam calibration or a monitor-unit (MU) calculation can affect every patient treated with it.
- On a held-out test split of 1,200 items (half for two models), 12 models scored 33.3 to 100.0% on calculation.
- Five models detected at least 96.7% of mostly large planted errors with at most 4% false alarms; for four of the other seven, checking only the result raised false alarms significantly.
- In an exploratory targeted test with 20 public cases of an omitted inverse-square correction, the three best models tested caught it in every worksheet.
Read aloud
MedPhysBench is a benchmark designed to evaluate language models on medical physics calculations and error detection. It includes ten seeded generators of calculation items across various medical physics domains and assesses model performance on both calculations and error detection. Medical physicists turn a prescription into a delivered dose and check one another's calculations, because one error in a beam calibration or a monitor-unit MU calculation can affect every patient treated with it. On a held-out test split of one thousand two hundred items half for two models, twelve models scored thirty three point three to one hundred point zero percent on calculation. Five models detected at least ninety six point seven percent of mostly large planted errors with at most four percent false alarms; for four of the other seven, checking only the result raised false alarms significantly. In an exploratory targeted test with twenty public cases of an omitted inverse-square correction, the three best models tested caught it in every worksheet.
Sources in this cluster
- MedPhysBench: A Benchmark for Evaluating Language Models on Medical Physics Calculations and Error Detection
arXiv cs.AI · 4 Nov 2026, 04:40 AEST