Can decision models extract clinical data from cancer pathology reports? An evaluation of accuracy, calibration, and failure modes
arXiv cs.AI · 16 Nov 2026, 02:57 AEST
Decision models have been evaluated for extracting clinical data from cancer pathology reports, showing promising accuracy and speed. The study highlights the need for validation in specific configurations and suggests combining these models with other methods for improved reliability.
Key facts
- We evaluated two open-weights decision models, Nimble 9B and Tev1 4B, for extracting 25 fields from 1,113 oral cavity cancer pathology reports from a single centre, against a prospectively maintained clinical database.
- Tev1 reached a mean field accuracy of 0.917 (95% CI 0.913 to 0.920), the same as Qwen3.5 9B (0.918; p = 0.26), and was four times faster (15 s against 61 s per report).
- Both decision models matched GLM-5.3-flash on simple dichotomous categories.
- Pooled over fields, the returned probabilities were well calibrated (expected calibration error 0.022 and 0.026), but they did not flag systematic errors within single fields.
- Decision models can extract explicitly stated findings from pathology reports quickly on local hardware and return useful probabilities.
Read aloud
Decision models have been evaluated for extracting clinical data from cancer pathology reports, showing promising accuracy and speed. The study highlights the need for validation in specific configurations and suggests combining these models with other methods for improved reliability. We evaluated two open-weights decision models, Nimble 9B and Tev1 4B, for extracting twenty five fields from one thousand one hundred thirteen oral cavity cancer pathology reports from a single centre, against a prospectively maintained clinical database. Tev1 reached a mean field accuracy of zero point nine one seven ninety five percent CI zero point nine one three to zero point nine two zero, the same as Qwen3.five 9B zero point nine one eight; p = zero point two six, and was four times faster fifteen s against sixty one s per report. Both decision models matched GLM-five point three-flash on simple dichotomous categories. Pooled over fields, the returned probabilities were well calibrated expected calibration error zero point zero two two and zero point zero two six, but they did not flag systematic errors within single fields. Decision models can extract explicitly stated findings from pathology reports quickly on local hardware and return useful probabilities.
Sources in this cluster
- Can decision models extract clinical data from cancer pathology reports? An evaluation of accuracy, calibration, and failure modes
arXiv cs.AI · 16 Nov 2026, 02:57 AEST