Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
arXiv cs.AI · 3 Oct 2026, 03:27 AEST
Large language models (LLMs) show varying reliability when applied to the Colombian legal system. A new benchmark reveals that while models perform well on closed questions, their accuracy on free-text legal answers is significantly lower.
Key facts
- We introduce an expert-validated benchmark for evaluating LLM reliability on the Colombian legal system.
- Accuracy on closed questions ranges widely, from 0.905 (Gemini 3.1 Pro) to 0.577, but on free-text legal answers factual correctness never exceeds 0.45 (on a 0-1 scale) for any model.
- We find a dissociation between answer relevancy and correctness (Spearman rho = -0.46): models reliably sound responsive while frequently being wrong, a pattern of particular concern for non-expert users.
- Reliability varies systematically by legal area and follows an inverted-U across question complexity.
Read aloud
Large language models L L M s show varying reliability when applied to the Colombian legal system. A new benchmark reveals that while models perform well on closed questions, their accuracy on free-text legal answers is significantly lower. We introduce an expert-validated benchmark for evaluating L L M reliability on the Colombian legal system. Accuracy on closed questions ranges widely, from zero point nine zero five Gemini three point one Pro to zero point five seven seven, but on free-text legal answers factual correctness never exceeds zero point four five on a zero-one scale for any model. We find a dissociation between answer relevancy and correctness Spearman rho = -zero point four six: models reliably sound responsive while frequently being wrong, a pattern of particular concern for non-expert users. Reliability varies systematically by legal area and follows an inverted-U across question complexity.
Sources in this cluster
- Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
arXiv cs.AI · 3 Oct 2026, 03:27 AEST