ProteinGeoBench: Frontier Structural-Biology Agents Are Fluent at Lookup but Lost in Geometry
arXiv cs.AI · 20 Oct 2026, 03:55 AEST
Frontier agents excel at retrieving structural metadata but struggle with geometric calculations. The study of ProteinGeoBench reveals significant variability in model performance based on evaluation configurations.
Key facts
- Frontier agents retrieve structural metadata well and compute geometry from coordinates less reliably.
- We study ProteinGeoBench, a suite of 131 protein-structure tasks, using 4,249 rollouts from four frontier models and 4,486 further trials across 26 models.
- Three of the four frontier models fall within 0.031 of one another (0.785 to 0.816) with no difference between them resolved, while the fourth scores 0.707 and all three gaps to it exclude zero.
- Three of the four do better on metadata lookup than on numeric calculation.
- Numeric tasks are 28 of the 131 but half of the twelve tasks that all four models fail.
Read aloud
Frontier agents excel at retrieving structural metadata but struggle with geometric calculations. The study of ProteinGeoBench reveals significant variability in model performance based on evaluation configurations. Frontier agents retrieve structural metadata well and compute geometry from coordinates less reliably. We study ProteinGeoBench, a suite of one hundred thirty one protein-structure tasks, using four thousand two hundred forty nine rollouts from four frontier models and four thousand four hundred eighty six further trials across twenty six models. Three of the four frontier models fall within zero point zero three one of one another zero point seven eight five to zero point eight one six with no difference between them resolved, while the fourth scores zero point seven zero seven and all three gaps to it exclude zero. Three of the four do better on metadata lookup than on numeric calculation. Numeric tasks are twenty eight of the one hundred thirty one but half of the twelve tasks that all four models fail.
Sources in this cluster
- ProteinGeoBench: Frontier Structural-Biology Agents Are Fluent at Lookup but Lost in Geometry
arXiv cs.AI · 20 Oct 2026, 03:55 AEST