Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
arXiv cs.AI · 20 Oct 2026, 03:57 AEST
LLM agents can compute correct answers but still provide incorrect responses due to context management issues. The study highlights how retrieval methods impact the accuracy of models like GPT-5 and Claude when using a geochemistry simulator.
Key facts
- LLM agents that compute the right answer can still answer wrongly.
- When agents read the simulator's report through a section index rather than in context, GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy.
- Under retrieval, they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution.
- Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models.
Read aloud
L L M agents can compute correct answers but still provide incorrect responses due to context management issues. The study highlights how retrieval methods impact the accuracy of models like G P T five and Claude when using a geochemistry simulator. L L M agents that compute the right answer can still answer wrongly. When agents read the simulator's report through a section index rather than in context, G P T five.one and G P T five.two lose eleven point three and seven point nine points of accuracy. Under retrieval, they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution. Naming the authoritative output raises accuracy by six point three-seven point nine points over plain retrieval for the three GPT models.
Sources in this cluster
- Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
arXiv cs.AI · 20 Oct 2026, 03:57 AEST