How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors
arXiv cs.CL · 10 Nov 2026, 04:22 AEST
A study tested AI peer review effectiveness by planting 100 errors in psychology papers. The best AI system identified 71 errors, while a combination of systems caught 93.
Key facts
- One hundred known errors were planted into 10 open-access psychology papers, and the modified papers were then run through frontier models and two commercial AI review tools.
- The best single system caught 71 of the 100 errors, while the worst caught 30.
- Pooling every system's output caught 93.
- Seven errors could not be caught by any system, and all seven were omissions: information deleted from a paper rather than mistakes inserted into it.
- The papers, errors, model outputs, and full experiment log are public, so that others can build on this toward a comprehensive evaluation benchmark across disciplines.
Read aloud
A study tested A I peer review effectiveness by planting one hundred errors in psychology papers. The best A I system identified seventy one errors, while a combination of systems caught ninety three. One hundred known errors were planted into ten open-access psychology papers, and the modified papers were then run through frontier models and two commercial A I review tools. The best single system caught seventy one of the one hundred errors, while the worst caught thirty. Pooling every system's output caught ninety three. Seven errors could not be caught by any system, and all seven were omissions: information deleted from a paper rather than mistakes inserted into it. The papers, errors, model outputs, and full experiment log are public, so that others can build on this toward a comprehensive evaluation benchmark across disciplines.
Sources in this cluster
- How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors
arXiv cs.CL · 10 Nov 2026, 04:22 AEST