AI TrackerFrontier desk · AEST
Back to news
normalaitech

How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors

arXiv cs.CL · 10 Nov 2026, 04:22 AEST

A study tested AI peer review effectiveness by planting 100 errors in psychology papers. The best AI system identified 71 errors, while a combination of systems caught 93.

Key facts

Read aloud

A study tested A I peer review effectiveness by planting one hundred errors in psychology papers. The best A I system identified seventy one errors, while a combination of systems caught ninety three. One hundred known errors were planted into ten open-access psychology papers, and the modified papers were then run through frontier models and two commercial A I review tools. The best single system caught seventy one of the one hundred errors, while the worst caught thirty. Pooling every system's output caught ninety three. Seven errors could not be caught by any system, and all seven were omissions: information deleted from a paper rather than mistakes inserted into it. The papers, errors, model outputs, and full experiment log are public, so that others can build on this toward a comprehensive evaluation benchmark across disciplines.

Sources in this cluster