Three LLM judges couldn't tell an AI agent's real memory from a counterfeit one
Hacker News · 18 Oct 2026, 05:31 AEST
Three LLM judges were unable to distinguish between an AI agent's real memory and a counterfeit one. This highlights vulnerabilities in current evaluation methods for agent memory.
Key facts
- Agent memory is usually evaluated one of two ways: an LLM judge scoring free text, or a recognition probe.
- We show that both, as commonly built, fail against two adversaries — a counterfeit memory, and a system with no memory that simply reasons.
- If you evaluate agent memory for a living, the useful part of this page is that you can run all three against your own system this afternoon.
Read aloud
Three L L M judges were unable to distinguish between an A I agent's real memory and a counterfeit one. This highlights vulnerabilities in current evaluation methods for agent memory. Agent memory is usually evaluated one of two ways: an L L M judge scoring free text, or a recognition probe. We show that both, as commonly built, fail against two adversaries — a counterfeit memory, and a system with no memory that simply reasons. If you evaluate agent memory for a living, the useful part of this page is that you can run all three against your own system this afternoon.
Sources in this cluster
- Three LLM judges couldn't tell an AI agent's real memory from a counterfeit one
Hacker News · 18 Oct 2026, 05:31 AEST