Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions
arXiv cs.AI · 28 Oct 2026, 03:49 AEST
A new study examines how memory fails in group decisions made by large language models (LLMs). It finds that common heuristics for selecting relevant information often lead to inaccuracies in decision-making.
Key facts
- When an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep?
- We introduce CrossProfile, a diagnostic benchmark of 60 templates with automatically verified answer keys, in which the ownership of supporting, decoy, and irrelevant facts is crossed while their text is held fixed (480 tasks).
- Selecting by relevance readmits decoys, because decoys are on topic: even a perfect relevance filter loses 23 to 31 points to rarest-first selection in the classic hidden-profile configuration.
- In both cases accuracy follows the balance of evidence the memory keeps, not the policy that kept it.
- In natural discussions the evidence is lost before any memory acts: the complete transcript yields lower accuracy than no notes at all.
Read aloud
A new study examines how memory fails in group decisions made by large language models L L M s. It finds that common heuristics for selecting relevant information often lead to inaccuracies in decision-making. When an L L M group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? We introduce CrossProfile, a diagnostic benchmark of sixty templates with automatically verified answer keys, in which the ownership of supporting, decoy, and irrelevant facts is crossed while their text is held fixed four hundred eighty tasks. Selecting by relevance readmits decoys, because decoys are on topic: even a perfect relevance filter loses twenty three to thirty one points to rarest-first selection in the classic hidden-profile configuration. In both cases accuracy follows the balance of evidence the memory keeps, not the policy that kept it. In natural discussions the evidence is lost before any memory acts: the complete transcript yields lower accuracy than no notes at all.
Sources in this cluster
- Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions
arXiv cs.AI · 28 Oct 2026, 03:49 AEST