KEEP-Bench: Evaluating Evidence Preservation in Context Compression
arXiv cs.CL · 20 Oct 2026, 03:55 AEST
KEEP-Bench is a new benchmark designed to evaluate the faithfulness of context compression in AI models. It reveals that even advanced models struggle to preserve necessary evidence, with significant variability in performance across different evidence structures.
Key facts
- Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck.
- Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-weight model (31B) preserves all required evidence in only 66.8% of cases, and the best proprietary model in only 61.6%.
- We further introduce a metric, Evidence Preservation Return (EPR), to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales (4B-8B), where recall remains low.
Read aloud
KEEP-Bench is a new benchmark designed to evaluate the faithfulness of context compression in A I models. It reveals that even advanced models struggle to preserve necessary evidence, with significant variability in performance across different evidence structures. Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only three.1K tokens, the best open-weight model 31B preserves all required evidence in only sixty six point eight percent of cases, and the best proprietary model in only sixty one point six percent. We further introduce a metric, Evidence Preservation Return EPR, to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales 4B-8B, where recall remains low.
Sources in this cluster
- KEEP-Bench: Evaluating Evidence Preservation in Context Compression
arXiv cs.CL · 20 Oct 2026, 03:55 AEST