AI TrackerFrontier desk · AEST
Back to news
normalaitech

KEEP-Bench: Evaluating Evidence Preservation in Context Compression

arXiv cs.CL · 20 Oct 2026, 03:55 AEST

KEEP-Bench is a new benchmark designed to evaluate the faithfulness of context compression in AI models. It reveals that even advanced models struggle to preserve necessary evidence, with significant variability in performance across different evidence structures.

Key facts

Read aloud

KEEP-Bench is a new benchmark designed to evaluate the faithfulness of context compression in A I models. It reveals that even advanced models struggle to preserve necessary evidence, with significant variability in performance across different evidence structures. Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only three.1K tokens, the best open-weight model 31B preserves all required evidence in only sixty six point eight percent of cases, and the best proprietary model in only sixty one point six percent. We further introduce a metric, Evidence Preservation Return EPR, to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales 4B-8B, where recall remains low.

Sources in this cluster