AI TrackerFrontier desk · AEST
Back to news
normalaitech

Elucidating the Design Space of LM Data Attribution

arXiv cs.CL · 30 Oct 2026, 03:59 AEST

Data attribution methods for Language Models are evaluated using downstream tasks, but existing evaluations complicate the isolation of failures. This study introduces the Controlled Attribution Benchmark to systematically analyze the impact of dataset configurations on attribution performance.

Key facts

Read aloud

Data attribution methods for Language Models are evaluated using downstream tasks, but existing evaluations complicate the isolation of failures. This study introduces the Controlled Attribution Benchmark to systematically analyze the impact of dataset configurations on attribution performance. Data attribution methods for Language Models LMs are typically evaluated using downstream tasks like fact retrieval. We introduce the Controlled Attribution Benchmark CATT for fact tracing, in which we fine-tune L L M s on entirely fictitious knowledge, providing unambiguous ground truth for attribution. We find that replacing standard cross-entropy loss gradients with the negative log-odds gradients surfaces the critical tokens from other uninformative ones. We can then aggregate only the top token scores to improve ground truth recall by thirty five to seventy two percentage points across multiple baselines on our diagnostic benchmark.

Sources in this cluster