Elucidating the Design Space of LM Data Attribution
arXiv cs.CL · 30 Oct 2026, 03:59 AEST
Data attribution methods for Language Models are evaluated using downstream tasks, but existing evaluations complicate the isolation of failures. This study introduces the Controlled Attribution Benchmark to systematically analyze the impact of dataset configurations on attribution performance.
Key facts
- Data attribution methods for Language Models (LMs) are typically evaluated using downstream tasks like fact retrieval.
- We introduce the Controlled Attribution Benchmark (CATT) for fact tracing, in which we fine-tune LLMs on entirely fictitious knowledge, providing unambiguous ground truth for attribution.
- We find that replacing standard cross-entropy loss gradients with the negative log-odds gradients surfaces the critical tokens from other uninformative ones.
- We can then aggregate only the top token scores to improve ground truth recall by 35 to 72 percentage points across multiple baselines on our diagnostic benchmark.
Read aloud
Data attribution methods for Language Models are evaluated using downstream tasks, but existing evaluations complicate the isolation of failures. This study introduces the Controlled Attribution Benchmark to systematically analyze the impact of dataset configurations on attribution performance. Data attribution methods for Language Models LMs are typically evaluated using downstream tasks like fact retrieval. We introduce the Controlled Attribution Benchmark CATT for fact tracing, in which we fine-tune L L M s on entirely fictitious knowledge, providing unambiguous ground truth for attribution. We find that replacing standard cross-entropy loss gradients with the negative log-odds gradients surfaces the critical tokens from other uninformative ones. We can then aggregate only the top token scores to improve ground truth recall by thirty five to seventy two percentage points across multiple baselines on our diagnostic benchmark.
Sources in this cluster
- Elucidating the Design Space of LM Data Attribution
arXiv cs.CL · 30 Oct 2026, 03:59 AEST