Tracing the Scientific Discovery Capacity of LLMs to Their Training Data
arXiv cs.AI · 3 Nov 2026, 04:50 AEST
Large language models (LLMs) show growing promise for scientific discovery, yet basic questions remain about what enables reliable discovery. Our results identify monofact rate in training data as a key factor governing discoveries in language models.
Key facts
- We study this question as one of missing probability mass and develop a theoretical account in which monofacts-the facts observed once during training-set the discovery capacity.
- In materials science and regulatory genomics, we directly manipulate monofact rates in training sets and find that more monofacts consistently yield more discoveries across two open-weight model families, sampling strategies, and reasoning budgets.
- Further experiments show the relationship cannot be explained by training-set diversity alone and can transfer across scientific domains when paired with in-domain data.
Read aloud
Large language models L L M s show growing promise for scientific discovery, yet basic questions remain about what enables reliable discovery. Our results identify monofact rate in training data as a key factor governing discoveries in language models. We study this question as one of missing probability mass and develop a theoretical account in which monofacts-the facts observed once during training-set the discovery capacity. In materials science and regulatory genomics, we directly manipulate monofact rates in training sets and find that more monofacts consistently yield more discoveries across two open-weight model families, sampling strategies, and reasoning budgets. Further experiments show the relationship cannot be explained by training-set diversity alone and can transfer across scientific domains when paired with in-domain data.
Sources in this cluster
- Tracing the Scientific Discovery Capacity of LLMs to Their Training Data
arXiv cs.AI · 3 Nov 2026, 04:50 AEST