What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
arXiv cs.AI · 28 Oct 2026, 03:52 AEST
Language models pretrained on web corpora learn both general and domain-specific knowledge over time. A study reveals that high-temporal variability score items improve faster during pretraining and outperform low-TVS items in most settings.
Key facts
- Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge.
- Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models.
- We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on.
- At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy.
- Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.
Read aloud
Language models pretrained on web corpora learn both general and domain-specific knowledge over time. A study reveals that high-temporal variability score items improve faster during pretraining and outperform low-TVS items in most settings. Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. Using these models, we propose a temporal variability score TVS, defined as the coefficient of variation of an item's negative log-likelihood across models. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in thirty four of thirty six model-benchmark settings, with an average advantage of seven point three points of chance-normalized accuracy. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.
Sources in this cluster
- What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
arXiv cs.AI · 28 Oct 2026, 03:52 AEST