AI TrackerFrontier desk · AEST
Back to news
normalaitech

What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models

arXiv cs.AI · 28 Oct 2026, 03:52 AEST

Language models pretrained on web corpora learn both general and domain-specific knowledge over time. A study reveals that high-temporal variability score items improve faster during pretraining and outperform low-TVS items in most settings.

Key facts

Read aloud

Language models pretrained on web corpora learn both general and domain-specific knowledge over time. A study reveals that high-temporal variability score items improve faster during pretraining and outperform low-TVS items in most settings. Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. Using these models, we propose a temporal variability score TVS, defined as the coefficient of variation of an item's negative log-likelihood across models. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in thirty four of thirty six model-benchmark settings, with an average advantage of seven point three points of chance-normalized accuracy. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.

Sources in this cluster