Paper review · November 14, 2026
Pretraining Data Statistics Determine the Stages in Which Language Models Learn Entity Comparison
Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models
By tracking training on entity comparison tasks, the authors showed that language models shift strategies in the order frequency → position → correct answer, and that this order is causally determined by the data distribution.
Problem
For comparison questions such as 'Which is larger, France or Brazil?', a language model may answer with heuristics instead of comparing the actual values. For example, it may pick the entity it has seen more often, or the entity in a particular position in the prompt. It has not been well understood when such heuristics appear and disappear during pretraining, or how data statistics determine this.
Method
The authors defined 9 comparison tasks (countries, US states, universities, movies, people). They trained a 163M-parameter GPT-2-class model from scratch on a mix of the OLMo-2 pretraining corpus and synthetic task data. At each checkpoint, they counted the cumulative occurrence frequency of each entity. They then fit a logistic regression that predicts model outputs from three features: frequency, position (intercept) and actual value. The weight of each strategy was measured by how much regression accuracy dropped when one feature was removed. They tested causality by varying the synthetic data ratio (0–1%) and the entity distribution (Zipf vs. uniform), and applied the same analysis to OLMo-2 1B–32B checkpoints.
Results
- The frequency stage was the same regardless of the synthetic data ratio. It peaked around step 200 and disappeared around step 1K.
- With 0% and 0.01% synthetic data, the correct algorithm did not emerge and the model stayed at position bias. At 0.1% the correct algorithm emerged at about 2.5K steps, and at 1% within 1K steps, with drop-one accuracy reaching 18. At 1% there was no position stage at all.
- Changing the entity distribution of natural text from Zipf to uniform weakened the frequency stage. The correct algorithm emerged earliest when both natural text and synthetic data were uniform.
- The three stages also appeared in the same order in OLMo-2 (1B, 7B, 13B, 32B). However, which stages actually appeared depended on the task and model scale.
- Frequency bias did not fully disappear by the end of training. Accuracy stayed higher on pairs where the correct entity appeared more often.
Limitations
The controlled experiments used only the 163M small model and 3 tasks. Synthetic data identical to the templates was added so the small model could learn the tasks, so the conditions differ from natural pretraining. Evaluation also used the same entities as training, so memorization is possible (the authors checked this with a 20% holdout experiment in the appendix). The analysis relies on three hand-picked features.
Why it matters
It provides causal evidence that the frequency distribution of pretraining data directly shapes a model's shortcut learning. A practical implication is that adjusting data composition can make models rely less on heuristics. It also showed that small-scale experiments can predict large-scale training dynamics.
Key terms
- Zipf distribution
- A long-tail distribution in which a few items appear very often and most items appear rarely. Word and entity frequencies in natural language are a typical example.
- Drop-one accuracy
- A value that measures how much prediction accuracy drops when one feature is removed from a regression model. It shows how much that feature explains the model's behavior.
- Position bias
- The tendency to pick the entity that appears first or last in the prompt, regardless of the correct answer.
Original
Authors: Yik Siu Chan, Jing Huang, Yanai Elazar, Atticus Geiger · Posted 2026-11-12 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.