← Key papers

Paper review · November 14, 2026

Pretraining Data Statistics Determine the Stages in Which Language Models Learn Entity Comparison

Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models

Language modelsSafety & alignment Intermediate cs.CL

By tracking training on entity comparison tasks, the authors showed that language models shift strategies in the order frequency → position → correct answer, and that this order is causally determined by the data distribution.

Problem

For comparison questions such as 'Which is larger, France or Brazil?', a language model may answer with heuristics instead of comparing the actual values. For example, it may pick the entity it has seen more often, or the entity in a particular position in the prompt. It has not been well understood when such heuristics appear and disappear during pretraining, or how data statistics determine this.

Method

The authors defined 9 comparison tasks (countries, US states, universities, movies, people). They trained a 163M-parameter GPT-2-class model from scratch on a mix of the OLMo-2 pretraining corpus and synthetic task data. At each checkpoint, they counted the cumulative occurrence frequency of each entity. They then fit a logistic regression that predicts model outputs from three features: frequency, position (intercept) and actual value. The weight of each strategy was measured by how much regression accuracy dropped when one feature was removed. They tested causality by varying the synthetic data ratio (0–1%) and the entity distribution (Zipf vs. uniform), and applied the same analysis to OLMo-2 1B–32B checkpoints.

Results

Limitations

The controlled experiments used only the 163M small model and 3 tasks. Synthetic data identical to the templates was added so the small model could learn the tasks, so the conditions differ from natural pretraining. Evaluation also used the same entities as training, so memorization is possible (the authors checked this with a 20% holdout experiment in the appendix). The analysis relies on three hand-picked features.

Why it matters

It provides causal evidence that the frequency distribution of pretraining data directly shapes a model's shortcut learning. A practical implication is that adjusting data composition can make models rely less on heuristics. It also showed that small-scale experiments can predict large-scale training dynamics.

Key terms

Zipf distribution
A long-tail distribution in which a few items appear very often and most items appear rarely. Word and entity frequencies in natural language are a typical example.
Drop-one accuracy
A value that measures how much prediction accuracy drops when one feature is removed from a regression model. It shows how much that feature explains the model's behavior.
Position bias
The tendency to pick the entity that appears first or last in the prompt, regardless of the correct answer.

Original

Authors: Yik Siu Chan, Jing Huang, Yanai Elazar, Atticus Geiger · Posted 2026-11-12 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.