Paper review · October 29, 2026

Lexical traces left by pretraining data

The Lexical Imprint of Pre-training Data

Language models Data & evaluation Data engineering Intermediate Upvotes 7 cs.CL HyperCLOVA X

The authors measure word-level unigram cross-entropy (UCE) between a pretraining corpus and a benchmark, show through controlled pretraining experiments that lower values go with higher zero-shot accuracy, and apply this to data selection and drafter routing.

Problem

It is hard to know how much of zero-shot benchmark accuracy is explained by how well the word distributions of the pretraining corpus and the benchmark line up. Earlier analyses examined already-trained checkpoints after the fact, so corpus effects were mixed up with training recipe, scale and history.

Method

UCE is defined from the word frequency distributions of the corpus and the benchmark. It needs no tokenizer or model, and up to a constant it equals KL divergence. With the training recipe fixed, the authors pretrained from scratch on 4 corpora (C4, DCLM, FineWeb-Edu, OpenWebText) at 3 model sizes (400M, 1.33B, 3.36B) and evaluated on 10 benchmarks. They then selected data with UCE-filter, which removes the lines in each document that raise UCE the most, and tested speculative decoding routing that picks the domain drafter with the lowest UCE for each prompt.

Results

In the setting where a 3.36B model was trained on 60B tokens, UCE rankings matched the accuracy rankings of the three corpora on all 8 benchmarks. Contrary to their quality reputations, C4 had lower UCE and higher accuracy on HellaSwag and PIQA.
When a 400M model was trained on 16 subsets of 8.5B tokens each, the inverse relationship between UCE and accuracy held (ARC Easy R²=0.928, PIQA R²=0.805).
On 7 target tasks, UCE-filter lowered UCE from 11.17 to 10.92 on DCLM and from 11.32 to 10.93 on FineWeb-Edu, lower than DSIR (11.08, 11.20). The authors reported that this selection improved target accuracy.
Per-prompt drafter routing was reported to raise throughput over a general-purpose drafter baseline. Specific numbers are not in the provided text.

Limitations

The authors state that corpus-level relationships are tangled with curation and genre differences and are only correlational. Unigram statistics are a coarse proxy that does not capture context. Experiments go up to a 3.36B model and 70B tokens. UCE-filter requires knowing the target benchmarks in advance, so the authors themselves do not present it as a standalone data selection method.

Why it matters

Controlled experiments show that part of benchmark scores comes from simple alignment of vocabulary distributions, which is a caution for evaluating pretraining data quality and interpreting benchmarks. Because it is a cheap metric that can be computed without a model, it can be used directly in data mixture design and domain routing.

Key terms

Unigram cross-entropy (UCE): The cross-entropy of encoding the benchmark's word distribution with the corpus's word distribution. Lower values mean the two distributions are closer.
Zero-shot likelihood-based evaluation: An evaluation method that computes the model's conditional likelihood for each answer choice and picks the most likely one.
Speculative decoding: A technique that speeds up generation by having a small drafter model propose tokens in advance and a large model verify them in a single pass.

Original

Authors: Jeonghoon Kim, Woojin Chung, Woomin Song, Cheonbok Park, Nako Sung, Jinwoo Shin · Posted 2026-02-11 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.