Extending English quality classifiers to data selection for multilingual LLM pretraining

Problem

Using quality classifiers to select only good data from web corpora has worked well for English. Low-resource languages, however, lack quality-labeled data. Existing multilingual methods either need training data for each language or have only been validated on about 17–20 high-resource languages.

Method

The authors used Qwen3-32B to translate 0.1% of the data for the top 100 languages in FineWeb 2 into English (about 5 million documents). They scored the translations with the English quality classifiers FineWeb-edu and DCLM. Using these scores as labels and embeddings of the original multilingual text as input, they trained an MLP. The MLP has two hidden layers, and the embedding model (XLM-RoBERTa or mmBERT) was kept frozen. They then pretrained 1B, 3B and 8B LLMs on the top 10% of data selected this way and compared performance.

Results

• For the 1B model (100B tokens) on English plus the top 20 languages, the average scores of the four combinations (mFW-edu·mDCLM × XLM-R·mmBERT) ranged from 0.4087 to 0.4145, a small spread.
• Even a classifier trained only on English scored well above the heuristic filtering baseline (FW+FW2). It was also on par with other model-based filtering methods. Adding more training languages gave small improvements.
• At the 1B, 3B and 8B scales, the method performed on par with the strongest baseline (FWHQ+FW2HQ+). The authors reported that mFW-edu had the highest average score at 1B, and mDCLM at 3B and 8B.
• The authors reported no performance drop on regional and cultural knowledge benchmarks (INCLUDE, CulturalBench, etc.).
• The authors compared scores on synthetic encyclopedia-style data with scores on web samples, and analyzed correlation with LLM-as-a-judge scores. Based on this, they interpreted the classifier as following the scoring criteria of the original English classifier even in languages not used to train it.

Limitations

LLM training was evaluated only on English plus the top 20 languages. Languages ranked 21–100 were evaluated only indirectly, by comparing scores on synthetic data in 8 languages, and languages beyond the top 100 only through correlation with LLM-as-a-judge scores. The closest prior work, MuRating, could not be compared directly because its code and data are not public. The quality criteria are inherited directly from the English classifiers, and results also depend on machine translation quality. Translation took about 750 H100 GPU hours.

Why it matters

When building a multilingual LLM, existing English classifiers can be reused to select data without creating separate quality labels for each language. This can lower the cost of data curation for low-resource languages. The classifier weights are also released, so they can be tried right away.

Key terms

Model-based quality filtering

A classifier assigns a quality score to each web document, and only the top-scoring documents are kept as pretraining data. FineWeb-edu and DCLM are well-known examples.

Cross-lingual generalization

The property that criteria learned in one language (usually English) carry over to other languages not used in training.

Frozen embeddings + MLP

An approach that keeps a pretrained encoder (such as mmBERT) frozen and trains only a small neural network on top of its embeddings. It has low compute cost, and the embeddings are easy to reuse for other tasks.

LLM-as-a-judge

A method in which an LLM is asked to score text quality, and those scores are used as an evaluation standard.

Original

• arXiv abstract
• PDF
• HTML full text
• Hugging Face

Authors: Vinko Sabolčec, Bettina Messmer, Yassine Turki, Martin Jaggi · Posted 2026-10-08 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.