Paper review · October 13, 2026
Host-Aware Web Crawling for LLM Pretraining Data
Using the finding that about half of web document quality varies at the host level, the authors show that sampling only a few pages per host and then crawling good hosts deeply increases the yield of high-quality pages several times over.
Problem
Much of LLM pretraining data comes from web crawling, but most collected pages are discarded by quality filters. Little research has examined how a crawler should allocate a limited budget across hosts, and how deeply, to collect more high-quality data.
Method
The authors extracted 3.01 million documents (532,628 hosts) from 224 WARC files (about 0.25%) of Common Crawl CC-MAIN-2024-10. They scored these with 10 quality and topic classifiers, including DCLM, FineWeb-Edu, Nemotron-CC and GneissWeb. They combined these scores with web-graph centrality (harmonic centrality, PageRank) and Common Crawl's per-host crawl coverage. The analyses are ICC1 variance decomposition, simulations of crawl yield by host ordering, simulations of crawl-depth policies, and correlation analysis between actual crawl budget and quality density.
Results
- Across all 10 classifiers, the between-host share of document score variance (ICC1) was 0.43–0.62. About half of quality variation arises between hosts.
- At a 10% crawl budget, crawling in harmonic-centrality order collected 0.7–1.8× as many high-quality pages as random. An oracle that knows host quality collected 4.0–8.0×. Under DCLM, centrality ordering did worse than random.
- The quality of a 3-page sample per host correlated with the quality of the remaining pages at Spearman ρ=0.53. A sample-then-deepen policy using this collected up to 7.8× more high-quality pages per fetch than a uniform-depth policy at small budgets (1.7× at larger budgets), coming close to the oracle.
- Across more than 5.65 million hosts, the correlation between Common Crawl's per-host crawl coverage and quality density was -0.08. The authors estimate about 58 million high-quality pages among URLs that were known in that crawl but not collected.
- The authors show that language composition distorts the centrality-quality correlation; for example, DCLM rates non-English hosts higher than English hosts (median 0.29 vs. 0.0005).
Limitations
"Quality" is a proxy metric: the pretraining usefulness predicted by a classifier. The authors did not pretrain actual models on the crawl results to compare performance. The classified sample is about 0.25% of all WARC files. The estimate of uncrawled high-quality pages assumes that uncrawled pages have the same quality density as crawled pages from the same host.
Why it matters
Quality control for pretraining data can be moved earlier, from filtering after the crawl to the crawl stage itself. Only a few pages per host need to be classified, so the compute cost is low. The approach can be applied directly to running large-scale web data collection pipelines.
Key terms
- ICC1 (intraclass correlation coefficient)
- The share of total score variance that comes from differences between groups (here, hosts). The closer it is to 1, the more uniform page quality is within the same host.
- Harmonic Centrality
- A centrality metric that measures how closely a node is connected to other nodes in the web graph. Common Crawl uses it to set crawl priorities.
- Quality classifier
- A model that reads a document's text and predicts a score for how useful it is for pretraining. fastText-based classifiers are common.
- Sample-then-deepen
- A policy that first collects only a few pages from each host to estimate quality, then spends more of the crawl budget on hosts with high scores.
Original
Authors: Malte Ostendorff · Posted 2026-10-09 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.