We all know LLMs are trained on internet-scale data. But do you really know what happens under the hood? Which algorithms? What does the preprocessing pipeline look like? How do you measure if a dataset is even good?
HuggingFace published a detailed write-up on exactly this , how they built FineWeb, a 15T token open dataset for LLM pretraining. They open-sourced the data, the models, and the full methodology. I would strongly recommend reading the original article.
Illustration of data filtering and preprocessing for LLM training, showing raw text tokens being refined into a high-quality dataset via a digital funnel
Whenever I go through a research paper or technical blog, I look for the tools, algorithms, and design decisions I can reuse. You may never train an LLM end-to-end, but these patterns show up in data-heavy ML work all the time.
Here are my key takeaways.
LLM training: quick context
Two phases: pre-training and post-training.
Pre-training is next-token prediction at scale. Raw text, trillions of tokens. This is where the model learns language itself.
Post-training is where you shape behaviour i.e., instruction following, RLHF, DPO, GRPO, etc.
FineWeb is a pre-training dataset. It’s the fuel, before any fine-tuning work begins.
Tools used across the pipeline
-
datatrove — HF’s own library for large-scale text processing. Filtering, deduplication, pipeline orchestration. Built to run at petabyte scale.
-
nanotron — pretraining library. Simplicity and performance were the design goals. Used to train small validation models throughout the experiments.
-
lighteval — evaluation framework, 1000+ tasks supported. Used to measure whether one version of the dataset produces a better model than another.
-
trafilatura — HTML-to-text extraction. Converts raw HTML into clean, structured text.
-
UT Capitole blacklist — a list of known bad URL categories. Used upfront to filter out spam, adult, and malicious domains.
CommonCrawl
CommonCrawl (CC) is the main raw material for LLM pretraining. They crawl the public web every ~2 months. 96+ dumps exist so far.
Two formats:
-
WARC — raw crawl data. Full HTML, HTTP headers, everything.
-
WET — pre-extracted plain text. No HTML.
WET sounds like the easier choice. HF went with WARC instead, extracted text using trafilatura, and showed empirically it was higher quality than WET.
Why? WET is generated by a generic extractor that does not distinguish between actual content, nav menus, cookie banners, and footer boilerplate. trafilatura is built for article and content extraction. Better signal-to-noise.
Don’t default to the pre-processed option just because it’s easier. Sometimes doing the extraction yourself gives real quality gains.
Base filtering
First pass over raw CC data:
-
URL blacklist filtering: drop anything matching known spam/adult/malicious domains
-
Language filtering:fastText language classifier, keep English docs with score >= 0.65
-
Repetition filtering: remove docs with high rates of repeated lines, paragraphs, or n-grams. High repetition is a reliable signal for low-quality scraped or generated content.
Deduplication: more is not always better
HF used MinHash for deduplication, a fuzzy hashing technique that finds near-duplicate documents even without byte-for-byte matches. Each document is represented as a compact hash signature, and LSH (Locality Sensitive Hashing) is used to efficiently find high similarity pairs.
The naive assumption is that more deduplication means cleaner data means better model. That’s not what the experiments showed.
Aggressive deduplication can hurt model performance. If genuinely useful content appears many times across the web like a solid tutorial, a key explanation — aggressive dedup removes most of those copies. The dataset becomes “clean” in the fuzzy-duplicate sense but loses coverage of important knowledge.
They ran controlled experiments across deduplication thresholds using small proxy models, trained and evaluated end-to-end. The sweet spot was per-dump deduplication (within each CC dump independently), not global deduplication across all dumps.
Every filter has a recall cost. Know what you’re trading off before cranking up the dedup threshold.
C4 filters
C4 is another CC-based dataset with a set of heuristic filters, both line-level and document-level. HF systematically evaluated which C4 filters actually improved downstream model quality and which didn’t.
One they dropped: terminal punctuation filtering, which removes any line not ending with ., ?, or !. Sounds reasonable. In practice it kills a lot of real content – code, tables, headers, bullet lists, non-English sentence structures. It hurt more than it helped.
The only honest way to know which preprocessing decisions matter is to run the filter, train a small model, evaluate on benchmarks, and compare. No shortcut.
FineWeb: the final recipe
15T tokens, ~44TB on disk.
Pipeline:
-
WARC extraction via trafilatura
-
URL blacklist filtering
-
fastText language filter (>= 0.65 English)
-
Repetition removal
-
Per-dump MinHash deduplication
-
Subset of C4 filters (terminal punctuation filter excluded)
-
Custom heuristic filters
Every step in this list was validated through actual model training and eval. Nothing was assumed to be obviously good.
FineWeb-Edu: quality over quantity
The hypothesis was that a smaller, highly curated dataset can outperform a massive generic one.
To filter for educational quality at scale, they built the following pipeline:
Step 1 — LLM annotation
Used Llama-3-70B-Instruct to score 500K FineWeb samples on a 0-5 scale for educational quality. 0 = useless. 5 = textbook-quality instructional content.
Step 2 — Train a cheap classifier
Running Llama-3–70B over 15T tokens is not feasible. So they used those 500K annotations to fine-tune a much smaller model — Snowflake Arctic Embed (medium) with a single regression head added on top. Embedding and encoder layers were frozen. Only the regression head was trained. 20 epochs, lr = 3e-4, 450K training samples.
Step 3 — Score everything
The lightweight classifier scored all of FineWeb and predicted an educational quality score per document.
Step 4 — Threshold
A threshold of 3 converted the continuous score into a binary keep/discard decision. F1 of 82% on the validation set.
Results: FineWeb-Edu outperforms FineWeb and all other open web datasets on MMLU, ARC, and OpenBookQA. It achieves the same benchmark performance with 10x fewer tokens compared to C4.
Why regression and not multi-class classification?
This is the part that made me stop and think.
You have 500K samples labeled with scores 0, 1, 2, 3, 4, 5. Six classes. The obvious framing is multi-class classification with a softmax head.
HF used regression instead and I think this is the right call.
In multi-class classification with cross-entropy loss, each label is treated as independent. The loss function has no concept of how far a prediction is from the truth. Predicting 0 when the true label is 5 gets the same penalty as predicting 4 , both are just “wrong class.”
In regression with MSE loss, the penalty is proportional to the squared distance. Predicting 0 on a true label of 5 gives a loss of (5-0)^2 = 25. Predicting 4 on a true label of 5 gives (5-4)^2 = 1. The model is explicitly taught that these labels have an ordering, and that some mistakes are far worse than others.
That ordinal structure (4 is closer to 5 than 0) is real information. Classification throws it away. Regression respects it. For a quality score where the distance from truth matters, regression is the right formulation.
Closing thought
HF trained hundreds of small models just to validate preprocessing decisions. That’s expensive but it’s the only honest approach. The alternative is trusting your intuition about what makes data “good”, and that’s riskier than it looks.
All datasets, models, and methodology are open. Worth reading the full article if you work on large-scale text pipelines.
Links: datatrove / nanotron / lighteval / trafilatura / FineWeb article