The FineWeb Datasets: Decanting the Web for the

The performance of a large language model (LLM) depends heavily on the quality and size of its pretraining dataset. However, the pretraining datasets for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly available and very little is known about how they were created. In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality pretraining datasets, we carefully document and ablate all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion token collection of educational text filtered from FineWeb. LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we publicly release our data curation codebase and all of the models trained during our ablation experiments.

Large Language Models (LLMs) have quickly become a ubiquitous technology thanks to their ability to competently perform a wide range of text-based tasks. A driving factor in the success of LLMs has been a steady increase in model sizes, which in turn necessitate ever-larger pretraining datasets. Beyond scale, other characteristics of pretraining data have proven to be important, including filtering out "low-quality" content and removing duplicate text. Ultimately, the curation choices made when developing a pretraining dataset can have a huge impact on the downstream capabilities and performance of an LLM. As such, pretraining dataset curation strategies are often treated as closely guarded trade secrets. In fact, there are many popular "open" language models whose parameters are publicly available but whose pretraining datasets were not released and are scarcely documented. The lack of access to high-quality large-scale pretraining datasets and lack of information about their curation has led to concerns of a growing gap between proprietary and public knowledge.

In this work, we aim to minimize this gap by developing and releasing the FineWeb datasets, a collection of large-scale pretraining datasets that can be used to train performant LLMs. Specifically, we first introduce FineWeb, a 15-trillion token dataset of text sourced from 96 Common Crawl snapshots. FineWeb is sufficiently large to train a Chinchilla-optimal model with more than 500 billion parameters. Beyond scale, FineWeb's recipe involves a principled strategy for choosing and tuning filtering heuristics that helped produce a small set of effective filters out of over fifty candidate filters from past work. In addition, we performed an in-depth exploration of how different deduplication strategies and granularities can impact performance. To validate our design choices, we ultimately demonstrate that models trained on FineWeb perform better than those trained on other public web-based pre-training datasets.

Inspired by recent work advocating for training LLMs on educational data, we additionally introduce FineWeb-Edu, a subset of 1.3 trillion tokens from FineWeb that was rated as highly educational by a custom classifier. Models trained on FineWeb-Edu exhibit significantly better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC. Both datasets are released under the permissive ODC-By License. Apart from contributing datasets, we also release datatrove, the data processing library we developed to create FineWeb. On the whole, our work represents a significant step towards improving public knowledge and resources for curating LLM pre-training datasets.

In this work, we focus on the curation of training datasets for autoregressive Transformer-based large language models (LLMs). At their core, LLMs aim to produce a distribution over the next token of text conditioned on past tokens, where each token is typically a word or subword unit. The generality of this paradigm allows LLMs to be applied to virtually any text-based task by formulating a prefix whose continuation corresponds to performing the task. Such models may undergo many stages of training including pretraining on unstructured text data, fine-tuning to improve performance on a specific task, multitask fine-tuning to improve generalization, and learning from human feedback to improve instruction-following capabilities. In this work, we focus solely on curating data for the pretraining stage.

While many sources have been considered for pretraining data including text from books, Wikipedia, and research papers, a highly common choice is to use web text, i.e. text scraped from webpages on the public internet. While some companies like OpenAI and Anthropic perform their own web scrapes, designing, implementing, and running a web scraper at scale requires significant resources and expertise. Many LLM pretraining datasets have therefore been constructed from text from the Common Crawl, a publicly available and continually updated collection of website snapshots that has been running since 2007. As of writing, Common Crawl has produced 100 web snapshots totaling petabytes of data. Although Common Crawl has produced more than enough data to train recent LLMs, it has been shown that the performance of an LLM can heavily depend on how web text has been filtered and preprocessed before being used for pretraining.

In particular, web text can contain a large amount of "unnatural" language (e.g. "boilerplate" text, gibberish, etc.). Training on unnatural language data can harm the performance of LLMs, possibly because most downstream uses of LLMs do not involve such data. On the other hand, filtering out too much content can produce a dataset that is too small to perform sufficient pretraining for a general use model. Separately, web text can contain a large amount of duplicated content, which has also been shown to be harmful in the context of pretraining data. While deduplication may seem as straightforward as "removing duplicate text", in practice many design choices must be made (line, paragraph, or document-level deduplication? fuzzy or exact matching? etc.). The curation and relative performance of different web text-based pretraining datasets therefore heavily depends on a given dataset's filtering and deduplication pipeline.

Survey of prior pipelines. OSCAR processes Common Crawl with a fastText language classifier + line-level non-cryptographic hash dedup. C4 uses langdetect + a series of heuristics (terminal-punctuation, length, "bad words" list, etc.) + 3-line-window dedup. CC-100 uses the cc_net pipeline: fastText for LID, paragraph-level hash dedup, and a Wikipedia-trained n-gram LM perplexity filter. The Pile contains "Pile-CC" from one snapshot, processed with pycld2 / jusText / WebText-similarity classifier and MinHash fuzzy dedup. RedPajama processes CC with cc_net plus a Wikipedia-vs-random classifier; SlimPajama further removes short docs and adds extra MinHash dedup. RefinedWeb uses trafilatura + fastText + MassiveText-style heuristics + MinHash + ExactSubstr dedup. RedPajama v2 ships 84 unfiltered snapshots with quality annotations (cc_net / C4 / MassiveText / RefinedWeb) and dedup labels (Bloom + MinHash). Dolma's CC subset uses fastText, MassiveText+C4 heuristics, toxicity filters, and URL/doc/paragraph Bloom-filter dedup. Among closed models, WebText scraped Reddit-linked pages, GPT-3 used a WebText-classifier + MinHash, and MassiveText (Gopher) used SafeSearch + content heuristics + MinHash.

Our design of FineWeb is primarily empirical: we performed a series of "data ablation" experiments to test different methods at each stage of the pipeline. All ablations follow an iterative process: the baseline for each subsection includes only the processing steps from previous subsections. We compare pipeline design choices by training data ablation models that are identical apart from the data they were trained on (same parameters, architecture, equal number of randomly sampled tokens). To minimize random-subset noise, we trained two models per dataset version with different random subsets and seeds, and averaged the scores.

All training used nanotron. Ablation models had 1.71B parameters (with embeddings), Llama architecture, 2048 sequence length, ~2M-token global batch size, GPT-2 tokenizer. Filtering ablations: ~28B tokens (roughly Chinchilla-optimal for this size). Deduplication ablations and final cumulative runs: 350B tokens. We trained over 70 models for a total of ~80,000 H100 GPU-hours. Evaluation used lighteval on CommonSenseQA, HellaSwag, OpenBookQA, PIQA, SIQA, WinoGrande, ARC, and MMLU (truncated to 1000 samples each).

CommonCrawl data comes in two formats: WARC (raw HTML + crawl metadata) and WET (text-only, extracted with a generic htmlparser). Similarly to Gao et al., we found that WET files retained too much boilerplate and menu text. We therefore extracted text from WARC files using the open-source trafilatura library, which by visual inspection gave less boilerplate than other libraries. Custom extraction is expensive but pays off: training on trafilatura-WARC clearly beats WET when only fastText English filtering is applied and no deduplication is done. We therefore use WARC-based data in all later experiments.

As a starting point, we applied a basic pipeline borrowed from RefinedWeb: URL filtering via a blocklist (adult content), fastText language classifier keeping only English with score ≥ 0.65, and MassiveText-style quality and repetition filters at their original thresholds. Applying this filtering to all 96 WARC-extracted snapshots produced roughly 36 trillion tokens (GPT-2 tokenizer). This step yields a clear performance uplift over unfiltered WARC.

Following RefinedWeb, we use MinHash, a fuzzy hash-based dedup technique. We collect each document's 5-grams (English word tokenizer), compute 112 MinHashes split into 14 buckets of 8 hashes each — targeting documents that are at least 75% similar. Documents sharing the same 8 MinHashes in any bucket are duplicates. We then perform transitive clustering and keep one random document per cluster.

Global dedup did not help. Our first attempt was iterative MinHash across all 96 snapshots, starting with the most recent (2023-50). For the oldest snapshots this removed up to 90% of base-filtered data, shrinking the dataset to ~4T tokens. But on a 350B-token training run, the ablation model showed almost no improvement and remained far below RefinedWeb. We then trained two probes on slices of the 2013-48 snapshot: (a) the ~31B tokens kept after global dedup ("originally kept"), and (b) 171B tokens obtained by independently deduplicating the ~460B tokens that had been removed ("originally removed"). The "originally removed" data trained a clearly better model — visual inspection confirmed the "kept" data was actually worse (more ads, gibberish keyword lists, badly formatted text).

Per-snapshot dedup matches RefinedWeb. We then deduplicated each snapshot independently with the same MinHash parameters, yielding ~20T tokens. Training on a random sample drawn from all snapshots matched RefinedWeb. Our hypothesis: the real win from dedup comes from killing the very large duplicate clusters (hundreds of thousands of docs) that recur across all crawls; pushing further to also kill small clusters (≲ 100 dups, i.e. on the order of the number of crawls) actually hurts, because such "rare cross-crawl repeats" tend to be exactly the canonical, high-quality content. Several lighter global-dedup variants we tried (Appendix E.3) all underperformed per-snapshot dedup.

At this point our pipeline matched RefinedWeb, but C4 (despite being smaller) still beat us on some benchmarks — notably HellaSwag, the highest-signal task. C4's filters drop lines without terminal punctuation, lines that mention "javascript" or "terms-of-use"/"cookie policy", short documents, and documents containing "lorem ipsum" or a curly bracket. We applied them to the 2019-18 crawl after base filtering + per-snapshot MinHash.

Results: applying all C4 filters matches C4's HellaSwag performance; the Curly Bracket filter removes 2.8% of tokens (small boost), Word Length removes 4.3% (small boost), and Terminal Punctuation alone gives the biggest single boost but removes ~30% of all tokens. The lorem_ipsum / javascript / policy rules remove < 0.5% each. We adopted all C4 filters except Terminal Punctuation — it deletes too much data for its benefit. "All-but-terminal" removes only ~7% in total.

Past heuristic filters were typically devised by data inspection. We instead use a more systematic approach: collect 50+ high-level statistics — from document-level metrics (number of lines, avg. line/word length, etc.) to inter-document repetition metrics (à la MassiveText) — on both a "high-quality" and a "low-quality" dataset. We took the independently and globally deduplicated 2013-48 snapshot as the high- and low-quality probes, respectively. For each metric, we plot the two distributions and pick a threshold targeting regions where the low-quality density is higher than the high-quality density. This gave 16 candidate metric-threshold pairs.

We then ran 28B-token ablations for each candidate on the 2019-18 crawl. Three filters were retained, with thresholds notably lighter than past work: (i) remove docs where fraction of lines ending with punctuation < 0.12 (removes 10.14% of tokens vs. C4's terminal-punctuation 30%); (ii) remove docs where fraction of characters in duplicated lines ≥ 0.1 (removes 12.47%, vs. MassiveText's 0.2); (iii) remove docs where fraction of lines shorter than 30 chars ≥ 0.67 (removes 3.73%). Together they remove ~22% of tokens and increase aggregate score by ~1% in 28B ablations — beating C4 while filtering less data.

The final FineWeb pipeline composes the choices made in the previous subsections in this exact order: (1) trafilatura text extraction on WARC files; (2) base filtering (URL blocklist + fastText English with score ≥ 0.65 + MassiveText quality/repetition filters at original thresholds); (3) per-snapshot independent MinHash deduplication; (4) all C4 heuristic filters except terminal punctuation; (5) the three custom heuristic filters identified in §3.6; (6) PII anonymisation (email + public IP via regex). Each individual step provides a measurable benchmark uplift on the aggregate of eight tasks. Applied to all 96 CommonCrawl snapshots available at the time of writing, the pipeline produces the 15-trillion-token FineWeb dataset, large enough to train a Chinchilla-optimal model with more than 500B parameters.

To benchmark FineWeb against the open ecosystem, the authors retrain identical 1.71B ablation models on 350B randomly sampled tokens from each candidate dataset and average eight benchmark scores (CommonSenseQA / HellaSwag / OpenBookQA / PIQA / SIQA / WinoGrande / ARC / MMLU). The contenders are: RefinedWeb (500B), C4 (172B), the CommonCrawl-based parts of Dolma 1.6 (3T) and 1.7 (1.2T), The Pile (340B), SlimPajama (627B), the deduplicated RedPajama 2 (20T), the English CommonCrawl slice of Matrix (1.3T), English CC-100 (70B), and Colossal-OSCAR (850B). No upsampling of any individual CC snapshot is performed. On the aggregate score (Figure 10), FineWeb finishes first among the base web datasets at ≈ 50% accuracy and FineWeb-Edu pushes the score to ≈ 52%, dominating every other open option; on MMLU specifically (Figure 11), FineWeb-Edu hits 33.6% accuracy with only 38B training tokens — Matrix needs roughly 300B tokens to reach the same point, a ≈ 10× token-efficiency gap that headlines the result. Each upstream pipeline step (base filtering → per-snapshot MinHash → C4 filters → custom filters) is independently shown to lift the FineWeb curve in Figure 9, validating that the gains compound rather than being driven by a single trick.

Looking at per-benchmark breakdowns in Figure 16: on MMLU FineWeb-Edu hits ≈ 37–38% vs. FineWeb ≈ 33% vs. Matrix ≈ 33% vs. C4 ≈ 30%; on ARC FineWeb-Edu reaches ≈ 56% vs. FineWeb ≈ 46% (an outright 10-point absolute gap that is the largest in the paper); CommonSenseQA spans 30–40% with FineWeb-Edu on top; HellaSwag is the only benchmark where FineWeb-Edu does not dominate — it lands ≈ 55–57% vs. the leaders' ≈ 62%, since highly-educational text is leaner on common-sense narrative. PIQA (70–78%), SIQA (38–42%) and WinoGrande (52–58%) cluster tightly across datasets, signalling these benchmarks are largely saturated at this scale. The Aggregate panel sums these up into the headline ≈ 52% vs. ≈ 50% gap. Importantly, since training compute is fixed at 350B tokens, the FineWeb-Edu lead is genuine token-efficiency — not merely "trained on more data".

An interesting approach has recently emerged for filtering LLM training datasets: using synthetic data to develop classifiers for identifying educational content. This technique was notably used in the non-public pretraining datasets of Llama 3 and Phi-3, but its large-scale impact on web data filtering has not been publicly explored. We applied this technique to FineWeb by filtering it with an educational quality classifier developed from synthetic annotations generated by Llama-3-70B-Instruct. The resulting dataset, FineWeb-Edu, contains 1.3 trillion tokens. FineWeb-Edu is specifically optimized for educational content and outperforms all openly accessible web-based datasets on reasoning- and knowledge-intensive benchmarks such as MMLU, ARC, and OpenBookQA by a significant margin.

To build the synthetic annotations, we used Llama-3-70B-Instruct to score 460,000 randomly sampled webpages from the FineWeb CC-MAIN-2024-10 snapshot for their educational quality on a scale from 0 to 5. We explored several prompt formats and found that the additive scale (Yuan et al.) worked best — the LLM evaluates each criterion and adds to the score step-by-step, unlike a single-rating scale that locks a category. To avoid favoring highly technical pages like arXiv abstracts, we prompted the model to focus on grade-school and middle-school level knowledge.

To scale to all of FineWeb we trained a small classifier on top of Snowflake-arctic-embed-m (a frozen embedding model): a linear regression head trained on 410,000 of the Llama-3 annotations for 20 epochs at learning rate $3 \times 10^{-4}$. We kept the checkpoint with the highest F1 on a 50,000-sample held-out validation set, treating Llama-3 annotations as ground truth. The model output was rounded to integers 0–5. We then experimented with thresholds; the chosen threshold ≥ 3 gave F1 = 82% on validation and the best trade-off between knowledge/reasoning benchmarks and HellaSwag-style tasks. Running this classifier over all 15T tokens of FineWeb took roughly 6,000 H100 GPU-hours.

At a larger ablation (1.71B model trained on 350B tokens) FineWeb-Edu surpasses FineWeb and all other open web datasets. The gains on educational benchmarks are striking: MMLU rises from 33% → 37% (≈ 12% relative) and ARC from 46% → 57% (≈ 24% relative). Even more notably, FineWeb-Edu reaches Matrix's final MMLU score with almost 10× fewer tokens — 38B vs. ~300B — demonstrating the leverage of classifier-based filtering on LLM-annotated data.

Topic distribution. Embedding 50k samples each from FineWeb / FineWeb-Edu with all-MiniLM-L6-v2 + UMAP + DBSCAN yields 100 topic clusters (labelled with Llama-3.1-70B). The classifier heavily up-samples "Education / Learning / Teaching" (+3.2 percentage points more documents in FineWeb-Edu than in FineWeb), "History / Culture / Politics" (+2.2 pp), "Health / Medicine / Biology" (+1.8 pp), and "Wildlife / Animals / Nature", "History / Royalty / Philosophy", "Water / Environment", "Energy / Environment" (each ≈ +1.2 pp). The largest down-samples are "Business / Finance / Law" (-3.2 pp), "Entertainment / Film / Theater" (-2.8 pp), "Places / Travel / Real Estate" (-2.5 pp), "Food / Summer / Recipes" (-2.1 pp), "Sports / Teams / Games" (-2.1 pp), and "Music / Entertainment / Arts" (-2.1 pp). In other words: educational filtering trades away consumer / commercial / entertainment web content in exchange for textbook-flavoured topics, which is exactly why MMLU / ARC jump and why downstream uses that want entertainment knowledge should not blindly use Edu.

Domain fit. On Paloma domains, FineWeb has lower perplexity on broad web sources (C4, mC4, Falcon, Dolma V1.5, RedPajama CC, Twitter AAE, Manosphere, Gab, 100 Subreddits, 4chan). FineWeb-Edu wins on Wikipedia-heavy sources (WikiText-103, M2D2 Wikipedia), academic sources (M2D2 S2ORC, ArXiv subset of RedPajama), and programming content (100 PLs).

Language models are known to reflect the biases present in their pretraining datasets. To provide a brief picture of dataset bias in FineWeb and FineWeb-Edu, we focus on subgroups recognised as "sensitive" or "protected" in English-speaking countries — historically subject to discrimination and disproportionately targeted by negative societal norms reflected in web text. We find that FineWeb has a relative over-representation of words reflecting hegemonic norms over-represented in online text, such as "man" and "christian". Although biases across gender, religion, and age subgroups are not strong, the most skewed associations are between religion words and intimacy, e.g. "christian dating" and "jewish singles". Fittingly, FineWeb-Edu captures associations less tied to intimacy and more expected from educational content of history and health — e.g. "man" being associated with "king" and "woman" with "pregnancy". Further details are in Appendix G.

Quantitatively, in a randomly sampled 10B-token slice of each dataset, man accounts for 67.9% of all gender-term occurrences, woman 32.0%, and non-binary only 0.1%; in FineWeb-Edu this skew gets slightly worse (man 70.2% / woman 29.6% / non-binary 0.2%), most likely because educational and historical content over-indexes on male historical figures (kings, scholars). For religion, FineWeb is heavily christianized — christian 46.1%, jewish 25.6%, muslim 16.7%, hindu / buddhist each 4.8%, atheist 1.9% — whereas FineWeb-Edu noticeably rebalances toward minority religions (christian drops to 33.1%, jewish rises to 36.3%, hindu / buddhist roughly double). On age, FineWeb is 64.5% old vs 35.5% young, while FineWeb-Edu lands at a near-even 51.5% / 48.5% — an unintended but positive side-effect of educational filtering, since textbook content uses "young people / young students" more uniformly.

The TF-IDF association heatmaps (Figs. 22–29 in the appendix) make the structure of these biases concrete: in FineWeb, jewish co-occurs with dating with TF-IDF 0.212 (the largest single cell in the religion grid) and with singles at 0.110; muslim shows muslim+ dating = 0.164; buddhist hits 0.133 with dating; even atheist is dominated by god (0.078) — implying "atheist" web text is mostly debate-style content about god. Outside religion, woman is mildly tied to dating / love / family and non-binary is highly concentrated in LGBTQ-specific vocabulary (gender / trans / queer / transgender with +0.04 to +0.06), so models trained on FineWeb describe non-binary identities in narrower, more stereotyped frames than they describe man or woman. FineWeb-Edu shifts these associations toward less-intimate, more curricular language — woman ↔ pregnancy / mother / children / family, man ↔ god / war / great / king.

To check how well each dataset covers different text domains, the authors evaluate macro-average perplexity on the Paloma benchmark with six ablation checkpoints, without decontamination. FineWeb generally achieves lower perplexity on broad web sources — C4, mC4, Falcon, Dolma V1.5, the CommonCrawl subset of RedPajama — and on noisy social-media-style sources such as Twitter AAE, Manosphere, Gab, the 100-Subreddits Paloma slice, and 4chan. FineWeb-Edu instead wins on Wikipedia-centric sources (WikiText-103, M2D2 Wikipedia), academic sources (M2D2 S2ORC, the ArXiv subset of RedPajama), and programming content (Paloma's 100 PLs benchmark). The Figure 12 panel shown above traces these curves over 50–350B training tokens: 100 Subreddits perplexity drops from ~8.5 down to ~6.0 over training; 100 PLs ranges 6–8.5; Twitter AAE 150–350; Manosphere 24–32; Gab 40–60; 4chan 15–20. In short, the two datasets divide the web cleanly — FineWeb owns "noisy general web", FineWeb-Edu owns "encyclopedic + technical".

In this paper, we developed the FineWeb datasets, a collection of large-scale LLM pretraining datasets that produce performant LLMs. Specifically, we release FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots, as well as FineWeb-Edu, a 1.3-trillion token dataset of educational content from FineWeb. FineWeb was created through a series of experiments that provided empirical evidence for our choice of text extraction strategy, deduplication procedure, and content filters. Both datasets are publicly released, along with the code and processing library that we used and all of the models we trained during our dataset ablation experiments.

While FineWeb and FineWeb-Edu attain state-of-the-art performance among public LLM pretraining datasets, we identify various paths to further improvement. First, both datasets are entirely comprised of web content scraped by CommonCrawl. It is possible that augmenting either datasets with other datatypes (books, speech transcripts, etc.) could further improve performance. In addition, most of the experiments we ran were at a smaller scale due to computational constraints. Designing datasets at more realistic scales could provide more reliable guidance. Our evaluation setup was also by necessity limited to performance on academic benchmarks without any further instruction tuning or alignment. An evaluation setup that better reflected current usage patterns of LLMs might also be more reliable. We hope that our released datasets, code, and models help further improve public knowledge and development of performant LLM pretraining datasets.

Practical takeaways for anyone curating a CommonCrawl-style pretraining corpus: (1) always extract from WARC with a real boilerplate-aware extractor (trafilatura) rather than reusing CC's WET — it is the cheapest single intervention in the entire pipeline; (2) deduplicate each snapshot independently with MinHash; do not run global cross-snapshot MinHash, because the surviving long-tail is often worse than what got removed; (3) do not blindly apply C4's terminal-punctuation rule — it deletes ~30% of tokens for a benefit the authors recover with a much softer 10%-removing alternative; (4) derive heuristic thresholds from histogram-difference between known-good and known-bad slices instead of from manual inspection; (5) scale the LLM-as-annotator pattern by distilling Llama-3-70B scores into a small classifier over a frozen embedding — the resulting FineWeb-Edu reaches Matrix's MMLU at ~10× fewer training tokens, the most striking efficiency result in the paper. Together these recipes account for essentially all of the public-dataset SOTA gap that FineWeb closed.