Dolma : an Open Corpus of Three Trillion Tokens

Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.

hf.co/datasets/allenai/dolma · github.com/allenai/dolma

Language models are now central to tackling myriad natural language processing tasks, including few-shot learning, summarization, question answering, and more. Increasingly, the most powerful language models are built by a few organizations who withhold most model development details (Anthropic, 2023; OpenAI, 2023; Anil et al., 2023; Gemini Team et al., 2023). In particular, the composition of language model pretraining data is often vaguely described, even in cases where the model itself is released for public use, such as Llama 2 (Touvron et al., 2023b). This hinders understanding of the effects of pretraining corpus composition on model capabilities and limitations, with impacts on scientific progress as well as on the public who interfaces with these models.

Our aim is to increase participation in scientific research of language models through open corpora:

• Data transparency helps developers and users of applications that rely on language models to make more informed decisions (Gebru et al., 2021). For example, models have shown to perform better on tasks that are more similar to their pretraining data (Razeghi et al., 2022; Kandpal et al., 2023), or social biases in models' pretraining data may necessitate additional consideration when using them.

• Open pretraining data is necessary to analyze how its composition influences model behavior, allowing those training models to interrogate and improve current data practices. Examples of this research include memorization (Carlini et al., 2022), deduplication (Lee et al., 2022), adversarial attacks, benchmark contamination, and training data attribution.

To support broader participation and inquiry in these lines of research, we present Data for Open Language Models' Appetite (Dolma), an open corpus of three trillion tokens designed to support language model pretraining research. We source much of our data from sources similar to those present in past work, including a mix of web text from Common Crawl, scientific research from Semantic Scholar, code from GitHub, public domain books, social media posts from Reddit, and encyclopedic materials from Wikipedia. Compared to other publicly-available pretraining corpora, Dolma offers a larger pool of tokens at comparable quality while maintaining diverse data composition. In summary, our contributions are two-fold:

• We release the Dolma Corpus, a diverse, multi-source collection of 3T tokens across over 4B documents acquired from 6 different data sources that are (i) commonly seen in large-scale language model pretraining and (ii) made accessible to the general public. Table 1 provides a high-level overview of the amount of data from each source.

• We open source the Dolma Toolkit, a high-performance, portable tool designed to efficiently curate large datasets for language model pretraining. Through this toolkit, practitioners can not only reproduce our dataset, but also study and improve data curation practices.

Table 1: The Dolma corpus at-a-glance. It consists of three trillion tokens sampled from a diverse set of domains; sourced from approximately 200 TB of raw text before curation down to an 11 TB dataset. Tokens calculated using the LLaMA tokenizer.

Closed data curation practices. Modern pretraining corpora have become increasingly opaque: proprietary systems such as GPT-4, PaLM 2 and Claude reveal essentially nothing about corpus size or provenance, and even "open-weight" models (Llama 2, Mistral, Yi, Qwen) ship without their data or a reproducible recipe. Among large-scale efforts, only a handful publish detailed curation notes: LLaMA documents methods (data not released), Gopher documents methods (neither model nor data released), and Falcon partially releases its data. This trend toward closure motivates Dolma's open-data + open-toolkit stance.

Existing open corpora and why a new one was needed. Prior open corpora have limitations: C4 (175B) and Pile (387B) are high quality but small; ROOTS (~400B) is multilingual so its English slice is only ~30%, too thin for English-only models; Falcon (580B) and RedPajama v2 (30T) hit scale but are entirely Common-Crawl — they lack source diversity (no papers, no code) and RedPajama v2 is only lightly cleaned (essentially raw CCNet output). RedPajama v1 (~1.2T) is the closest cousin and inspired Dolma, but it reproduces LLaMA narrowly; Dolma broadens to peS2o / Reddit and adds heavier cleanup. Concurrent releases (FineWeb, Zyda, LLM360 Amber/K2, MAP-Neo) appeared during review.

Four explicit design goals drove every decision. (i) Consistency with prior recipes: match sources/methods used by known corpora so Dolma can be used to scrutinize today's LMs, including closed ones; this also drives the English-only scope. (ii) Evidence-backed choices: where best practice is unsettled or implementations differ, pick the option that maximizes downstream performance via the §4.2 ablation pipeline. (iii) Scale: target 2–3T tokens so future Chinchilla-style scaling-law studies remain possible. (iv) Preserve openness: deviate from established recipes when needed for legal/ethical reasons — e.g., drop Books3 (active copyright litigation) and add PII filtering even though prior recipes rarely document it.

§4.1 The Dolma Toolkit. The pipeline is implemented as a high-performance, open-sourced toolkit with two operations: filtering and mixing. Filtering unifies language/quality/content/PII filters into a single config-driven runner — a text unit (document, paragraph, sentence), a scorer (linear classifier, KenLM perplexity, regex), and a removal policy (delete, replace-with-string). Throughput on a C4-replication benchmark was 122 CPU-hours per TB, so processing the full ~200 TB raw Dolma on a 192-vCPU c6a.48xlarge takes about 5 days. Mixing unifies up/down-sampling, deduplication and decontamination into one Rust module that streams documents across files; up-sampling = reading the same path repeatedly. A built-in Bloom filter (Bloom 1970) gives linear-time probabilistic duplicate detection — and is repurposed for test-set decontamination by seeding it with eval examples and flagging any train-side hits.

§4.2 Data ablations. Every non-trivial curation decision is validated by training a 1.2B-parameter OLMo-family decoder-only LM on the dataset under that intervention and comparing against a baseline on a fixed 8-task zero-shot evaluation suite (ARC-E/C, BoolQ, HellaSwag, OpenBookQA, PIQA, SciQ, WinoGrande). Models are early-stopped at 150B tokens — full training would be prohibitive across so many sweeps. Tasks were chosen to (i) match prior LM evaluation practice, (ii) span QA + commonsense reasoning, and (iii) avoid being contaminated by Dolma itself (full contamination report in Appendix §L). Evaluation casts each task as ranked text classification with PromptSource prompts, à la the Eleuther harness.

§5 Dolma-Web overview. The web subset contributes ~2.28T tokens drawn from Common Crawl. CC is organized into snapshots (full re-crawls over seed URLs); as of Feb 2024 there are 97 snapshots, and Dolma uses 25 snapshots spanning 2020-05 to 2023-06. Only enough snapshots were downloaded to hit the 2–3T token target, assuming ~10× shrinkage after all cleanup.

§5.1 Acquisition + language filtering (CCNet). The web pipeline begins with CCNet (Wenzek et al. 2020), the same first-stage tool used by LLaMA / RedPajama v1 / RedPajama v2. CCNet routes each page through a FastText language-ID model and Dolma keeps pages with English score ≥ 0.5 — this alone removes 61.7% of bytes. CCNet additionally removes very-common paragraphs within each snapshot shard group; that step strips ~70% of paragraphs (mostly headers and navigation chrome). End-to-end, CCNet pipeline drops Common Crawl from 175.1 TB → 27.7 TB (84.2% gone) before Dolma's own filters even start.

§5.2 Heuristic quality filters (Gopher + C4 NoPunc). Following Rae et al. (Gopher) and Almazrouei et al. (Falcon), Dolma avoids model-based quality classifiers and instead combines two heuristic stacks: Gopher All (keep all Gopher rules) plus a single C4 rule — C4 NoPunc, which drops paragraphs that do not end in punctuation. The §1 ablation shows C4 NoPunc alone beats both C4 All and Gopher All on perplexity and downstream tasks, and the combination Gopher All + C4 NoPunc is best overall. On the chosen recipe, Gopher All tags 15.23% of UTF-8 chars for removal and C4 NoPunc tags 22.73%. Separately, CCNet ships a KenLM-perplexity bucket (Wikipedia-likeness; high/med/low ≈ 21.9 / 28.5 / 49.6%); heuristic filtering left those bucket proportions essentially unchanged — suggesting model-based quality signals capture an orthogonal axis to heuristics.

§5.3 Toxicity filter (self-trained FastText on Jigsaw). To avoid downstream toxic generation, Dolma self-trains two FastText classifiers on the Jigsaw Toxic Comments corpus — one for "hate" and one for "NSFW" — and applies them at sentence granularity (BlingFire splitter), removing any sentence whose score exceeds τ. Two thresholds are ablated: "High" τ = 0.4 removes 5.5–7.3% of content; "Low" τ = 0.0004 removes 29.1–34.9%. Low generally yields slightly higher downstream accuracy, but is too costly in token count, so Dolma adopts High. The surprise from this experiment: quality / content / dedup filters overlap very little in which texts they remove, so stacking them produces a compounded reduction — future Dolma versions will start from more CC shards and tighten thresholds. (Sentence scores form a strong bimodal distribution at 0 and 1, so threshold choice mostly drives how much data is kept, not which kind.)

§5.3 (cont.) PII filtering with regex. Model-based PII detection at Dolma scale is impractical, so Dolma uses carefully-crafted regexes targeting three high-precision categories: email addresses, IP addresses, phone numbers. The policy is two-tier: ≤ 5 PII spans in a document → replace each with a special token (|||EMAIL_ADDRESS||| etc.), affects 0.02% of documents; > 5 spans → drop the whole document, affects 0.001%. Ablations confirmed the replace-vs-remove choice has no measurable effect on downstream performance — expected given how few documents are touched.

§5.4 Three-stage dedup. Prior work (Lee et al. 2022; Abbas et al. 2023; Tirumala et al. 2023) shows dedup substantially improves token efficiency, so Dolma applies it in three stages — all sharing the same Bloom-filter mixer (§4.1): (i) Exact URL dedup drops 53.2% of documents (cheap, runs first); (ii) Exact document dedup drops 14.9% of the survivors, including empties; (iii) Exact paragraph dedup drops 18.7% of paragraphs from the URL-deduped pool, including empties. Stages (i)+(ii) target straight copies (same URL re-crawled, same article on different sites) and run before content/quality filters to cut compute. Stage (iii) targets boilerplate (e.g. author bylines repeated across articles) and runs last because paragraph removal can damage discourse.

§5.5 Pipeline integration. The full Dolma-Web pipeline is therefore: CCNet output → URL dedup → document dedup → quality filters (Gopher All + C4 NoPunc) → content filters (hate / NSFW / PII) → paragraph dedup. The §3 ablation shows a clean compounding effect on HellaSwag (and the appendix shows the same on all 8 tasks): "quality only" < "quality + dedup" < "quality + dedup + content"; each layer monotonically improves the 1.2B model's downstream accuracy curve over training tokens.

§6.1 Source: The Stack. The 411B-token code subset is derived from The Stack (Kocetkov et al. 2022) — same source as StarCoder — a deduplicated collection of permissively-licensed GitHub repos snapshotted in March 2023. Files with data-heavy extensions (JSON, CSV) are filtered out up front.

§6.2 Quality filters: RedPajama-v1 + StarCoder rules. Two heuristic stacks are combined: RedPajama-v1 rules drop file preambles (license headers), excessively long lines, and mostly-numeric/template-generated files; StarCoder rules filter repos with too-few stars, files with too-few or too-many comments, and HTML with low code-to-text ratio. Ablation: RedPajama-v1 + StarCoder beats RedPajama-v1 alone on HumanEval perplexity and on the 8-task suite — so the combined recipe is adopted.

§6.3 Content filters. Same web-style PII regex (§5), plus code-specific: the detect-secrets library is run over every file, and any document containing a match (API keys, hard-coded passwords, etc.) is removed entirely.

§6.4 Dedup. The Stack itself was already deduplicated using MinHash + LSH (Broder 2002) via the Allal et al. 2023 pipeline — so Dolma-Code inherits near-dup removal from upstream, and Dolma's Bloom-filter stages act as belt-and-suspenders for any residual exact duplicates.

§7.1 Acquisition + linearization ablation. The 80B-token social subset comes from 378M Reddit posts spanning Dec 2005 – Mar 2023, sourced through Pushshift. Both submissions (top-of-thread messages) and comments (replies) are included. Because Reddit threads have tree structure, multiple linearizations are possible; an ablation compares three: (1) Atomic Content — each comment and submission is its own document; (2) Partial Threads — comments from one thread fused into a multi-turn dialogue, submissions kept separate; (3) Full Threads — submission + all child comments fused into one document. The §4 figure shows Atomic Content wins; the authors hypothesize the artificial formatting introduced when stitching thread elements actively hurts LM training. Non-English content is then removed via the §5.1 CCNet FastText filter.

§7.2 Quality filters. Building on Henderson et al. 2019: drop comments < 500 chars, drop submissions < 400 chars (submissions tend to be higher quality, so a more permissive minimum); cap document length at 40 000 chars; drop comments with < 3 net votes (deeply nested or inflammatory comments score low). Discard any document that was deleted by author, removed by moderator, or marked NSFW. Exclude any post originating in one of 26 123 banned or NSFW subreddits (the curated blocklist is published with the toolkit; any subreddit with > 10% NSFW posts is also included).

§7.3 Content filters + §7.4 dedup. Same §5.3 hate / NSFW classifier and PII regex are reused — but because Reddit documents are short, when PII is detected the entire document is removed (no in-place token replacement). Dedup is the same Bloom-filter approach as the web subset but only at document level (paragraph dedup makes little sense on such short texts) — this still effectively kills "copypasta" (the same joke text reposted across subs).

§8 Four auxiliary sources, briefly.

C4 (curated web). Mirroring LLaMA and Llama 2, Dolma appends C4 (Raffel et al. 2020) to its web subset, re-processing it through the full §5 web pipeline except URL dedup — this picks up additional low-quality / duplicate scrubbing and adds PII masking on top of the original C4 recipe.

peS2o (academic papers). The ~40M open-access papers in peS2o (Soldaini & Lo 2023) — derived from S2ORC, Semantic Scholar's open research corpus — already ship cleaned, filtered, deduplicated and formatted for LM pretraining, so Dolma ingests them as-is. This is the source of the 70B-token science slice.

Project Gutenberg (public-domain books). Project Gutenberg's archive (>70k books) was snapshotted in April 2023. Dolma keeps only English-language books (CCNet FastText, §5.1) and deduplicates on exact book-title match — yielding 6.0B tokens from 56k titles.

Wikipedia + Wikibooks (encyclopedic). Built from March-2023 Wikimedia dumps using WikiExtractor, covering the English and Simple editions of both Wikipedia and Wikibooks. Any document with ≤ 25 UTF-8-segmented words is dropped (short pages are usually template stubs or XML-parse errors). The dump is duplicate-free by construction, so no extra dedup pass is needed. Output: 4.3B tokens.

§9.1 End-to-end validation: OLMo-1B. As a final sanity check that the pipeline ships a usable corpus, the authors train and release a 1.2B-parameter decoder-only LM called OLMo-1B on full Dolma, and zero-shot it on the §4.2 evaluation suite. Numbers in Table 2:

Among truly comparable models (same parameter count and similar training-token budget), the only fair head-to-head is TinyLlama-1.1B: OLMo-1B beats it on 4/8 tasks and on average (60.3 vs 59.4). Pythia trained on ~10× fewer tokens, while StableLM2-1.6B is larger, trained on 2T tokens for two epochs, and its data composition is not disclosed — so its 66.5 is informative but not a fair comparison.

§9.2 Domain fit via Paloma. To check that Dolma's multi-source mix really produces an LM that generalizes across textual domains, the authors repeat the §4.2 ablation methodology — train six 1.2B models on 150B-token samples of C4, mC4 (English), RedPajama v1, RefinedWeb, Pile, and Dolma — and evaluate perplexity on Paloma (Magnusson et al. 2023), a stratified collection of fine-grained textual sources that samples each marked domain equally, not by its frequency in the source. Figure 5 conclusions:

• Pile gets a low perplexity despite small total scale, because its sources are themselves a hand-picked stratified mix.
• Large multi-source corpora (Dolma, and to a lesser extent RedPajama v1) match Pile's domain coverage at much greater scale.
• Single-source web-only corpora (C4, mC4-English, RefinedWeb) fit poorly to diverse domains — clearly higher perplexity.

The takeaway: when scaling tokens, the open question is "how do we keep diverse-domain sample efficiency while piling on more web data?" — Dolma's answer is to mix high-volume web with smaller curated non-web sources (papers, books, Reddit, wiki), and Figure 5 shows OLMo-1B's perplexity curve on Paloma tracks the Pile-trained model despite carrying much more web data.

Conclusion. The paper releases Dolma, a 3T-token English pretraining corpus drawn from six sources (web, scientific papers, code, public-domain books, social media, encyclopedic), and documents the full curation pipeline alongside ablation evidence for every non-trivial choice. Both the corpus and the toolkit are open-sourced as part of the OLMo project (Groeneveld et al. 2024). Since the manuscript was written, the team has continued shipping improvements — e.g. Dolma v1.7 yields meaningful downstream gains under a fixed model. The corpus is released under ODC-By and the toolkit under Apache 2.0. The stated hope: close the data-availability gap between commercial and "open" LMs and enable scientific work on transparency, reproducibility and data-centric LM research.

Limitations. (i) English-only. Dolma was scoped to English; FastText LID has false negatives, so trace non-English data may remain, but is too sparse to enable real non-English downstream tasks — and by being English-only, Dolma reinforces the "English-as-default" assumption in NLP. (ii) Representativeness. No single open corpus can cover all data practices, and many open/closed LMs train on data that simply cannot be redistributed. (iii) Ablation model size. Validation uses 1.2B dense decoder-only models — most production LMs sit at 7B–70B, and alternative architectures were not explored. So conclusions may not transfer cleanly to larger scales; downstream developers are expected to re-validate. (iv) Limited eval suite. The 8 tasks were chosen partly because Dolma is provably uncontaminated against them — but they miss capabilities like executable code generation, which only emerge after instruction-tuning. (v) Manual inspection infeasible. At 11 TB / 4B docs, programmatic tools (WIMBD, Data Portraits) help but no human can audit Dolma exhaustively — so the corpus' full distributional and harm profile remains an open question.

Ethics: harm minimization + copyright. The team consulted legal/ethics experts throughout, made case-by-case calls (mask known-precise PII, take a "measured" position on toxicity where literature disagrees), and provides a data-removal request form. The authors explicitly accept compromising on reproducibility/performance/extensibility when there is meaningful individual harm. On copyright: the legal status of training on copyrighted material under U.S. fair use is unsettled (Cooper et al. 2023; Lee et al. 2024) and varies sharply by jurisdiction — Israel and (currently) Japan permit AI training on copyrighted content; the U.S. position is still being litigated. Most Dolma sources were chosen for permissive licensing (peS2o = open-access; The Stack = permissively licensed; Wikipedia = CC) but large web crawls likely still contain copyrighted text — at this scale, reliable detection is infeasible. The decision to release was based on (a) all sources being publicly available, (b) all sources already being used in large-scale LM pretraining elsewhere, and (c) the team's commitment to track the rapidly-evolving legal landscape.