DataComp-LM: In search of the next generation of

We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters.

As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-BASELINE, enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-BASELINE represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute.

Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6× less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation. We release the DCLM benchmark, framework, models, and datasets at https://datacomp.ai/dclm.

Large training datasets are an important driver of progress in the recent language modeling (LM) revolution. As the cost of training state-of-the-art language models continues to grow, researchers increasingly focus not only on scaling but also on improving training datasets that enable efficient generalization on a wide range of downstream tasks. Indeed, there is a growing number of proposals for filtering data, removing (near-) duplicates, finding new data sources, weighting data points, generating synthetic data, and so on.

A key challenge in this emerging research area is a lack of controlled comparisons. While the aforementioned proposals generally use the same evaluation datasets, researchers often compare models that are trained with different architectures, compute, or hyperparameters. Hence, it is often unclear what data curation strategies work best: Are the results of training set A better than training set B because training set A is truly better, or because the model trained on A was combined with a better architecture, learning rate schedule, or more compute? Disentangling the many factors influencing the quality of a language model is crucial to understanding which data curation strategies work best and ultimately building better language models.

Beyond the lack of standardized benchmarks, another challenge for research on training data is that details about training sets are becoming increasingly rare, even for open weight models such as the Llama, Mistral, or Gemma models. For all of these models, the training sets are not publicly available, and the corresponding model documentation only provides a coarse description of the respective training data, if any at all. As a result, it is currently unclear what ingredients constitute a state-of-the-art training set for language models.

To address these challenges, we introduce DataComp for Language Models (DCLM), the first large-scale benchmark for language model training data curation. In DCLM, researchers propose new training sets and data curation algorithms and then evaluate their datasets by training LMs with a fixed training recipe on their data. By measuring the performance of the resulting model on downstream tasks, researchers can quantify the strengths and weaknesses of the corresponding training set.

To enable DCLM, we contribute a comprehensive experimental testbed. A key component is DCLM-POOL, a corpus of 240 trillion tokens derived from Common Crawl. DCLM-POOL is the largest public corpus for language model training and forms the cornerstone of the DCLM filtering track, where participants aim to curate the best possible training set out of DCLM-POOL. In addition, we provide open-source software for processing large datasets with several filtering approaches.

The high cost of training language models makes it necessary to understand the performance of training recipes across different compute and data scales. Hence, our third contribution is an investigation of scaling trends for dataset design. We find that models as small as 400M parameters can still provide signal on which training sets perform better at larger scales. Based on our experiments, we organize DCLM into five compute scales spanning a range of about 600× in compute from 400M parameter models to over-trained 7B models. This multi-scale design makes DCLM accessible to researchers with varying compute budgets.

As a starting point for DCLM, we conduct 416 baseline experiments with different training sets and compute scales. Our experiments identify model-based filtering as a key component for effective data curation. We also show that details of the filtering model can have a large impact on performance, ranging from 35% to 44% accuracy on MMLU 5-shot at the 7B parameter scale (280B training tokens). Interestingly, a simple bigram classifier, combined with a carefully selected set of positive and negative examples, performs best among the classifiers we experimented with. In addition, we find that human quality judgments have only limited value in identifying high-quality training data.

Finally, we combine our results into DCLM-BASELINE, a new state-of-the-art public training set for language models. When training a 7B parameter language model on 2.6 trillion tokens using DCLM-BASELINE, the resulting model achieves 64% on MMLU, which is state-of-the-art among open-data models and close to models such as Mistral-7B-v0.3 (63%) or Llama 3 8B (66%) that are trained with up to 6.6× more compute (Llama 3 8B). Compared to Llama 2 7B, training a 7B parameter model on 280B tokens from DCLM-BASELINE achieves 5 pp higher MMLU while being trained with 7× less compute. As our 7B model uses a standard decoder-only Transformer, our results also highlight that a systematic approach to data curation is key to training performant language models.

We publicly release our DCLM framework, models, and training sets at https://datacomp.ai/dclm to enable other researchers to participate in DCLM and to strengthen the empirical foundations for data-centric research on language models.

Data curation for language models. To collect large datasets for training LMs, researchers typically resort to web crawls. However, these crawls often contain considerable amounts of undesirable content, necessitating curation to develop high-quality training data. Most data curation efforts focus on methods for improving model performance, including filtering by language, heuristic-based filtering, quality filtering, data deduplication and mixing. While prior work examines a limited set of filters, we conduct the largest public investigation of data curation, resulting in a strong DCLM-BASELINE dataset. Since the initial release of DCLM, newer works such as WebOrganizer, Nemotron-CC, and Olmo-2 have also built upon our benchmark or curation strategies to further advance the state-of-the-art for LLM pre-training datasets.

Open-source datasets. As the scale of LMs has increased over the past years, the community has curated larger datasets to match. Early works include the C4 dataset with 160 billion (B) tokens and The Pile with 300B tokens. More recently, RefinedWeb contains 600B tokens, Dolma 3 trillion (T) tokens, FineWeb 15T tokens, and RPv2 30T tokens. There are also large domain-specific datasets, such as the code-focused StackV2 with 900B tokens, as well as high-quality filtered subsets such as FineWeb-Edu with 1.3T tokens. We include performance comparisons with various datasets in Figure 1 and examine FineWeb's LightEval evaluation framework more closely in Appendix G. We release the largest pool of raw text data to date with 240T web-crawled tokens. We also release DCLM-BASELINE, a 3.8T token high-quality dataset from our pool that yields better models than prior datasets.

Data-centric benchmarks. Past work on benchmarking data improvements includes dataset distillation, curriculum learning, and transfer learning. In DataComp and DataPerf, participants iterate on a dataset with a fixed model and training recipe for vision, vision-language, and speech tasks. For LMs, the Data-Juicer effort includes benchmarks for cleaning and mixing fine-tuning data while the BabyLM challenge Loose track focuses on efficient development of 125M to 220M parameter LMs pretrained on 10M to 100M tokens. With a 240T token pool and 7B models, DCLM is the largest data-centric benchmark for language models.

This section describes the main components of DCLM. We start with DCLM-POOL, the raw text corpus underlying our benchmark (§3.1). We then describe the DCLM workflow, visualized in Figure ②: selecting a competition scale (§3.2), curating a dataset by filtering DCLM-POOL and potentially mixing in other sources (§3.3), training a model with fixed hyperparameters (§3.4), and evaluating the model to score the dataset (§3.5).

§3.1 DCLM-POOL. DCLM-POOL is an unfiltered web-text corpus comprised of all Common Crawl data prior to 2023. Based on §4.2, we re-extract text from HTML using resiliparse instead of using Common Crawl's pre-extracted text. DCLM-POOL contains 200B documents (370TB after gzip compression), resulting in 240T GPT-NeoX tokens.

Decontamination. Test set samples often contaminate language model training sets; however, the effect of such samples on downstream performance remains largely unclear. To allow researchers to better understand contamination, we release decontamination tooling instead of decontaminating DCLM-POOL directly. We implement our own decontamination process for two popular tasks, MMLU and HellaSwag, and provide tooling to examine datasets for overlap with all of our test sets. We ask all submissions to disclose a decontamination report and avoid using highly-contaminated data.

§3.2 Competition scales. To ensure DCLM is accessible to researchers with different compute constraints and to facilitate the study of scaling trends, we create different competition scales spanning three orders of compute magnitude. Each scale (i.e., 400M-1x, 1B-1x, 3B-1x, 7B-1x, and 7B-2x) specifies the number of model parameters (e.g., 7B) and a Chinchilla multiplier (e.g., 1x). The number of training tokens for each scale is $D = 20 \times N \times \text{mult}$ so that a multiplier of 1x corresponds to a compute allocation near-optimal under Hoffmann et al.

A potential pitfall in our multi-scale design is that the ranking of data curation methods may change when increasing the compute scale. To better understand this concern, we plot the performance of 10 methods at the 7B-1x scale as a function of their performance at smaller scales (Figure 3). We find high rank correlation between the results for smaller scales (400M-1x, 1B-1x, 3B-1x) and those for the larger 7B-1x scale (Pearson's $r = 0.838$, $r = 0.956$, $r = 0.982$ respectively), suggesting better curation strategies at smaller scales transfer to larger scales.

§3.3 Benchmark tracks. After choosing a scale, participants choose one of two tracks. (i) In the filtering track, participants propose algorithms to select training data from a candidate pool. We start with five pools, one for each scale, which are random document subsets of DCLM-POOL. We restrict initial pool sizes by scale to encourage scalable filtering strategies and reflect realistic data download and storage constraints. (ii) In the mixing track, a submission may combine documents from DCLM-POOL with those from any external sources (i.e., outside of Common Crawl). For instance, participants can add in documents directly curated from StackExchange and Wikipedia data dumps (as is done by RPJ) or perhaps even their own custom crawl.

§3.4 Training. To isolate the effect of dataset interventions, we fix a training recipe at each scale. Based on prior ablations on model architectures and training, we adopt a decoder-only Transformer (e.g., GPT-2, Llama), implemented in OpenLM. We also provide unified data processing utilities.

§3.5 Evaluation. Our full evaluation suite, based on LLM-Foundry, contains 53 downstream tasks suitable for base model evaluation (i.e., without finetuning): from question answering to open-ended generation formats, considering varied domains like mathematics, text-book knowledge, and common-sense reasoning. To evaluate data curation algorithms, we focus on three main performance metrics. First, we consider MMLU 5-shot accuracy, which is widely used to compare state-of-the-art models like GPT-4 and Llama 3 70B. Second, we propose the CORE centered accuracy, computed over a subset of 22 tasks (e.g., HellaSwag and ARC-E) that provide a low-variance signal even at small scales, linearly rescaling the accuracy per task so that 0 corresponds to random guessing and 1 corresponds to perfect accuracy. Finally, we report the EXTENDED centered accuracy, which averages the centered performance for all of our 53 tasks.

We now show how the DCLM workflow can lead to high-quality datasets and quantify the effects of data curation methods. This section describes the process of converting Common Crawl into our dataset, DCLM-BASELINE. We first evaluate open-source datasets as a starting point (§4.1). Next, we experiment with alternatives for several key phases of dataset construction: text extraction (§4.2), deduplication (§4.3), and model-based filtering (§4.4). We then experiment with mixing in high-quality sources (§4.5) and provide a contamination analysis (§4.6).

§4.1 Evaluating existing datasets. We evaluate several well-known open-source datasets (C4, RefinedWeb, RedPajama, and Dolma-V1). While all four datasets use various heuristic filters and data cleaning steps, we find that RefinedWeb performs the best on our CORE and EXTENDED metrics at the 7B-1x scale (CORE: C4 34.2, Dolma-V1 35.0, RedPajama 35.3, RefinedWeb 36.9). Interestingly, RefinedWeb is solely filtered from Common Crawl, unlike RedPajama and Dolma-V1, which additionally mix in curated, "high-quality" sources like Wikipedia. The comparison suggests the relative strength of filtering. Takeaway: For DCLM-BASELINE we adopt RefinedWeb's heuristic filters.

§4.2 Text extraction. Text extraction is a crucial early processing step that pulls content from raw HTML. To understand its effect, we compare three approaches: resiliparse, trafilatura (used by RefinedWeb), and the Common Crawl-provided WET files that contain pre-extracted text. We then apply RefinedWeb's heuristic quality filters to each. We find both resiliparse and trafilatura improve CORE by at least 2.5 points over the WET extraction. This is significant because most open source datasets, including C4, RedPajama, and Dolma-V1, use WET files, which could partially explain their worse performance. While resiliparse and trafilatura have similar downstream performance, resiliparse is 8× faster to run and hence more practical for large-scale processing. Takeaway: We use resiliparse for DCLM-POOL.

§4.3 Deduplication. Web-crawled datasets often contain many duplicate or near-duplicate data strings. Removing these duplicates serves the dual purpose of improving performance by reducing memorization and increasing data diversity. For deduplication, we explore MinHash, as part of a suffix array pipeline, and near-duplicate Bloom filtering, which modifies an exact document and paragraph deduplication scheme. We find that both approaches provide comparable downstream performance: within 0.2 CORE percentage points at the 7B-2x scale. However, our modified Bloom filter scales more easily to datasets surpassing 10TB. Takeaway: We use a Bloom filter for DCLM-BASELINE and MinHash for other experiments.

§4.4 Model-based quality filtering. Recent literature indicates that training models to serve as quality filters leads to downstream improvements. We compare many strategies: 1) PageRank score filtering; 2) Semantic Deduplication (SemDedup); 3) linear classifiers fit on pre-trained BGE text embeddings; 4) AskLLM which prompts an LM to see if a document is helpful; 5) Perplexity filtering where we retain low perplexity sequences following CCNet; 6) Top-k average logits; and 7) fastText binary classifiers to distinguish data quality. For training classifiers, we train on $\sim$400k documents split equally between positive and negative classes. We compare the approaches and find that fastText-based filtering outperforms all other approaches (fastText OH-2.5+ELI5 30.2 CORE vs. Top-k avg logits 29.2, Perplexity 29.0, AskLLM 28.6, BGE-linear 27.2, SemDedup 27.1, RefinedWeb 27.5, PageRank 26.1 at 1B-1x).

Text classifier ablations. To better understand the limits of fastText, we train several variants, exploring different choices for the reference data (i.e., the examples given positive labels), filtering threshold, and feature space. For reference data, we considered commonly used sources like Wikipedia, OpenWebText2, and RedPajama-books, following the reference data used for GPT-3. We also try a novel approach, using instruction-formatted data, drawing examples from OpenHermes 2.5 (OH-2.5) and high-scoring posts from the r/ExplainLikeImFive (ELI5) subreddit. Overall, we find, when controlling for other hyperparameters, the fastText OH-2.5+ELI5 approach gives a 3.5 percentage point lift on CORE compared to the other more conventional choices. We also observe that using a fairly strict threshold, which keeps the top-10% of examples, helps over more permissive top-15% and top-20% thresholds. Takeaway: For DCLM-BASELINE we use fastText OH-2.5+ELI5 classifier score to keep the top 10% of documents.

§4.5 Dataset mixing. Often, Common Crawl (CC) is combined with other data sources that are considered high-quality (e.g., Wikipedia, StackExchange, and peS2o). Since DCLM participants can include additional data sources in our mixing track, we examined the potential benefits of adding high-quality sources to training sets derived from Common Crawl only. We compare a model trained on 100% filtered CC data to models trained with the mixing ratios from Llama 1 and RPJ: 67% CC, and 33% from Wikipedia, Books, Stack Exchange, arXiv, and GitHub. For the CC component, we consider different variants: a subset of our DCLM-BASELINE, RPJ's CC portion, RefinedWeb, and C4. The results show that mixing improves performance for the lower-performing CC subsets (C4 +2.2, RPJ-CC +1.7, RefinedWeb +1.4 CORE). In the case of DCLM-BASELINE, however, mixing actually hurts performance on average ($-1.2$ CORE), which suggests it can be counterproductive given performant filtering.

§4.6 Decontamination. Here, we examine whether contamination of our pretraining data with evaluation data influences our results for DCLM-BASELINE. We focus on MMLU and HellaSwag. Specifically, we attempt to remove examples from these two test sets that exist in DCLM-BASELINE. For both, we flag pages that contain the question text along with at least one of the corresponding answer options. We then train a 7B-2x model with our DCLM-BASELINE without the detected overlaps. This does not lead to decreases in model performance (MMLU 51.8 → 52.7, HellaSwag 77.9 → 78.4), so our performance gains on these two tasks are not likely to be caused by increased presence of their test examples in our dataset. We also apply the same strategy to Dolma-V1.7 and FineWeb-Edu and observe DCLM-BASELINE has roughly similar contamination stats.

Here, we test if datasets that perform well on the DCLM benchmark also maintain their strength with an order of magnitude more compute. To ensure our trained model is broadly useful, including for math and coding tasks, we combine our 3.8T DCLM-BASELINE with the StarCoder and ProofPile2 datasets to arrive at a 4.1T token dataset. We train a 7B model for 2.5T tokens on this dataset with the same hyperparameters as our largest competition scale except for two separate cool-down phases for the 200B and 270B tokens on a modified distribution that was 70% DCLM-BASELINE with a tighter fastText threshold, and 30% math datasets.

We then take a "model soup" of these two separate cool-downs. Finally, we adopt the continual pretraining methodology from Pouransari et al. for 100B tokens on the same distribution to increase the context length from 2048 to 8192. We show that our model outperforms all 7B models trained on public training sets and approaches closed-data models trained for more tokens such as Llama-8B, Mistral-7B, and Gemma-7B. Additionally, our model achieves strong instruction-tuning (IT) performance: after instruction tuning on publicly available IT datasets, our model maintains most of its benchmark performance and achieves an AlpacaEval2.0 LC Win-rate of 16.6, which outperforms Gemma-Instruct (10.4), while approaching the strong performance of Mistral-v0.2-7B (17.1) and Llama3-Instruct (22.9). Finally, we show results from training a 1B model on 4.3T tokens from DCLM-BASELINE, StarCoder and ProofPile2 combined, resulting in a strong, small model that outperforms prior small models including Gemma-2B and Qwen2-1.5B.

We introduced the DCLM testbed and demonstrated how it leads to new state-of-the-art training sets. Our exploration of the dataset design space is only the beginning and has clear limitations. Due to compute constraints, we could only ablate design dimensions individually and could not test all approaches at larger scales nor train models beyond 7B parameters. We also could not sufficiently explore run-to-run variation. Moreover, there are many variations of DCLM-BASELINE that we did not explore, such as alternatives to sharded deduplication and using differently trained filtering models. We also conducted most of our experiments with only one tokenizer (GPT-NeoX), and other tokenizers may perform better on multilingual tasks or math.

While models trained on DCLM-BASELINE are competitive on common language understanding tasks, they currently do not perform as well on code and math. We view this as a consequence of our focus on language understanding in the first version of DCLM, and not an inherent limitation of our benchmark or dataset. Prior work has shown that adding specific training data and post-training methods for code and math can substantially improve models on those domains; combining DCLM-BASELINE with these domain-specific training sets and extending DCLM to cover code and math are interesting future directions. Other important dimensions to expand DCLM along are fairness, multilinguality, and safety. We hope that our open-source testbed can strengthen data-centric research in these directions as well.