DeepSeekMath: Pushing the Limits of Mathematical

Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.

Large language models (LLM) have revolutionized the approach to mathematical reasoning in artificial intelligence, spurring significant advancements in both the quantitative reasoning benchmark and the geometry reasoning benchmark. Moreover, these models have proven instrumental in assisting humans in solving complex mathematical problems. However, cutting-edge models such as GPT-4 and Gemini-Ultra are not publicly available, and the currently accessible open-source models considerably trail behind in performance. In this study, we introduce DeepSeekMath, a domain-specific language model that significantly outperforms the mathematical capabilities of open-source models and approaches the performance level of GPT-4 on academic benchmarks. To achieve this, we create the DeepSeekMath Corpus, a large-scale high-quality pre-training corpus comprising 120B math tokens. This dataset is extracted from the Common Crawl (CC) using a fastText-based classifier.

In the initial iteration, the classifier is trained using instances from OpenWebMath as positive examples and a diverse selection of other web pages as negatives. We then employ the classifier to mine additional positive instances from the CC, which are further refined through human annotation, and the classifier is updated with this enhanced dataset. The evaluation results indicate that the large-scale corpus is of high quality: our base model DeepSeekMath-Base 7B achieves 64.2% on GSM8K and 36.2% on the competition-level MATH dataset, outperforming Minerva 540B. The DeepSeekMath Corpus is multilingual, so we also notice improvements on Chinese mathematical benchmarks. DeepSeekMath-Base is initialized with DeepSeek-Coder-Base-v1.5 7B, as we notice that starting from a code training model is a better choice compared to a general LLM. Furthermore, math training also improves model capability on MMLU and BBH benchmarks, indicating it enhances not only mathematical abilities but also general reasoning.

After pre-training, we apply mathematical instruction tuning to DeepSeekMath-Base with chain-of-thought, program-of-thought, and tool-integrated reasoning data. The resulting model DeepSeekMath-Instruct 7B beats all 7B counterparts and is comparable with 70B open-source instruction-tuned models. Furthermore, we introduce the Group Relative Policy Optimization (GRPO), a variant reinforcement learning algorithm of Proximal Policy Optimization (PPO). GRPO foregoes the critic model, instead estimating the baseline from group scores, significantly reducing training resources. By solely using a subset of English instruction-tuning data, GRPO obtains a substantial improvement over the strong DeepSeekMath-Instruct: in-domain GSM8K 82.9% → 88.2%, MATH 46.8% → 51.7%, and out-of-domain CMATH 84.6% → 88.8%. We also provide a unified paradigm that subsumes RFT, DPO, PPO, and GRPO as direct or simplified RL techniques, and conduct extensive experiments to investigate the essential elements of this paradigm.

Math Pre-Training at Scale. Our research shows that publicly accessible Common Crawl data contains valuable mathematical information. With a carefully designed pipeline, we construct DeepSeekMath Corpus, a 120B-token math corpus filtered from the web — almost 7× the size of Minerva's math web pages and 9× OpenWebMath. Our pre-trained base DeepSeekMath-Base 7B reaches Minerva 540B-level performance, demonstrating that parameter count is not the only key factor for mathematical reasoning. Code training prior to math training improves models' ability to solve mathematical problems both with and without tool use — offering a partial answer to the long-standing question of whether code training improves reasoning. In contrast, although training on arXiv is common in prior math LLMs, it brings no notable improvements on any of our benchmarks.

Exploration and Analysis of Reinforcement Learning. We introduce GRPO, an efficient and effective RL algorithm. GRPO foregoes the critic model and estimates the baseline from group scores, significantly reducing training resources versus PPO. GRPO significantly enhances DeepSeekMath-Instruct using only the instruction-tuning data, and we also observe out-of-domain gains during RL. We further provide a unified paradigm to understand RFT, DPO, PPO and GRPO, and conduct extensive experiments — online vs offline training, outcome vs process supervision, single-turn vs iterative RL — to deeply investigate the essential elements. Finally, based on this paradigm, we explore why RL is effective and summarize potential directions for more effective RL of LLMs.

English & Chinese Mathematical Reasoning. We benchmark on English GSM8K / MATH / SAT / OCW Courses / MMLU-STEM and Chinese MGSM-zh / CMATH / Gaokao-MathCloze / Gaokao-MathQA, evaluating both tool-free text solutions and Python-aided solutions. On English benchmarks, DeepSeekMath-Base is competitive with closed-source Minerva 540B and surpasses every open-source base model (e.g., Mistral 7B, Llemma-34B) by a large margin. On Chinese benchmarks, our model is superior — likely because we do not restrict pre-training data to English. With instruction tuning and RL, DeepSeekMath-Instruct and DeepSeekMath-RL obtain >50% accuracy on competition-level MATH for the first time in the open-source community.

Formal Mathematics & General Capability. We evaluate informal-to-formal theorem proving on miniF2F with Isabelle, where DeepSeekMath-Base shows strong few-shot autoformalization. To build a comprehensive profile, we also report MMLU (57-subject multiple choice), BBH (challenging multi-step reasoning), HumanEval and MBPP (code). Math pre-training benefits both language understanding and general reasoning — DeepSeekMath-Base outperforms its DeepSeek-Coder-Base-v1.5 precursor on MMLU/BBH while maintaining HumanEval/MBPP coding performance.

We outline the iterative pipeline that constructs the DeepSeekMath Corpus from Common Crawl, starting from a small high-quality seed corpus (OpenWebMath). Using the seed corpus we train a fastText model to recall OpenWebMath-like math web pages: we randomly select 500K positives from the seed and 500K negatives from CC, with vector dim 256, lr 0.1, max n-gram 3, min word occurrence 3, 3 training epochs. We apply URL-based and near-duplicate deduplication to the original CC, yielding 40B HTML pages, then recall math pages with the fastText model and only keep top-ranking ones. The volume of data preserved is assessed through pre-training experiments on the top 40B / 80B / 120B / 160B tokens — in the first iteration we keep the top 40B.

After the first iteration, many math web pages remain uncollected because the seed lacks diversity. We therefore identify additional math web sources: we partition CC into domains (pages sharing the same base URL); domains where over 10% of pages were collected in the first iteration are classified as math-related (e.g. mathoverflow.net); we then manually annotate the URL paths associated with math content within those domains (e.g. mathoverflow.net/questions) and add uncollected pages under those paths back into the seed. This yields more positives and a better fastText model for the next iteration. After 4 iterations we end up with 35.5M math web pages, 120B tokens — in iteration 4 nearly 98% of data was already collected by iteration 3, so we stop.

To avoid benchmark contamination, we filter out web pages containing questions or answers from English benchmarks (GSM8K, MATH) and Chinese benchmarks (CMATH, AGIEval). The criteria: any text segment containing a 10-gram string that exactly matches any substring of an evaluation benchmark is removed. For benchmark texts shorter than 10-grams but ≥ 3-grams, we use exact matching.

We pre-train DeepSeek-LLM 1.3B for 150B tokens separately on each of four math corpora — MathPile (8.9B, >85% arXiv), OpenWebMath (13.6B), Proof-Pile-2 (51.9B; OpenWebMath + AlgebraicStack + arXiv at 2:4:1) and our DeepSeekMath Corpus (120B) — using AdamW ($\beta_1=0.9, \beta_2=0.95$, wd $0.1$), peak lr $5.3\text{e-}4$ with multi-step decay (to $31.6\%$ after 80% and $10\%$ after 90%), batch 4M tokens, 4K context length.

The DeepSeekMath Corpus wins on three dimensions: (1) High quality — on 8 math benchmarks with few-shot CoT, the model trained on DeepSeekMath leads clearly (e.g. GSM8K 23.8% vs Proof-Pile-2 14.3%, OpenWebMath 11.5%, MathPile 2.7%; MATH 13.6% vs 11.2/8.9/3.3); Figure 3 shows DeepSeekMath-trained model already beats Proof-Pile-2 at 50B tokens (= 1 full epoch of Proof-Pile-2), indicating higher per-token quality. (2) Multilingual — the corpus contains both English and Chinese; existing English-centric corpora may even hurt Chinese math performance (CMATH: 41.5% on DeepSeekMath vs 1.2% on MathPile, vs 12.3% no-math-training baseline). (3) Large-scale — DeepSeekMath Corpus is several times larger than alternatives, so the 1.3B model shows a steeper, longer-lasting learning curve; baseline corpora are repeated for many epochs and quickly plateau.

DeepSeekMath-Base 7B is initialized from DeepSeek-Coder-Base-v1.5 7B and trained for 500B tokens with the following mix: 56% DeepSeekMath Corpus, 4% AlgebraicStack, 10% arXiv, 20% GitHub code, 10% natural-language CC (English & Chinese). We use the training setting from §2.2.1 except peak lr $4.2\text{e-}4$ and batch size 10M tokens.

Step-by-step CoT reasoning. Across 8 English/Chinese benchmarks with few-shot CoT, DeepSeekMath-Base 7B leads every open-source base model — including general Mistral 7B and Llemma 34B (which is itself math-pretrained on Proof-Pile-2). On competition-level MATH, DeepSeekMath-Base surpasses other open-source base models by over 10% absolute, and even beats closed-source Minerva 540B — a model 77× larger built on PaLM. Concretely: GSM8K 64.2% (vs Minerva 540B 58.8%), MATH 36.2% (vs 33.6%), CMATH 71.7%, Gaokao-MathQA 35.3%.

Tool-aided math & formal proof. With few-shot program-of-thought (Python) prompting, DeepSeekMath-Base 7B beats prior SOTA Llemma 34B on both GSM8K+Python (66.9% vs 64.6%) and MATH+Python (31.4% vs 26.3%). On informal-to-formal proving on miniF2F-valid/test with Isabelle (using model-generated proof sketches plus Sledgehammer), it also leads (25.8% / 24.6% vs Llemma 34B's 21.0% / 21.3%).

General capability. Compared to its precursor DeepSeek-Coder-Base-v1.5, DeepSeekMath-Base 7B gains significantly on MMLU (49.1% → 54.9%) and BBH (55.2% → 59.5%), while maintaining HumanEval / MBPP coding scores — i.e., math training also amplifies general reasoning without sacrificing code ability.

We construct a 776K-example math instruction-tuning dataset covering English and Chinese problems across diverse fields and difficulty levels, with solutions in three formats: chain-of-thought (CoT), program-of-thought (PoT), and tool-integrated reasoning. The English split annotates GSM8K and MATH with tool-integrated solutions plus a subset of MathInstruct and the Lila-OOD training set covering algebra, probability, number theory, calculus and geometry. The Chinese split covers K-12 problems spanning 76 sub-topics (e.g., linear equations) with CoT and tool-integrated solutions.

DeepSeekMath-Instruct 7B is trained on the above SFT data starting from DeepSeekMath-Base; examples are randomly concatenated up to 4K context length, trained for 500 steps with batch 256 and constant lr $5\text{e-}5$. Under no-tool evaluation, DeepSeekMath-Instruct 7B surpasses all open-source models and most proprietary ones (Inflection-2, Gemini Pro) by ≥9% absolute on MATH — even beating substantially larger Qwen 72B and math-RL-enhanced WizardMath-v1.1 7B. It rivals Chinese proprietary GLM-4 and Baichuan-3 on MATH but still trails GPT-4 / Gemini Ultra. With tool integration, DeepSeekMath-Instruct 7B approaches 60% on MATH — beating all open-source models — and rivals DeepSeek-LLM-Chat 67B (10× larger) on other benchmarks.

Reinforcement learning has been proven effective in further improving the mathematical reasoning ability of LLMs after SFT. Proximal Policy Optimization (PPO) is the de-facto actor-critic RL algorithm for LLM RL fine-tuning. It optimizes the following clipped surrogate objective:

§4.1.2 Outcome Supervision RL with GRPO. For each question $q$, sample $\{o_1, \dots, o_G\}$ from $\pi_{\theta_{\text{old}}}$; the reward model produces $G$ scalar rewards $\mathbf{r} = \{r_1, \dots, r_G\}$. Normalize by group mean and std, then assign the same normalized reward to every token of $o_i$:

Then optimize Eq. (3).

§4.1.3 Process Supervision RL with GRPO. Outcome supervision only rewards the end of $o_i$, which may not be sufficient for complex math. We additionally explore process supervision: a process reward model (PRM) scores the end of each reasoning step, yielding step rewards $\mathcal{R} = \{\{r_i^{\mathrm{index}(1)},\dots,r_i^{\mathrm{index}(K_i)}\}\}_{i=1}^G$ where $\mathrm{index}(j)$ is the end-token index of the $j$-th step and $K_i$ is the number of steps in $o_i$. We normalize over the whole $\mathcal{R}$, then set each token's advantage as the sum of normalized rewards from following steps:

§4.1.4 Iterative RL with GRPO. As RL progresses, the old reward model may not adequately supervise the current policy. We therefore explore iterative GRPO: generate new training sets for the reward model from the current policy's samples, continually train the old reward model with 10% replay of historical data, then set the reference model to the current policy and continue policy training with the new reward model. See Algorithm 1 in the paper.

We RL-train from DeepSeekMath-Instruct 7B. The RL training data are CoT-format questions related to GSM8K and MATH from the SFT data — ~144K questions; we exclude other SFT questions to investigate the impact of RL on out-of-distribution benchmarks. The reward model training set follows Math-Shepherd (Wang et al., 2023b); the initial reward model is trained on DeepSeekMath-Base 7B with lr $2\text{e-}5$. For GRPO we use policy lr $1\text{e-}6$, KL coefficient $\beta = 0.04$, $G = 64$ outputs per question, max length 1024, batch 1024, and a single policy update per exploration stage.

Results: DeepSeekMath-RL 7B reaches 88.2% on GSM8K and 51.7% on MATH with CoT — beating every open-source 7B-70B model and most closed-source models. Crucially, although DeepSeekMath-RL is only RL-trained on GSM8K/MATH CoT questions starting from DeepSeekMath-Instruct, it improves over DeepSeekMath-Instruct on all benchmarks (including OOD: CMATH 84.6 → 88.8, MGSM-zh 73.2 → 79.6, tool-aided MATH 57.4 → 58.8). This out-of-domain transfer is one of GRPO's most striking empirical findings.

§5.1.1 Code training benefits mathematical reasoning. Using DeepSeek-LLM 1.3B, we compare two-stage settings (general/code 400B → math 150B) and one-stage settings (math-only 150B, mixed code+math). Findings: (i) code training, even alone, improves Python-aided GSM8K/MATH; with math added in stage 2, gains compound. (ii) Code training also improves tool-free math reasoning. (iii) One-stage mixed code+math mitigates the catastrophic forgetting that the two-stage setup causes on code benchmarks, but slightly hurts tool-free math — likely because 1.3B lacks capacity to absorb both simultaneously.

§5.1.2 ArXiv papers seem ineffective for math reasoning. Counter-intuitively, training on arXiv-heavy corpora (MathPile, ArXiv-RedPajama) brings no notable improvements — and sometimes degradation — on GSM8K, MATH, MMLU-STEM and miniF2F. We tested both DeepSeek-LLM 1.3B (trained 150B tokens) and DeepSeek-Coder-Base-v1.5 7B (40B tokens). Caveats: we have not studied arXiv on tasks like theorem informalization, in combination with other data, or at much larger model scale.

We provide a unified paradigm to analyze SFT, RFT, DPO, PPO, GRPO. Generally, the gradient w.r.t. $\theta$ can be written as

§5.2.2 Why does RL work? A striking finding: RL improves Maj@K but not Pass@K (Figure 7, $K \in [1, 64]$). This indicates that RL is not adding new capabilities — instead, RL is making the output distribution more robust by boosting the correct response that already exists within the model's Top-K. Wang et al. (2023a) similarly identified a "misalignment" in SFT models that preference-alignment fixes. In short: RL = distribution sharpening over already-present correct solutions.

§5.2.3 How to achieve more effective RL? Within the unified paradigm (Eq. 5), there are three knobs to tune: Data Source (we only used SFT-stage questions and naive nucleus sampling — explore OOD prompts, tree-search decoding, efficient inference like PagedAttention / speculative decoding to widen exploration); Algorithm (all current methods fully trust the reward — explore noisy-reward-robust RL, weak-to-strong alignment); Reward Function (improve generalization to OOD, reflect uncertainty as a bridge to weak-to-strong, build efficient high-quality process reward models).

We present DeepSeekMath — which outperforms all open-source models on competition-level MATH and approaches closed models. It is initialized with DeepSeek-Coder-v1.5 7B and continually trained for 500B tokens, of which 120B math tokens come from Common Crawl. Our ablations show web pages offer significant potential for high-quality mathematical data, while arXiv may be less beneficial than expected. We introduce GRPO, a PPO variant that notably improves mathematical reasoning with less memory; GRPO is effective even when starting from a strong DeepSeekMath-Instruct 7B. We also provide a unified paradigm to understand a family of methods and summarize potential directions for more effective RL.

Limitations. Although DeepSeekMath scores well on quantitative reasoning benchmarks, its capability on geometry and theorem-proof is weaker than closed models — in dry runs it struggles on triangle/ellipse problems, suggesting data selection bias in pre-training and fine-tuning. Limited by 7B scale, DeepSeekMath is worse than GPT-4 on few-shot prompts — GPT-4 improves with few-shot, whereas DeepSeekMath performs similarly in zero-shot and few-shot. Future work: better-engineered data pipeline for higher-quality pre-training corpus, and exploring the directions in §5.2.3 for more effective RL.