DAPO: An Open-Source LLM Reinforcement Learning

Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.

Test-time scaling such as OpenAI's o1 and DeepSeek's R1 brings a profound paradigm shift to Large Language Models (LLMs). Test-time scaling enables longer Chain-of-Thought thinking and induces sophisticated reasoning behaviors, which makes the models superior in competitive math and coding tasks like AIME and Codeforces. The central technique driving the revolution is large-scale Reinforcement Learning (RL), which elicits complex reasoning behaviors such as self-verification and iterative refinement. However, the actual algorithm and key recipe for scalable RL training remains a myth, hidden from technical reports of existing reasoning models. In this paper, we reveal significant obstacles in large-scale RL training and open-source a scalable RL system with fully open-sourced algorithm, training code and dataset that provides democratized solutions with industry-level RL results.

We experiment over Qwen2.5-32B as the pretrained model for RL. In our initial GRPO run, we achieved only 30 points on AIME — a performance significantly below DeepSeek's RL (47 points). A thorough analysis reveals that the naive GRPO baseline suffers from several key issues such as entropy collapse, reward noise, and training instability. The broader community has encountered similar challenges in reproducing DeepSeek's results, suggesting that critical training details may have been omitted in the R1 paper that are required to develop an industry-level, large-scale, and reproducible RL system.

To close this gap, we release an open-source state-of-the-art system for large-scale LLM RL, which achieves 50 points on AIME 2024 based on Qwen2.5-32B model, outperforming previous state-of-the-art results achieved by DeepSeek-R1-Zero-Qwen-32B (47 points) using 50% training steps (Figure 1). We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and introduce 4 key techniques to make RL shine in the long-CoT RL scenario:

1. Clip-Higher, which promotes the diversity of the system and avoids entropy collapse;
2. Dynamic Sampling, which improves training efficiency and stability;
3. Token-Level Policy Gradient Loss, which is critical in long-CoT RL scenarios;
4. Overlong Reward Shaping, which reduces reward noise and stabilizes training.

Our implementation is based on verl. By fully releasing our state-of-the-art RL system including training code and data, we aim to reveal valuable insights to large-scale LLM RL that benefit the larger community.

2.1 Proximal Policy Optimization (PPO). PPO introduces a clipped surrogate objective for policy optimization. By constraining the policy updates within a proximal region of the previous policy using clip, PPO stabilizes training and improves sample efficiency. Specifically, PPO updates the policy by maximizing the following objective:

We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm. DAPO samples a group of outputs $\{o_i\}_{i=1}^G$ for each question $q$ paired with the answer $a$, and optimizes the policy via the following objective:

where $r_{i,t}(\theta)=\frac{\pi_\theta(o_{i,t}\mid q,o_{i,

3.1 Raise the Ceiling: Clip-Higher. In our initial experiments using naive PPO or GRPO, we observed the entropy collapse phenomenon: the entropy of the policy decreases quickly as training progresses (Figure 2b). The sampled responses of certain groups tend to be nearly identical. This indicates limited exploration and early deterministic policy, which can hinder the scaling process. We propose the Clip-Higher strategy to address this issue.

Concretely, when $\varepsilon=0.2$ (the default value of most algorithms) and $\hat A_{i,t}>0$ (the system tries to increase the probability), consider two actions with probabilities $\pi_{\theta_\text{old}}(o_i\mid q)=0.01$ and $0.9$. The upper bounds of the increased probabilities $\pi_\theta(o_i\mid q)$ are $0.012$ and $1.08$, respectively. This implies that 'exploitation' tokens with a higher probability (e.g., 0.9) are not constrained to get even extremely larger probabilities like 0.999. Conversely, for low-probability 'exploration' tokens, achieving a non-trivial increase in probability is considerably more challenging. Empirically, we also observe that the mean probability of up-clipped tokens is low: $\pi_\theta(o_i\mid q)<0.2$ (Figure 3a).

Adhering to the Clip-Higher strategy, we decouple the lower and higher clipping range as $\varepsilon_\text{low}$ and $\varepsilon_\text{high}$. We increase the value of $\varepsilon_\text{high}$ to leave more room for the increase of low-probability tokens. As shown in Figure 2, this adjustment effectively enhances the policy's entropy and facilitates the generation of more diverse samples. We keep $\varepsilon_\text{low}$ as it is, because increasing it will suppress the probability of these tokens to 0, resulting in the collapse of the sampling space.

3.2 The More the Merrier: Dynamic Sampling. Existing RL algorithms suffer from the gradient-decreasing problem when some prompts have accuracy equal to 1. For example for GRPO, if all outputs $\{o_i\}_{i=1}^G$ of a particular prompt are correct and receive the same reward, the resulting advantage for this group is zero. A zero advantage results in zero policy gradients, shrinking the magnitude and increasing the noise sensitivity of the batch gradient, thereby degrading sample efficiency. Empirically, the number of samples with accuracy equal to 1 continues to increase, as shown in Figure 3b.

To this end, we propose to over-sample and filter out prompts with the accuracy equal to 1 and 0, leaving all prompts in the batch with effective gradients and keeping a consistent number of prompts. Before training, we keep sampling until the batch is fully filled with samples whose accuracy is neither 0 nor 1. Note that this strategy does not necessarily impede training efficiency, because the generation time is typically dominated by the generation of long-tail samples if the RL system is synchronized and the generation stage is not pipelined. Besides, we find that with dynamic sampling the experiment achieves the same performance faster as shown in Figure 6.

3.3 Rebalancing Act: Token-Level Policy Gradient Loss. The original GRPO algorithm employs a sample-level loss calculation, which involves first averaging the losses by token within each sample and then aggregating the losses across samples. In this approach, each sample is assigned an equal weight in the final loss computation. However, we find that this method of loss reduction introduces several challenges in the context of long-CoT RL scenarios.

Since all samples are assigned the same weight in the loss calculation, tokens within longer responses (which contain more tokens) may have a disproportionately lower contribution to the overall loss, which can lead to two adverse effects. First, for high-quality long samples, this effect can impede the model's ability to learn reasoning-relevant patterns within them. Second, we observe that excessively long samples often exhibit low-quality patterns such as gibberish and repetitive words. Thus, sample-level loss calculation, due to its inability to effectively penalize those undesirable patterns in long samples, leads to an unhealthy increase in entropy and response length (Figure 4a, 4b).

We introduce a Token-Level Policy Gradient Loss in the long-CoT RL scenario. In this setting, longer sequences can have more influence on the overall gradient update compared to shorter sequences. Moreover, from the perspective of individual tokens, if a particular generation pattern can lead to an increase or decrease in reward, it will be equally prompted or suppressed, regardless of the length of the response in which it appears.

3.4 Hide and Seek: Overlong Reward Shaping. In RL training, we typically set a maximum length for generation, with overlong samples truncated accordingly. We find that improper reward shaping for truncated samples can introduce reward noise and significantly disrupt the training process. By default, we assign a punitive reward to truncated samples. This approach may introduce noise into the training process, as a sound reasoning process can be penalized solely due to its excessive length.

To investigate the impact of this reward noise, we first apply an Overlong Filtering strategy which masks the loss of truncated samples. We find that this approach significantly stabilizes training and enhances performance (Figure 5). Furthermore, we propose Soft Overlong Punishment, a length-aware penalty mechanism designed to shape the reward for truncated samples. Specifically, when the response length exceeds the predefined maximum value, we define a punishment interval. Within this interval, the longer the response, the greater the punishment it receives. This penalty is added to the original rule-based correctness reward:

3.5 Dataset Transformation. Our dataset is sourced from the web and official competition homepages through a combination of web scraping and manual annotation. The answers of math datasets typically come in a variety of formats, such as expression, formula and number, which makes it challenging to design comprehensive rules to parse them. To provide accurate reward signals using rules and minimize errors introduced by formula parsers, inspired by AIME, we select and transform the answers into integers, which are easy to parse. For example, if the original answer is expressed in the form of $\frac{a+\sqrt{b}}{c}$, we instruct the LLM to modify the question so that the expected answer becomes $a+b+c$. After selection and transformation, we obtained the DAPO-Math-17K dataset, which consists of 17K prompts, each paired with an integer as the answer.

4.1 Training Details. In this work, we focus specifically on mathematical tasks to evaluate our algorithm, which can be readily transferred to other tasks. We adopt the verl framework for training. We use naive GRPO as our baseline algorithm and estimate advantages using group reward normalization. For hyper-parameters, we utilize the AdamW optimizer with a constant learning rate of $1\times10^{-6}$, incorporating a linear warm-up over 20 rollout steps. For rollout, the prompt batch size is 512 and we sample 16 responses for each prompt. For training, the mini-batch size is set to 512, i.e., 16 gradient updates for each rollout step. For Overlong Reward Shaping, we set the expected maximum length as 16,384 tokens and allocate additional 4,096 tokens as the soft punish cache; therefore, the maximum number of tokens for generation is 20,480. As for the Clip-Higher mechanism, we set $\varepsilon_\text{low}=0.2$ and $\varepsilon_\text{high}=0.28$, which effectively balance the trade-off between exploration and exploitation. For evaluation on AIME, we repeat the evaluation set for 32 times and report avg@32 for results stability. The inference hyperparameters of evaluation are set to temperature 1.0 and top-$p$ 0.7.

4.2 Main Results. Experiments on AIME 2024 demonstrate that DAPO has successfully trained the Qwen-32B Base model into a powerful reasoning model, achieving performance superior to DeepSeek's experiments on Qwen2.5-32B using the R1 approach. In Figure 1, we observe a substantial improvement of performance on AIME 2024, with accuracy increasing from near 0% to 50%. Notably, this improvement is achieved with only 50% of the training steps required by DeepSeek-R1-Zero-Qwen-32B. We analyze the contributions of each training technique in our methodology, as detailed in Table 1: Naive GRPO 30 -> +Overlong Filtering 36 -> +Clip-Higher 38 -> +Soft Overlong Punishment 41 -> +Token-level Loss 42 -> +Dynamic Sampling (DAPO) 50. The observed improvements demonstrate the effectiveness of these techniques in RL training, each contributing several accuracy points in AIME 2024. Notably, given the vanilla GRPO setting, only 30% accuracy can be reached by training from a Qwen2.5-32B base model.

For token-level loss, although it brings less performance improvement, we find it enhances training stability and makes the length increase more healthily. When applying Dynamic Sampling, although more data needs to be sampled due to the filtering out of zero-gradient data, the overall training time is not significantly affected. As shown in Figure 6, although the number of sampling instances increases, the model's convergence time is even reduced, due to fewer training steps required.

4.3 Training Dynamics. Reinforcement learning on large language models is not only a cutting-edge research direction but also an intrinsically complex systems engineering challenge, characterized by the interdependence of its various subsystems. Modifications to any single subsystem can propagate through the system, leading to unforeseen consequences. Even seemingly minor changes in initial conditions, such as variations in data and hyperparameters, can amplify through iterative reinforcement learning processes, yielding substantial deviations in outcomes. Therefore, monitoring of key intermediate results during experimentation is essential for swiftly identifying the sources of discrepancies and, ultimately, for refining the system.

The Length of Generated Responses is a metric closely related to training stability and performance. The increase in length provides the model with a larger space for exploration, allowing more complex reasoning behaviors to be sampled and gradually reinforced. However, length does not always maintain a continuous upward trend during training; it can exhibit stagnation or even decline. We typically use length in conjunction with validation accuracy as indicators to assess whether an experiment is deteriorating.

The Dynamics of Reward during training is one of the crucial monitoring indicators. In most of our experiments, the trend of reward increase is relatively stable. However, we find that the final reward on the training set often exhibits little correlation with the accuracy on the validation set, which indicates overfitting to the training set. The Entropy of the Actor Model and Generation Probability are related to the model's exploration capability. An excessively low entropy indicates a sharp distribution and loss of exploration; an excessively high entropy is often associated with over-exploration such as gibberish and repetitive generation. By applying the Clip-Higher strategy, we effectively addressed entropy collapse; we find that maintaining a slow upward trend in entropy is conducive to performance improvement.

4.4 Case Study. During the RL training process, we observe an interesting phenomenon: the reasoning patterns of the actor model evolve dynamically over time. Specifically, the algorithm not only reinforces existing reasoning patterns that facilitate correct problem-solving but also gradually gives rise to entirely new modes of reasoning that were initially absent. This finding reveals the adaptability and exploration capability of RL algorithms and offers new insights into the learning mechanisms of the model. For example, in the early stages of model training, there was virtually no occurrence of checking and reflecting on previous reasoning steps. However, as training progresses, the model exhibits distinct behaviors of reflection and backtracking, as shown in Table 2. This observation sheds light on further exploration into interpreting the emergence of reasoning abilities during RL.

In this paper, we release a fully open-sourced system for large-scale LLM RL, including algorithm, code infrastructure, and dataset. The system achieves state-of-the-art large-scale LLM RL performance (AIME 50 using Qwen-32B pretrained model). We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and introduce 4 key techniques to make RL powerfully effective and efficient in the long-CoT RL scenario. Additionally, by open-sourcing the training code and dataset, we provide the broader research community and society with practical access to a scalable reinforcement learning solution, enabling all to benefit from these advancements.

A. Dataset Transformation. The appendix provides a worked example of how DAPO-Math-17K is built: take the original problem "Let $x,y$ be reals with $x^2+y^2-22x-16y+113=0$; determine the smallest possible value of $x$" with answer $11-2\sqrt{6}$, and ask the LLM to rewrite the prompt so the new expected answer is the integer $k+m+n=11+2+6=19$ (matching the format $k-m\sqrt{n}$). The LLM follows a four-step CoT: extract the answer format, rewrite the problem statement, solve the modified problem, output an integer. Few-shot solutions and detailed guidelines are provided for each step, and the authors report that the LLM produces satisfactory reformulations in most cases.

B. Supplementary Case. A second case illustrates the emergence of reflective behavior: an AIME-style inclusion-exclusion problem about 900 Aimeville residents owning combinations of 4 items. The model first tries a direct inclusion-exclusion derivation, gets a non-integer $a_4=54.75$, recognizes that the count must be a whole number, and explicitly says "our current approach needs to be reconsidered" — then switches to a different decomposition. This kind of self-checked back-tracking did not exist at the start of RL training and emerged through DAPO RL.