MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining

MiMo: Unlocking the Reasoning Potential of Language Model — From Pretraining to Posttraining. LLM-Core, Xiaomi (arXiv:2505.07608v2 [cs.CL], 5 Jun 2025).

We present MiMo-7B, a large language model born for reasoning tasks, with optimization across both pre-training and post-training stages. During pre-training, we enhance the data preprocessing pipeline and employ a three-stage data mixing strategy to strengthen the base model's reasoning potential. MiMo-7B-Base is pre-trained on 25 trillion tokens, with an additional Multi-Token Prediction objective for enhanced performance and accelerated inference speed. During post-training, we curate a dataset of 130K verifiable mathematics and programming problems for reinforcement learning, integrating a test-difficulty–driven code-reward scheme to alleviate sparse-reward issues and employing strategic data resampling to stabilize training. Extensive evaluations show that MiMo-7B-Base possesses exceptional reasoning potential, outperforming even much larger 32B models. The final RL-tuned model, MiMo-7B-RL, achieves superior performance on mathematics, code and general reasoning tasks, surpassing the performance of OpenAI o1-mini.

Large language models (LLMs) with advanced reasoning capabilities, such as OpenAI o-series, DeepSeek R1, and Claude 3.7, have achieved remarkable performance in complex tasks like mathematical reasoning and code generation. Through large-scale reinforcement learning (RL), these models develop sophisticated reasoning patterns, including step-by-step analysis, self-reflection and backtracking, enabling more robust and accurate problem-solving capabilities across diverse domains. Currently, most successful RL works rely on relatively large base models, e.g., 32B models, particularly for enhancing code reasoning capabilities. Moreover, it is widely considered that achieving uniform and simultaneous improvements in both mathematical and code capabilities within a small model is challenging.

Nonetheless, we believe that the effectiveness of the RL-trained reasoning model relies on the inherent reasoning potential of the base model. To fully unlock the reasoning potential of language models, efforts must focus not only on post-training but also on pre-training strategies tailored to reasoning. In this work, we present MiMo-7B, a series of models trained from scratch and born for reasoning tasks. Our RL experiments from MiMo-7B-Base show that our model possesses extraordinary reasoning potential, even outperforming much larger 32B models. Additionally, we perform RL training on a cold-started SFT model, resulting in MiMo-7B-RL, which demonstrates superior performance on both mathematics and code reasoning tasks, surpassing OpenAI o1-mini.

Our contributions span three axes. Pre-Training (base model born for reasoning): we optimize the data preprocessing pipeline, enhance text extraction toolkits and apply multi-dimensional data filtering to increase reasoning-pattern density; we employ multiple strategies to generate massive diverse synthetic reasoning data; we adopt a three-stage data mixture strategy and pre-train on $\sim 25$T tokens; we incorporate Multi-Token Prediction (MTP) as an additional training objective for higher performance and faster inference. Post-Training (pioneering reasoning recipe): we curate 130K mathematics and code problems verifiable by rule-based verifiers; we introduce a test-difficulty–driven code reward that turns sparse rewards into dense ones; we implement an easy-data re-sampling strategy to stabilize policy updates in late RL. RL Infrastructure: we develop the Seamless Rollout Engine (continuous rollout + asynchronous reward + early termination) achieving $2.29\times$ faster training and $1.96\times$ faster validation, and we add MTP support to vLLM with hardened robustness.

Summary of evaluation results. MiMo-7B-Base outperforms SoTA open-source models of comparable size and even matches a 32B baseline in pass@k reasoning curves; on BBH it reaches 75.2 and on SuperGPQA it handles graduate-level questions strongly. MiMo-7B-RL-Zero (RL directly from base) already surpasses the RL performance of a 32B base on both math and code. The final MiMo-7B-RL scores 55.4 on AIME 2025 (+4.7 over o1-mini), significantly outperforms o1-mini on LiveCodeBench v5 and v6, and still keeps competitive general performance.

The pre-training corpus integrates web pages, academic papers, books, programming code, and synthetic data. We believe that incorporating more data with high-quality reasoning patterns during pre-training can substantially enhance the reasoning potential of the resulting language model. To this end, we first optimize the natural text preprocessing pipeline to improve quality and, most importantly, reasoning-data density; second, we leverage advanced reasoning models to generate extensive synthetic reasoning data; finally, we implement a three-stage data mixture strategy to maximize reasoning across various tasks and domains.

Better reasoning data extraction. Web pages naturally contain content with high-density reasoning patterns, such as coding tutorials and mathematics blogs, yet commonly used extractors often fail to preserve embedded mathematics equations and code snippets. We develop a novel HTML-extraction tool specially optimized for mathematics content, code blocks, and forum websites. For papers and books, we enhance PDF parsing toolkits to better handle STEM and code content, preserving massive reasoning patterns for subsequent stages.

Fast global deduplication & multi-dimensional filtering. We adopt URL deduplication plus MinHash deduplication across all webpage dumps and complete this global deduplication within a single day. Since dedup is content-agnostic, we then re-balance the final data distribution by multi-dimensional quality scores. Heuristic rule-based filters tend to mis-filter high-quality math/code web pages, so we instead fine-tune small LLMs as data-quality taggers performing domain classification and multi-dimensional quality assessment.

Synthetic reasoning data. We employ multiple strategies to generate diverse synthetic reasoning responses: (1) select STEM content tagged with high reasoning depth and prompt models to develop in-depth analyses; (2) gather mathematics and code problems and prompt reasoning models to solve them; (3) incorporate general-domain queries, particularly creative writing. Notably, unlike non-reasoning data, our preliminary experiments reveal that synthetic reasoning data can be trained for an extremely high number of epochs without overfitting.

Three-stage data mixture. Stage 1: all sources except synthetic reasoning responses; downsample ads / news / job postings and upsample high-value professional content. Stage 2: significantly increase math and code data to $\sim 70\%$ of the mixture (without compromising general language abilities). Stage 3: add $\sim 10\%$ synthetic responses for math, code and creative writing, and extend context length from 8,192 to 32,768. Stages 1–2 use 8K context; only Stage 3 uses 32K. Through this process we build a $\sim$25T-token pre-training dataset.

MiMo-7B follows the general decoder-only Transformer architecture, with Grouped-Query Attention (GQA), pre-RMSNorm, SwiGLU activation, and Rotary Positional Embedding (RoPE), similar to Llama and Qwen. Reasoning models often face an inference-speed bottleneck due to lengthy auto-regressive generation, despite the high correlation among consecutive tokens along reasoning paths.

MTP modules. Inspired by DeepSeek-V3, we incorporate Multi-Token Prediction (MTP) as an additional training objective. This lets the model strategically pre-plan and produce representations that facilitate more accurate and potentially faster prediction of future tokens. We implement distinct MTP setups for pre-training and inference: during pre-training we use a single MTP layer (preliminary studies show multiple MTP layers yield no further improvement), while at inference we replicate the pre-trained single MTP layer into multiple identical copies and, with the main model and first MTP layer frozen, fine-tune the new MTP layers for speculative decoding.

MTP inference speedup. On AIME24, the first MTP layer achieves a remarkably high acceptance rate of about $90\%$, while even the third MTP layer maintains an acceptance rate above $75\%$. This enables MiMo-7B to deliver enhanced decoding speed, particularly in reasoning scenarios that require extremely long outputs.

Model & training hyper-parameters. Transformer layers $=36$, hidden dim $=4{,}096$, FFN intermediate dim $=11{,}008$, attention heads $=32$ with $8$ KV groups. Optimizer is AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, weight decay $0.1$, grad-clip $1.0$. Stages 1–2 use seq-len $8{,}192$ with RoPE base $10{,}000$; Stage 3 extends to $32{,}768$ with RoPE base $640{,}000$.

Learning-rate schedule. Stage 1 linearly warms from $0$ to $1.07\times 10^{-4}$ over the first 84B tokens, holds constant for 10.2T tokens, then cosine-decays to $3\times 10^{-5}$ over 7.5T tokens. This rate is held through Stage 2 (4T tokens) and the first 1.5T tokens of Stage 3, then cosine-decays to $1\times 10^{-5}$ over the final 500B tokens. Batch size warms linearly to $2{,}560$ over the first 168B tokens (held through Stage 1–2); in Stage 3 it is fixed at $640$. MTP loss weight is $0.3$ for the first 10.3T tokens, then $0.1$ for the remainder.

We evaluate MiMo-7B-Base on a wide set of benchmarks spanning language understanding (BBH, MMLU/-Redux/-Pro, ARC, HellaSwag, PIQA), QA (TriviaQA, NaturalQuestions, GPQA, SuperGPQA), reading comprehension (DROP, RACE), mathematics (AIME, GSM8K, MATH), coding (LiveCodeBench, HumanEval(+), MBPP(+), CRUXEval), Chinese (C-Eval, CMMLU) and long-context (RULER).

Upper bounds of reasoning capability. Traditional evaluation methods often underestimate a model's true reasoning potential by relying on single-pass success rates or average performance across multiple samplings. Following Yue et al. (2025), we adopt the pass@k metric — a problem counts as solved if any of $k$ sampled solutions is correct — to better assess the reasoning capacity boundary of different models. As illustrated in Figure 3, MiMo-7B-Base achieves significantly higher pass@k scores than all compared models, including the 32B baseline, across all benchmarks and values of $k$. The gap widens steadily as $k$ increases, particularly on LiveCodeBench, establishing MiMo-7B-Base as a strong base policy for RL training.

General reasoning. On BBH, MiMo-7B-Base scores 75.2, surpassing Qwen2.5-7B by about 5 points. SuperGPQA results further highlight robust performance on graduate-level problems. On DROP it outperforms compared models, showing advanced language understanding.

Mathematics & code. On LiveCodeBench v5 it scores 32.9, far surpassing Llama-3.1-8B and Qwen2.5-7B; on AIME 2024 it scores 32.9, significantly outperforming other comparably sized base models. These results highlight the model's extraordinary problem-solving abilities and its huge potential for complex reasoning tasks.

Long-context comprehension. On RULER, MiMo-7B-Base achieves near-perfect needle-in-a-haystack retrieval across all positions within the 32K window. Beyond pure retrieval, it excels in tasks requiring long-context reasoning — Common Words Extraction (CWE), Frequent Words Extraction (FWE) and Variable Tracking (VT) — and surpasses Qwen2.5-7B in most scenarios. These results validate the efficacy of incorporating diverse high-reasoning-pattern data during pre-training.

After pre-training, post-training is implemented on MiMo-7B-Base. Specifically, we develop MiMo-7B-RL-Zero through direct RL from MiMo-7B-Base, and MiMo-7B-RL trained from an SFT version of MiMo-7B. The SFT data combines open-source and proprietary distilled data. To ensure optimal quality and diversity, we implement a three-stage preprocessing pipeline.

First, we eliminate all training queries that have 16-gram overlap with evaluation benchmarks to prevent data leakage. Then we exclude samples with mixed language or incomplete responses. Finally, we cap the number of responses per query at eight, striking a balance between preserving diversity and preventing redundancy. After preprocessing, our final SFT dataset comprises about 500K samples.

SFT hyper-parameters: we fine-tune MiMo-7B-Base with a constant learning rate of $3\times 10^{-5}$ and batch size of $128$. Samples are packed to the maximum length of $32{,}768$ tokens during training.

Later in the report (Discussion §3.6), we further scale SFT data from $\sim$500K to $\sim$6M instances. This substantial expansion materially improves reasoning ability and generalized dialogue capability while preserving (rather than compromising) the model's potential for subsequent RL. Table 6 shows the 6M-SFT model significantly beats the 500K-SFT model on AIME 24/25, MATH500, GPQA Diamond, LiveCodeBench v5 and Alignbench v1.1.

This SFT design serves two purposes: (1) align the base model with the expected answer-extraction format (e.g., \boxed{} for math); (2) provide a strong starting point so that subsequent RL can climb a higher ceiling (MiMo-7B-RL ultimately exceeds MiMo-7B-RL-Zero, which uses only RL on the base).

We use two categories of verifiable problems — mathematics and code — to formulate our RL training data. Our preliminary studies demonstrate that high-quality problem sets play a critical role in stabilizing RL training and further enhancing the LLM's reasoning capabilities.

Mathematical data. The math problem set is drawn from open-source datasets and proprietary competition-level collections. To mitigate reward hacking, we use an LLM to filter proof-based and multiple-choice problems. Unlike recent approaches that modify problems to ensure integer answers, we preserve the originals to minimize hacking surface. We also perform global n-gram deduplication and carefully decontaminate against evaluation benchmarks.

Model-based difficulty assessment. We first filter out problems that cannot be solved by advanced reasoning models — identifying problems that are either too difficult or contain incorrect answers. For the remaining problems, we roll out an SFT version of MiMo-7B $16$ times and eliminate problems with pass-rate $> 90\%$. Notably, this removes about $50\%$ of easy problems from the original set. After cleaning, we obtain a math training set of $100$K problems.

Code data. For coding problems we curate a high-quality training set from open-source datasets and our newly collected problem set. We remove problems without test cases; for problems with golden solutions we discard those whose golden solution fails any test; for problems without golden solutions we discard those where no test case can be solved in $16$ rollouts of advanced reasoning models. As with math, we use an SFT version of MiMo-7B to filter out easy problems that pass perfectly in all $16$ rollouts. This rigorous cleaning yields $30$K code problems.

Online judge & reward function. During each RL iteration we evaluate thousands of problems to compute rewards, with each problem potentially containing hundreds of test cases. To improve reward computation efficiency and eliminate GPU idle time, we developed an online-judge environment that enables parallel execution of an extremely high volume of unit tests. We employ only rule-based accuracy rewards: for math, the rule-based Math-Verify library; for code, the test-difficulty–driven reward described next. No additional rewards (such as format reward or length penalty) are incorporated.

We employ a modified version of Group Relative Policy Optimization (GRPO) with recent community improvements. For each problem $q$, the algorithm samples a group of responses $\{o_1, \dots, o_G\}$ from the old policy $\pi_{\theta_{\text{old}}}$ and updates $\pi_\theta$ by maximizing:

where the advantage is standardized within group: $A_{i,j} = \dfrac{r_i - \mathrm{mean}(\{r_i\}_{i=1}^G)}{\mathrm{std}(\{r_i\}_{i=1}^G)}$. On top of the original GRPO we add three community improvements: (a) Removal of KL loss — simply removing KL effectively unleashes the full potential of the policy without compromising stability; (b) Dynamic sampling — over-sample and filter out prompts with pass-rate exactly $0$ or $1$, leaving every batch with effective gradients while keeping batch size consistent; (c) Clip-Higher — increase the upper clip bound $\varepsilon_{\text{high}}$ in Eq. (1) (with fixed lower $\varepsilon_{\text{low}}$), which mitigates entropy collapse and helps the policy explore new solutions.

Test-difficulty–driven reward (IOI-style). Standard rule-based code reward gives a reward only if the solution passes all test cases. For hard algorithmic problems the model may never receive any reward, blocking learning from these examples and reducing dynamic-sampling efficiency. Inspired by the IOI scoring rule (each problem splits into subtasks of varying difficulty, partial credit allowed), we propose a new reward. We run several models with multiple rollouts on each problem and cluster test cases by their pass rate into easy / medium / hard groups. Then two reward schemes are defined: (i) Strict — a solution earns the reward of a difficulty level only if it passes all tests in that group and in all lower-difficulty groups; (ii) Soft — distribute the total score of each group equally among its tests, and sum scores of all passed tests. Both schemes outperform a vanilla rule-based reward on LiveCodeBench v5 / AtCoder.

Easy data filter and re-sampling. As the policy improves, more and more problems reach perfect pass-rate $=1$ and get filtered out by dynamic sampling, which drastically degrades sampling efficiency. Removing perfect-pass-rate problems entirely caused significant instability in our preliminary studies. So instead we maintain an "easy data pool" of perfect-pass-rate problems and during rollout sample from this pool with probability $\alpha = 10\%$. This stabilizes policy updates and improves sampling efficiency, especially in the later phases of RL.

RL hyper-parameters. Training batch size $512$ with an actor mini-batch size $32$; $16$ gradient updates per iteration at learning rate $1\mathrm{e}{-}6$; max sequence length $32{,}768$; both temperature and top-$p$ set to $1.0$ during training to promote output diversity.

We build our RL system on verl, an open-source RL library that uses Ray to manage computation and communication, implementing rollout and training phases as Ray Actors and exchanging training data via Ray Objects. Although verl supports flexible algorithm implementations, it suffers from GPU idle time in both rollout and reward-computation phases: due to skewed response lengths, most GPUs wait for a few long-sequence workers, wasting compute. Prior works rely on asynchronous training, which modifies the algorithm and introduces staleness in long responses. Rule-based reward computation is also time-consuming, especially for code data. Our dynamic sampling, while improving sample efficiency, exacerbates GPU idle time. To simultaneously optimize GPU utilization and reduce sample waste, we develop the Seamless Rollout Engine, opportunistically filling sample batches into rollout while performing asynchronous reward computation.

Continuous rollout. Unlike naive implementations that delay reward computation until all rollout workers complete, Seamless Rollout Engine eliminates synchronization barriers between generation and reward phases. It actively monitors completed workers, immediately computes their rewards, and triggers new rollouts on demand. After computing rewards we update the number of valid samples and the current step's pass-rate statistics, then launch new rollout tasks only if active tasks are insufficient to meet training demand based on these statistics.

Asynchronous reward computation. While math reward is rapid, judging code data incurs significant overhead, leading to prolonged GPU idle time; the sequential nature of naive reward computation fails to utilize multiprocessing on modern processors. We use Ray to launch asynchronous reward computation, concurrently managing rollout and reward tasks; dedicated servers handle code-specific reward to prevent pipeline bottlenecks.

Early termination. When valid samples exceed the required training batch size, we must manage ongoing tasks carefully. Abrupt termination suppresses long-sequence generation and destabilizes RL dynamics; waiting for all tasks may extend wait times when a long-sequence rollout starts near the end. To preserve data distribution, we use a first-in-first-out selection: terminate ongoing tasks only if valid count meets the batch requirement and all tasks initiated prior to the selected samples have completed.

Results. On a 256× H20-GPU 5-step trace, all three components contribute to faster dynamic sampling and smaller GPU idle time. Overall, Seamless Rollout achieves $2.29\times$ end-to-end speedup, $2.61\times$ rollout-stage speedup, reduces GPU idle ratio from $69.3\%$ to $27.7\%$, and reduces sample waste ratio from $22.1\%$ to $12.9\%$. For validation, asynchronous reward alone gives a $1.96\times$ speedup with idle time down to $25\%$. We additionally support MTP in vLLM (external-launch mode) and harden robustness — clearing computed blocks in prefix caching during pre-emption to maintain KV-cache consistency, and disabling async output processing when increasing scheduler steps.

We comprehensively evaluate reasoning models on a diverse set of benchmarks: MMLU-Pro (language understanding); GPQA Diamond (avg of 8 reps) and SuperGPQA (scientific QA); IFEval (avg of 8 reps, instruction following); DROP (reading comprehension); MATH500, AIME 2024 / 2025 (avg of 32 reps, mathematics); LiveCodeBench v5 (20240801–20250201) and v6 (20250201–20250501) (avg of 8 reps, coding). Sampling uses temperature $0.6$ and top-$p$ $0.95$ for all benchmarks; max generation length is $32{,}768$ for math / code / scientific QA and $8{,}192$ for others. Baselines: GPT-4o-0513, Claude-Sonnet-3.5-1022 (non-reasoning); o1-mini, QwQ-32B-Preview, DeepSeek-R1-Distill-Qwen-14B, DeepSeek-R1-Distill-Qwen-7B (reasoning).

Mathematics. MiMo-7B-RL achieves top-tier performance among models of comparable parameter sizes, trailing only slightly behind DeepSeek-R1-Distill-Qwen-14B on AIME 2024. On AIME 2025 it scores 55.4, exceeding o1-mini by 4.7 points; on MATH500 it scores 95.8.

Code. For algorithmic code generation MiMo-7B-RL is extremely impressive: on LiveCodeBench v5 it significantly outperforms o1-mini (57.8 vs 53.8); on the latest v6 it scores 49.3, surpassing QwQ-32B-Preview by over 10 points — demonstrating robust and stable capabilities.

General. Despite only using math and code problems for RL, MiMo-7B-RL maintains strong general performance, exceeding both QwQ-32B-Preview and DeepSeek-R1-Distill-Qwen-7B on average across general benchmarks.

Base-vs-SFT-vs-RL trajectory. Table 5 shows MiMo-7B-RL-Zero (trained from Base) exhibits a stronger growth trend, e.g., AIME 2024 improves from $32.9$ (Base) to $56.4$. However, MiMo-7B-RL (trained from SFT) achieves a higher ceiling, reaching the best results across all evaluated benchmarks (AIME24 $68.2$, AIME25 $55.4$, MATH500 $95.8$, LCB v5 $57.8$, v6 $49.3$). Conclusion: RL-from-base has steeper slope, RL-from-SFT reaches higher altitude.

SFT for format alignment. In the initial RL steps from MiMo-7B-Base, we observed that the model primarily learns to adapt the answer-extraction function (e.g., \boxed{} for math). So we investigated a "light-weight" SFT to help base align with the expected answer format. But MiMo-7B-RL-LiteSFT fails in both reasoning potential and final performance: while it starts higher than MiMo-7B-RL-Zero, it falls behind the base model's trajectory after just $500$ steps. Compared with MiMo-7B-RL (which undergoes "heavier" SFT), MiMo-7B-RL-LiteSFT exhibits a similar growth trend but significantly underperforms due to its inferior starting point — ultimately leading to poorer final results.

Interference between math and code. During later stages of RL training from MiMo-7B-Base, maintaining performance balance between math and coding tasks proved challenging. Between steps $2000$ and $2500$, code performance keeps improving while math fluctuates and even declines. In contrast, RL training on the cold-started SFT model shows consistent improvements across both domains. Analysis reveals that the base model, with its strong exploration capabilities, tends to hack the reward for math problems — for code, however, the test-case-based verifier makes reward exploitation significantly harder. This highlights the critical need for high-quality math problem sets to ensure robust RL.

Language-mixing penalty. Like DeepSeek-R1-Zero, we also observe language-mixing issues during RL training on MiMo-7B-Base. To mitigate, we introduced a language-mixing penalty into the reward, but designing it well is hard: detecting Chinese characters inside English responses is easy, but the reverse is far harder because math equations and code inherently contain English. As a result, the penalty fails to fully resolve language mixing and introduces a new reward-hacking risk (e.g., always generating English regardless of the question language).

SFT data scaling. We scale SFT data from $\sim 500$K to $\sim 6$M instances. This substantially improves reasoning and generalized dialogue capability without compromising the potential for subsequent RL. As detailed in Table 6, the 6M-SFT model markedly improves over the 500K-SFT counterpart on AIME 24 ($58.7\!\to\!68.3$), AIME 25 ($44.3\!\to\!50.9$), MATH500, GPQA Diamond, LiveCodeBench v5 and Alignbench v1.1. RL after the enhanced SFT also yields sustained gains (MiMo-7B-RL-0530 reaches AIME 24 $80.1$, AIME 25 $70.2$).

On-policy RL with extended generation budget. Our earlier empirical investigations indicated that a vanilla GRPO implementation was markedly prone to premature performance saturation. To mitigate, we adopted an on-policy RL algorithm, drawing parallels with the approach in MiMo-VL-7B-RL. On-policy RL proved remarkably stable while enabling sustained growth. Further, we found that continuous elevation of the generation-length budget during on-policy RL training consistently boosts performance: we systematically increased generation length from $32$K to $38$K to $48$K. This progressive extension was instrumental in our 7B model ultimately achieving parity with DeepSeek-R1 in mathematical reasoning. MiMo-7B-RL-0530 has been open-sourced.

This work introduces MiMo-7B, a series of LLMs which unlock advanced reasoning capabilities through optimized pre-training and post-training processes. Exposed to diverse reasoning patterns during pre-training, MiMo-7B-Base possesses exceptional reasoning potential, outperforming models of significantly larger scale. For post-training, with our robust and efficient RL frameworks, we trained MiMo-7B-RL-Zero and MiMo-7B-RL, which demonstrate superior reasoning capabilities across mathematics, code and general tasks. We hope this work offers insights for developing more powerful reasoning models.

Foundational architecture / pre-training references. Ainslie et al. (2023) — GQA: Grouped-Query Attention, used in MiMo-7B's attention block. Vaswani et al. (2017) — the original Transformer. Su et al. (2024) — RoFormer / Rotary Positional Embedding (RoPE), used as MiMo's position encoding. Dauphin et al. (2017) — Gated Linear Units, ancestor of SwiGLU. Zhang & Sennrich (2019) — RMSNorm, used in MiMo's pre-RMSNorm.

Loshchilov & Hutter (2019) — AdamW with decoupled weight decay, the MiMo-7B optimizer. Touvron et al. (2023) — Llama 2; Grattafiori et al. (2024) — Llama 3; Yang et al. (2024) — Qwen2.5; G. Team (2024) — Gemma 2; A. Liu et al. (2024a) — DeepSeek-V3 — all serve as architecture / training references and baseline comparisons.

Multi-token prediction & speculative decoding. Gloeckle et al. (2024) — multi-token prediction; Leviathan et al. (2023) and Xia et al. (2023) — speculative decoding (both motivating MiMo's MTP-inference design). Kwon et al. (2023) — vLLM / PagedAttention (MiMo adds MTP support on top of vLLM).

Reasoning-RL lineage. Guo et al. (2025) — DeepSeek-R1 (incentivizing reasoning via large-scale RL, the most direct comparator to MiMo). OpenAI (2024) — o1 / "learning to reason with LLMs". Anthropic (2025) — Claude 3.7 Sonnet. K. Team (2025b) — Kimi K1.5. B. Seed (2025) — Seed-Thinking-v1.5. C. Team (2025a) — MiMo-VL technical report (sibling work).

RL algorithms. Shao et al. (2024) — DeepSeekMath / GRPO, MiMo's base RL algorithm. Yu et al. (2025) — DAPO (an open-source LLM RL system at scale, source of community improvements like Clip-Higher and Dynamic Sampling that MiMo adopts). Hu et al. (2025) — Open-Reasoner-Zero. He et al. (2025) — Skywork Open Reasoner. Yue et al. (2025) — "Does RL really incentivize reasoning capacity beyond the base model?" (the pass@k upper-bound methodology MiMo follows).

RL infrastructure. Sheng et al. (2024) — HybridFlow / verl, the RLHF framework on which MiMo's Seamless Rollout Engine is built. Moritz et al. (2018) — Ray, the distributed runtime. Zhong et al. (2024b) — RLHFuse, on inter/intra-stage fusion for RLHF.

Data / corpus references. Penedo et al. (2023) — RefinedWeb; Penedo et al. (2024) — FineWeb. Paster et al. (2024) — OpenWebMath. Zhou et al. (2025) — MegaMath. Barbaresi (2021) — Trafilatura (web text extraction; MiMo replaces it with a self-developed HTML extractor optimized for math/code). Broder (1997) — original MinHash (used in MiMo's global deduplication).

Math & code benchmarks. MAA (2024, 2025) — AIME 2024 / 2025. Hendrycks et al. (2021b) — MATH dataset. Cobbe et al. (2021) — GSM8K. Lightman et al. (2024) — "Let's verify step by step" (process reward / verifier). Jain et al. (2024) — LiveCodeBench v5 / v6 (MiMo's main code benchmark). Chen et al. (2021) — HumanEval. J. Liu et al. (2023) — HumanEval+. Austin et al. (2021) — MBPP. Gu et al. (2024) — CRUXEval. IOI (2024) — International Olympiad in Informatics (inspires MiMo's test-difficulty–driven reward).

Language understanding / QA benchmarks. Suzgun et al. (2023) — BBH. Hendrycks et al. (2021a) — MMLU. Gema et al. (2024) — MMLU-Redux. Wang et al. (2024) — MMLU-Pro. Du et al. (2025) — SuperGPQA. Rein et al. (2024) — GPQA. Clark et al. (2018) — ARC. Zellers et al. (2019) — HellaSwag. Bisk et al. (2020) — PIQA. Sakaguchi et al. (2020) — Winogrande. Joshi et al. (2017) — TriviaQA. Kwiatkowski et al. (2019) — Natural Questions. Dua et al. (2019) — DROP. Lai et al. (2017) — RACE. Hsieh et al. (2024) — RULER (long-context). J. Liu et al. (2025) — long-context survey.

Chinese / instruction-following / alignment benchmarks. Huang et al. (2023) — C-Eval. H. Li et al. (2023) — CMMLU. X. Liu et al. (2024b) — AlignBench. Y. Liu et al. (2024c) — FineMath (Chinese). Zhou et al. (2023) — IFEval (instruction-following evaluation). Zhong et al. (2024a) — AGIEval. Radford et al. (2018) — original GPT pre-training paper. Zhu et al. (2024) — DeepSeek-Coder-V2 (closed-source code-model gap).

Footnote. MiMo-7B-RL-0530 (the further-scaled SFT+RL checkpoint discussed in §3.6) is open-sourced at https://huggingface.co/XiaomiMiMo/MiMo-7B-RL-0530.

Appendix A — Contributions and Acknowledgments. Within each role authors are listed alphabetically by their first name. Core Contributors: Bingquan Xia, Bowen Shen, Cici, Dawei Zhu, Di Zhang, Gang Wang, Hailin Zhang, Huaqiu Liu, Jiebao Xiao, Jinhao Dong, Liang Zhao, Peidian Li, Peng Wang, Shihua Yu, Shimao Chen, Weikun Wang, Wenhan Ma, Xiangwei Deng, Yi Huang, Yifan Song, Zihan Jiang.

Contributors: Bowen Ye, Can Cai, Chenhong He, Dong Zhang, Duo Zhang, Guoan Wang, Hao Tian, Haochen Zhao, Heng Qu, Hongshen Xu, Jun Shi, Kainan Bao, Kai Fang, Kang Zhou, Kangyang Zhou, Lei Li, Menghang Zhu, Nuo Chen, Qiantong Wang, Shaohui Liu, Shicheng Li, Shuhao Gu, Shuhuai Ren, Shuo Liu, Sirui Deng, Weiji Zhuang, Weiwei Lv, Wenyu Yang, Xin Zhang, Xing Yong, Xing Zhang, Xingchen Song, Xinzhe Xu, Xu Wang, Yihan Yan, Yu Tu, Yuanyuan Tian, Yudong Wang, Yue Yu, Zhenru Lin, Zhichao Song, Zihao Yue.