MiniMax-01: Scaling Foundation Models with

We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention.

This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering a 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI.

Large Language Models (LLMs) and Vision Language Models (VLMs) have made rapid progress in recent years, excelling at tasks like knowledge Q&A, complex reasoning, mathematics, coding, and vision-language understanding. The context window for most models currently ranges from 32K to 256K tokens. However, these lengths often fall short of practical needs—whether using a professional book as context, assisting with an entire programming project, or maximizing the potential of in-context learning through many-shot examples.

To meet these demands of long context, we identify that the primary bottleneck lies in the attention mechanism of the conventional Transformer architecture. The quadratic computational complexity associated with this mechanism severely impedes the ability to efficiently train and serve models with arbitrarily long context windows. Although various approaches have been proposed to address this issue, including sparse attention, linear attention, long convolutions, state space models, and linear RNNs, few have been successfully scaled to commercial-grade models that match the capabilities of the top LLMs.

In this report, we adopt linear attention to extend the model's context length and select Lightning Attention—a novel I/O-aware implementation of linear attention. To maintain the in-context retrieval capability, we integrate Lightning Attention with softmax attention in a hybrid manner, where every 8th layer is a softmax attention layer. To further scale the model and increase its capacity, we adopt MoE. Building upon these innovations, we propose MiniMax-Text-01, a model with 456 billion total parameters and 45.9 billion activated parameters per token, trained on 12 trillion tokens.

We summarize our contributions as follows: (1) We build a model that rivals the top-tier closed-source models on standard academic benchmarks while supporting a context window 20-32 times longer. (2) We demonstrate the first successful large-scale implementation of linear attention, scaling the model to 456 billion parameters. (3) We outline a practical approach and experimental methodology for the exploration of new architectures, including the integration of linear attention with softmax attention, the scaling-law-based architecture selection, and various engineering optimizations. (4) We publicly release the weights and offer a cost-effective API.

In this section, we present the design of our network architecture. To achieve optimal performance within constrained resources and better handle longer sequences, we adopt MoE approach and employ linear attention as much as possible instead of the traditional softmax attention used in standard transformers. Our design follows the Transformer-style block, with each block comprising a channel mixer (an attention block) and a feature mixer (an MLP block). We employ two types of channel mixers: Lightning Attention and softmax attention.

The two attention mechanisms are stacked in a fixed pattern: every 8 transformer blocks consist of 7 lightning attention blocks plus 1 softmax attention block. This 7:1 hybrid ratio is empirically the sweet spot—pure lightning attention significantly underperforms on in-context retrieval tasks (such as Needle-in-A-Haystack and SCROLLS), while inserting one softmax attention layer per 8 layers restores retrieval capability with negligible throughput cost. For the MoE side, each MLP block is a 32-expert mixture with top-2 routing.

MoE provides a pathway to enhance both scalability and efficiency compared to the dense version. Typically, MoE is a substitute for the feed forward networks (FFN) in feature-mixer layers. Given an input token x, the MoE layer routes it to the top-k experts among E experts, and the output is a weighted sum of the selected experts' outputs.

To prevent routing collapse, we adopt the GShard-style auxiliary loss to balance expert utilization. The auxiliary loss is defined as $L_{\mathrm{aux}} = \alpha_{\mathrm{aux}} \cdot \tfrac{1}{E}\sum_{i=1}^{E} f_i \cdot m_i$, where $f_i$ denotes the fraction of tokens dispatched to expert $i$ and $m_i$ denotes the average of softmax probabilities assigned to expert $i$ across the batch. To prevent stragglers in distributed training, each expert has a capacity limit and tokens exceeding this limit are dropped.

However, naively applying capacity-based token drop can result in severe load imbalance across expert parallel (EP) groups, especially in early training. To mitigate this, we propose a Global Router: before dispatching, the router first AllGathers the routing statistics across all EP groups to compute a global view of how many tokens each expert is about to receive. Tokens are then redistributed based on this global statistic to minimize cross-group drops. This global routing mechanism effectively reduces the overall token drop rate, ensuring training stability.

Linear attention utilizes the "right product kernel trick" to transform quadratic computational complexity into linear complexity. By taking TransNormer as an example, the NormAttention mechanism can be written as $O = \mathrm{Norm}((QK^\top)V)$, where $Q, K, V \in \mathbb{R}^{n\times d}$ are the query, key, and value matrices, with $n$ for sequence length and $d$ for feature dimension. The equation can be transformed into its linear variant using right matrix multiplication:

The linear formulation facilitates efficient recurrent prediction with a training complexity of $O(nd^2)$. Furthermore, linear attention ensures a constant computational complexity of $O(d^2)$ per step during inference, irrespective of the sequence length. This is accomplished by recurrently updating the term $K^\top V$, obviating the need for repetitive computation of the entire attention matrix. In contrast, softmax attention incurs $O(nd^2)$ complexity per step during inference because the KV cache grows linearly with $n$.

When addressing causal language modeling tasks, the efficacy of the right product is compromised, necessitating the computation of cumsum. This limitation impedes the realization of highly efficient parallel computation, which likely explains why, despite being proposed nine years ago, none of the current leading open-source LLMs—including LLaMA3, Qwen2.5, DeepSeek-V3, and Mistral—have adopted this linear attention mechanism.

Lightning Attention represents an I/O-aware, optimized implementation of TransNormer. This approach identifies the primary bottleneck in the computational efficiency of existing linear attention mechanisms: the slow cumsum operation inherent in causal language modeling. To alleviate this problem, Lightning Attention proposes a novel tiling technique that effectively circumvents the cumsum operation. The key innovation lies in the strategic division of the attention calculation into two distinct components: intra-block and inter-block computations.

The left product attention calculation is employed for intra-block operations, while the right product is utilized for inter-block operations. This division is crucial because the intra-blocks can be significantly reduced in size, thereby ensuring that the overall computational complexity remains linear. The left product in causal attention is defined as $O = [(QK^\top)\odot M]V$ where $M_{ts}=1$ if $t \geq s$, otherwise $0$. The right product can be computed recursively as:

While the recursion exhibits linear computational complexity, it is inherently unparallelizable. The fundamental concept underlying Lightning Attention involves the use of tiling: partition $Q, K, V$ into blocks along the row dimension. For a block of size $m$, the block-form formula becomes $O_1 = Q_1 \mathrm{kv}_0 + [(Q_1 K_1^\top)\odot M] V_1$, where the intra-block term uses the left product (efficient because $m$ is small) and the inter-block term $Q_1 \mathrm{kv}_0$ uses the right product (efficient because the recurrence only needs to step block-by-block, not token-by-token).

In our model, we set $d=128$ per head and the block size to 256, which empirically optimizes the throughput on Hopper-class GPUs. We further design dedicated CUDA kernels for the forward and backward passes that fuse intra-block softmax-free attention with inter-block KV update, achieving end-to-end linear scaling in both training and inference on sequences from 8K up to 1M tokens.

Although Lightning Attention achieves linear complexity and even slightly lower language modeling loss than softmax attention, our ablations reveal a critical weakness: in pure-Lightning models, retrieval-style tasks (NIAH, SCROLLS) degrade sharply once the context exceeds the trained length. The reason is that linear attention compresses the entire history into a single bounded state $K^\top V \in \mathbb{R}^{d\times d}$, which has finite capacity. While softmax attention has effective capacity O(d) per query but re-reads the full history each step, linear attention has bounded state but cannot revisit raw tokens.

We propose a hybrid architecture: 7 lightning attention layers followed by 1 softmax attention layer, repeated. This 7:1 ratio combines the throughput of linear attention (87.5% of layers) with the precise retrieval capability of softmax attention (12.5% of layers). Empirically, the hybrid-lightning model matches softmax attention on retrieval benchmarks while being 2-3× faster on long context. We fit scaling laws on three architectures (softmax / pure lightning / hybrid-lightning) across model sizes from 70M to 7B and observe the hybrid scaling exponent slightly outperforms both pure variants.

We conduct extensive ablations to choose: (a) the location of MoE (replace FFN vs. mix with dense), (b) the routing strategy (top-1 vs. top-2 vs. top-K with K>2), (c) the normalization scheme (Pre-LayerNorm vs. Post-LayerNorm vs. DeepNorm), and (d) the position embedding for softmax attention (RoPE on all dims vs. half dims vs. no RoPE). The winning configuration is: 32 experts with top-2 routing replacing the FFN, Post-LayerNorm with DeepNorm scaling factors $\alpha=(2N)^{0.25}, \beta=(8N)^{-0.25}$, and RoPE applied to half of the softmax attention dimensions with base frequency progressively scaled from 10K to 10M.

A noteworthy finding is that Post-LayerNorm with DeepNorm scaling is critical for deep MoE models with hybrid attention. Pre-LayerNorm models we trained tend to diverge at 80 layers beyond 2T tokens, while Post-LN + DeepNorm remains stable. Another key finding: applying RoPE to only half of the dimensions enables length extrapolation without performance degradation, while applying RoPE to all dimensions causes the model to overfit to the trained length.

Our primary goal is to strike a balance between performance and inference efficiency. Single-device inference offers superior efficiency compared to multi-device implementations by eliminating cross-machine communication overhead. Consequently, we constrain the model's total parameters to under 500B, ensuring compatibility with single-node inference on an 8×80GB configuration for sequences up to 1M tokens under 8-bit quantization. Given our limited training budget, we formulate the following optimization problem to determine optimal parameter allocations:

We propose a parametric loss form conditioned on the number of experts $E$: $L(P_{\mathrm{act}}, T \mid E) = d + a P_{\mathrm{act}}^\alpha + b T^\beta + c (P_{\mathrm{act}} T)^\gamma$, where $a, b, c, d, \alpha, \beta, \gamma$ are fitted as functions of $E$. Based on this, the optimal configuration is: 45.9B activation parameters / 456B total parameters / 32 experts / top-2 routing. The final architecture has 80 transformer blocks, hidden size 6144, attention heads 64 × head dim 128 (GQA group size 8 for softmax layers), FFN intermediate 9216 per expert.

In this section, we present our computation part, including the training and inference. In this project, we have a dynamically changing GPU cluster, where the number of H800 GPUs ranges from 1500 to 2500. An efficient architecture necessitates robust implementation optimization to fully harness its computational benefits at scale. We present three key optimization strategies that primarily address: (1) all-to-all (a2a) communication overhead in MoE training; (2) supporting at least 1M token context in both training and inference; (3) making lightning attention practical for production inference.

The primary objective in optimizing the MoE architecture is to minimize communication overhead, particularly for MoE models that utilize all-to-all (a2a) communication. To address this, we implement a token-grouping-based overlap scheme. In this scheme, the a2a communication is performed within the expert parallel (EP) communication group, and it overlaps with the processing of tokens from different expert groups. To ensure the correctness of the communication results, we restructure the all-to-all dispatch into a sequence of pair-wise sends and receives that can be pipelined with FFN compute.

We further introduce two new parallel dimensions for the MoE component: Expert Tensor Parallel (ETP) and Expert Data Parallel (EDP). ETP partitions each expert's weight tensor across multiple GPUs (analogous to tensor parallelism for dense MLPs but applied per-expert), while EDP replicates each expert across multiple GPUs to absorb load imbalance. By carefully overlapping EP-ETP communication with computation, we reduce the pure communication overhead of the MoE component by 50% compared to the pre-optimization state.

A significant challenge in long context training is that real training samples are difficult to standardize into a uniform length. The conventional approach of padding samples to a uniform length leads to substantial computational waste. At 1M sequence length scale, this waste becomes particularly significant. To address this, we adopt a "data-packing" technique: different samples are concatenated end-to-end along the sequence dimension, and an attention mask is constructed to prevent cross-sample attention. This eliminates almost all padding waste.

For sequence parallelism, vanilla Ring Attention requires each sample to be padded to a multiple of $2\cdot\mathrm{size}_{CP}$, which—combined with the packed data—still wastes significant compute. We propose varlen Ring Attention: rather than padding each packed segment, we directly slice the packed sequence along its true length boundaries and have each CP rank handle its own length range. This eliminates intra-pack padding altogether.

For lightning attention specifically, we propose LASP+ (Linear Attention Sequence Parallel Plus). The original LASP transfers $KV$ states between CP ranks via a serial send-recv chain (rank 0 → 1 → 2 → ... → CP-1), making the latency proportional to CP size. LASP+ instead has each rank locally compute its prefix-sum of $KV$, then performs one AllGather across CP ranks, and each rank picks the appropriate prefix-sum tensor. This converts the O(CP) serial chain into an O(log CP) parallel AllGather, drastically speeding up sequence parallelism for linear attention.

LASP+ proceeds in three steps: (1) Local Prefix Sum Calculation: each CP rank computes its local $K^\top V$ block. (2) Global Synchronization via AllGather: all ranks AllGather their local $K^\top V$ blocks. (3) Prefix Sum Computation: each rank selects the specific CP rank's $KV_L$ on which it depends. The integration of varlen and LASP+ ensures the system handles diverse input lengths without compromising efficiency.

The initial implementation of the lightning attention mechanism is primarily research-oriented and not yet suitable for practical applications, especially for inference. We implement four optimization strategies: (a) Batched Kernel Fusion: fuse the $K^\top V$ update, $Q \cdot KV$ projection, and intra-block masked GEMM into a single kernel; (b) Separated Prefill and Decoding Execution: use different kernels and schedule for the prefill (long, parallel) vs. decoding (short, sequential) phases.

(c) Multi-level Padding: instead of a single fixed block size, we offer multiple block-size kernels (32 / 64 / 128 / 256) and choose the smallest that fits the current sequence length—this reduces the padding waste in the decoding phase where each step processes only one new token; (d) StridedBatchedMatmul: a CUDA-level optimization that batches many small matmuls (one per sequence) into a single launch using stride-based addressing, eliminating kernel launch overhead. On H20 GPUs, the end-to-end MFU exceeds 75%.

The pre-training corpus for MiniMax-Text-01 encompasses a comprehensive and meticulously curated dataset, incorporating diverse sources including academic literature, books, web content, and programming code. We enhance corpus quality through several strategic dimensions: (1) Data Quality Enhancement via a sophisticated filtering pipeline combining rule-based cleaning and deduplication; (2) Data Source Diversity: we explicitly target high-knowledge-density domains; (3) Domain-Specific Data Augmentation: we identify domains under-represented in raw web crawls and supplement them with curated sources.

We design a controlled data-experimentation framework that allows us to evaluate the impact of a candidate data recipe with a small training run (a few billion tokens), then extrapolate the result to the full-scale 12T-token run. By carefully controlling the repetition and quality of the training data, we achieve a more efficient and effective data mixture. The final pre-training corpus contains roughly 12 trillion tokens after deduplication, tokenized with a custom 200K BPE vocabulary that balances English, Chinese, and code coverage.

Initial Pre-training. We initialize all parameters using Xavier, with DeepNorm scaling factors $\alpha = (2N)^{0.25}, \beta = (8N)^{-0.25}$, where $N$ denotes the number of layers. We use AdamW with $\beta_1=0.9, \beta_2=0.95$ and weight decay 0.1. The training sequence length is 8192 and the batch size is progressively scaled: 16M → 32M (at 69B tokens) → 64M (at 790B tokens) → 128M (at 4.7T tokens), where it remains for the rest of training. The base learning rate starts at 2e-4, decays to 1.3e-4 across the bulk of training, then fast-decays to 3e-5 in the final 1T tokens.

Long Context Extension. After initial pre-training, we extend the context length in three progressive stages. Stage 1: 128K, RoPE base 5M, 300B tokens, with a sample mix of 30% Short (≤32K) / 70% Medium (32-128K) / 0% Long. Stage 2: 512K, RoPE base 10M, 32B tokens, with 35% Short / 35% Medium / 30% Long. Stage 3: 1M, RoPE base 10M, 26B tokens, with 30% Short / 30% Medium / 40% Long. After Stage 3, the model can extrapolate to 4M token NIAH without further training—a property attributed to the half-dimension RoPE design.

We present a thorough post-training framework designed to enhance the model's general performance, long-context capability, and real-world applicability. Our approach begins with the creation of a diverse, high-quality prompt dataset, accompanied by a hierarchical reward system that evaluates responses across multiple dimensions: correctness, truthfulness, helpfulness, and harmlessness. The training consists of Supervised Fine-Tuning (SFT), offline Reinforcement Learning (DPO), and online Reinforcement Learning (a modified GRPO variant).

We incorporate the offline RL phase, i.e., Direct Preference Optimization (DPO), to optimize the model's performance across diverse prompt distributions, owing to its simplicity and ease of data construction for long-context scenarios. We specifically focus on prompts that maintain distributional consistency with those utilized in the SFT stage. The experimental protocol involves generating responses with varying temperature parameters for each prompt, followed by systematic evaluation using the reward models. We then identify the best and the worst responses to construct preference pairs for DPO training.

Online learning demonstrates superior sample efficiency and cross-domain generalization compared to offline methodologies. Therefore, we implement online RL to improve model performance, particularly in mathematical reasoning. We propose a modified Group Relative Policy Optimization (GRPO) approach with three key innovations: (a) Importance Sampling Weight Clipping—we add a second clipping that abandons cases with large policy ratio and negative advantage; (b) KL Divergence Optimization—we reformulate the KL term; (c) Balanced Advantage Estimation—we equalize reward contributions between positive and negative examples.

The reformulated KL divergence above (where $\mathrm{SG}(\cdot)$ denotes stop-gradient) maintains policy consistency while reducing gradient variance, addressing the gradient instability seen in vanilla GRPO. Combined with two-sided clipping for the importance sampling weight, our modified GRPO trains stably on long-sequence math prompts where vanilla GRPO would diverge.

We propose a systematic multi-stage training methodology to enhance the model's capacity for processing extended contexts. The RoPE base frequency is maintained at 10 million throughout the post-training phase. Stage I: Initial Short-Context SFT with sequences of 8,192 tokens (epoch 2, batch 128, lr 1e-5 cosine to 1e-6). Stage II: Long-Context SFT with sequences of 1,032,192 tokens (epoch 2, batch 80, lr 3e-6 constant). Stage III: Short-Context DPO at 8K (epoch 1, batch 64, lr 5e-7 cosine).

Stage IV: Long-Context DPO at ~1M tokens (epoch 1, batch 64, lr 5e-7 constant). Stage V: Short-Context online RL at 8K (epoch 1, batch 512, lr 1e-6 cosine). This staged design—short → long → short → long → short—prevents catastrophic forgetting of short-context capabilities during long-context training, and prevents the model from drifting too far when running RL only at short context.

By integrating an image encoder and an image adapter into our MiniMax-Text-01 model, we develop MiniMax-VL-01, which extends the model's capabilities to visual understanding. To ensure robust visual understanding, we design a proprietary dataset and implement a multi-stage training strategy, where the newly introduced image encoder and adapter first undergo large-scale visual pre-training, followed by comprehensive fine-tuning of the entire pipeline. The image encoder is a ViT-L/14 trained from scratch with 694M image-caption pairs (80% at 224×224 resolution → fine-tuned at 336×336).

The ViT achieves 80.55% zero-shot accuracy on ImageNet-1K, comparable to the openly released ViT-L/14 by CLIP. The image adapter is a 2-layer MLP that maps the 1024-dim ViT outputs into the 6144-dim hidden space of MiniMax-Text-01. The four-stage training is: Stage I—alignment with 80B caption tokens (only ViT + adapter updated); Stage II—visual instruction SFT with 420B tokens at multimodal:text = 20:1, full network updated; Stage III—UX enhancement with 44.8B tokens of curated user-experience data; Stage IV—DPO with 40K image-text preference pairs, with early stopping to prevent over-alignment. Total vision-language training: 512B tokens.

In this report, we present MiniMax-Text-01 and MiniMax-VL-01, two novel models developed entirely from the ground up. These models demonstrate top-tier performance across standard benchmarks, particularly excelling in long-context processing with the ability to handle context windows of up to 4 million tokens. Our research findings challenge the prevailing assumption that state-of-the-art language models must be built upon traditional attention mechanisms. By strategically integrating linear attention with softmax attention in a 7:1 ratio, we have developed a more efficient hybrid architecture that scales gracefully to commercial-grade size.

While MiniMax-Text-01 and MiniMax-VL-01 show strong performance in general language and vision-language tasks, we acknowledge several limitations: (1) Long-Context Evaluation: current evaluation datasets for long-context retrieval do not fully reflect real-world usage. (2) Model Architecture: the model currently retains a 1/8 component with vanilla softmax attention, which still incurs O(n²) cost; we plan to explore replacing this with more efficient attention mechanisms in the next version. (3) Complex Programming Tasks: the model's performance on advanced programming tasks (e.g., competitive programming) is still limited at the moment.