2025-05-15

The pursuit of AGI / ASI has long motivated humanity, and recent advances in large foundation models—GPT-4o, Claude 3.7, Gemini 2.5, DeepSeek-V3, Llama-4, and Qwen2.5—have shown significant progress. These models are trained on vast datasets across trillions of tokens, effectively distilling human knowledge into parameters. Recent reasoning models optimized through reinforcement learning (e.g., o3, DeepSeek-R1) further highlight the potential of inference-time scaling. While most state-of-the-art models remain proprietary, the open-source community has dramatically narrowed the gap, with an increasing number of top-tier models being released openly.

In this work we introduce Qwen3, the latest series in our Qwen foundation-model family. We release both dense and Mixture-of-Experts (MoE) models ranging from 0.6B to 235B parameters; the flagship Qwen3-235B-A22B is an MoE model with 235B total parameters and 22B activated per token. Qwen3 integrates two operating modes—thinking mode and non-thinking mode—into a single model, eliminating the need to switch between separate models like Qwen2.5 and QwQ. It further introduces thinking budgets, allowing fine-grained user control over reasoning effort. Qwen3 is pre-trained on 36 trillion tokens covering up to 119 languages and dialects, dramatically expanding multilingual capability.

Pre-training of Qwen3 follows a three-stage strategy on the 36T-token corpus. Stage 1 trains on about 30T tokens to build a strong general-knowledge foundation. Stage 2 further trains on knowledge-intensive data emphasizing STEM and coding to strengthen reasoning. Stage 3 extends context length from 4,096 to 32,768 tokens using long-context corpora. To efficiently expand the data, the team fine-tunes Qwen2.5-VL to extract text from large PDF collections, and uses Qwen2.5-Math and Qwen2.5-Coder to synthesize trillions of math and code tokens.

For post-training, Qwen3 uses a multi-stage approach that empowers both thinking (reasoning) and non-thinking modes. The first two stages develop strong reasoning via long-CoT cold-start fine-tuning followed by RL focused on math and code. The next two stages merge data with and without reasoning paths and then apply general-domain RL to broaden capabilities. For smaller models the team adopts strong-to-weak distillation, combining off-policy and on-policy knowledge transfer from larger teachers, which significantly outperforms direct RL in both quality and training efficiency.

Evaluation across a broad benchmark suite shows that Qwen3 base models achieve state-of-the-art performance, while post-trained models—in either mode—compete with leading proprietary and large MoE models such as o1, o3-mini, and DeepSeek-V3. The flagship Qwen3-235B-A22B reaches 85.7 on AIME'24, 81.5 on AIME'25, 70.7 on LiveCodeBench v5, 2,056 on CodeForces, and 70.8 on BFCL v3. Other Qwen3 models also show strong performance relative to size. Crucially, increasing the thinking budget yields consistent improvements across tasks, demonstrating user-controllable inference-time scaling.

The Qwen3 series includes 6 dense models—Qwen3-0.6B / 1.7B / 4B / 8B / 14B / 32B—and 2 MoE models—Qwen3-30B-A3B and the flagship Qwen3-235B-A22B (235B total, 22B activated per token). The dense architecture is similar to Qwen2.5, using Grouped Query Attention (GQA), SwiGLU, Rotary Positional Embeddings (RoPE), and RMSNorm with pre-normalization. We remove the QKV-bias used in Qwen2 and introduce QK-Norm into the attention mechanism to ensure stable training. Key dense-model configurations (layers, Q/KV heads, tie-embedding, context length) appear in Table 1.

The Qwen3 MoE models share the same fundamental architecture as the dense models. Following Qwen2.5-MoE, we use fine-grained expert segmentation, with 128 total experts and 8 activated per token. Unlike Qwen2.5-MoE, Qwen3-MoE excludes shared experts. We further adopt a global-batch load balancing loss to encourage expert specialization. These architectural and training innovations yield substantial performance gains on downstream tasks. Qwen3 reuses Qwen's byte-level BPE (BBPE) tokenizer with vocabulary size 151,669.

Dense model configurations (Table 1): Qwen3-0.6B / 1.7B use 28 layers, 16/8 Q/KV heads, tied embeddings, 32K context; Qwen3-4B uses 36 layers, 32/8 heads, tied, 128K context; Qwen3-8B uses 36 layers, 32/8 heads, untied, 128K; Qwen3-14B has 40 layers, 40/8 heads, untied, 128K; Qwen3-32B has 64 layers, 64/8 heads, untied, 128K. MoE configurations (Table 2): Qwen3-30B-A3B: 48 layers, 32/4 Q/KV heads, 128/8 experts, 128K; Qwen3-235B-A22B: 94 layers, 64/4 Q/KV heads, 128/8 experts, 128K.

Compared with Qwen2.5, we significantly expanded the scale and diversity of training data—collecting twice as many pre-training tokens covering three times more languages. All Qwen3 models are trained on a dataset spanning 119 languages and dialects with 36 trillion tokens in total. The data includes high-quality content across coding, STEM, reasoning tasks, books, multilingual texts, and synthetic data. To expand the corpus, we use Qwen2.5-VL to OCR a large volume of PDF documents and then refine the recognized text with Qwen2.5, yielding trillions of additional high-quality tokens.

We also employ Qwen2.5, Qwen2.5-Math, and Qwen2.5-Coder to synthesize trillions of tokens in diverse formats—textbooks, question answering, instructions, and code snippets—covering dozens of domains. We further enlarge the corpus by adding multilingual data and increasing supported languages from 29 to 119, sharply improving cross-lingual capability. A multilingual data-annotation system was developed to enhance quality and diversity, annotating over 30T tokens along dimensions such as educational value, fields, domains, and safety. Unlike prior work that optimizes data mixtures at source or domain level, we optimize at the instance level through ablation experiments on small proxy models with these fine-grained labels.

Qwen3 is pre-trained in three stages. (1) General Stage (S1): all Qwen3 models are trained on over 30T tokens at a sequence length of 4,096; the data covers 119 languages and dialects, building broad language proficiency and world knowledge. (2) Reasoning Stage (S2): we increase the proportion of STEM, coding, reasoning, and synthetic data, and further pre-train with about 5T higher-quality tokens at sequence length 4,096, accelerating learning-rate decay during this stage. (3) Long Context Stage: we collect high-quality long-context corpora and pre-train on hundreds of billions of tokens at sequence length 32,768—75% of which is text between 16,384 and 32,768 tokens long, the remaining 25% between 4,096 and 16,384.

Following Qwen2.5, we raise the RoPE base frequency from $10{,}000$ to $1{,}000{,}000$ using the ABF technique. At inference we further introduce YARN and Dual Chunk Attention (DCA), giving a four-fold increase in sequence-length capacity. Similar to Qwen2.5, we develop scaling laws to predict optimal hyper-parameters (learning rate schedule, batch size) across the three stages. Through extensive experiments we systematically study how model architecture, training data, training stage, and optimal training hyper-parameters interact, then set predicted optima for each dense or MoE model.

We comprehensively evaluate Qwen3 base models on 15 benchmarks covering general (MMLU / MMLU-Pro / MMLU-Redux / BBH / SuperGPQA), math & STEM (GPQA / GSM8K / MATH), coding (EvalPlus, MultiPL-E, MBPP-3shot, CRUX-O), and multilingual tasks (MGSM, MMMLU, INCLUDE). Baselines include Qwen2.5 base, DeepSeek-V3 Base, Gemma-3, Llama-3, and Llama-4 series. Three key findings: (i) Qwen3-235B-A22B-Base outperforms previous open-source SOTA dense and MoE models on most tasks with significantly fewer total or activated parameters; (ii) Qwen3 MoE base models match Qwen3 dense base with only 1/5 activated parameters; (iii) Qwen3 dense base models match Qwen2.5 at higher parameter scales (e.g., Qwen3-1.7B/4B/8B/14B/32B-Base match Qwen2.5-3B/7B/14B/32B/72B-Base respectively).

Specifically, Qwen3-235B-A22B-Base achieves the highest scores on most evaluated benchmarks. Versus Llama-4-Maverick-Base (about 2× the parameters), Qwen3-235B-A22B-Base still wins on most benchmarks. Versus DeepSeek-V3-Base, Qwen3-235B-A22B-Base wins 14 of 15 benchmarks with only ~1/3 total parameters and 2/3 activated parameters. Versus the previous flagship Qwen2.5-72B-Base, Qwen3-235B-A22B-Base surpasses it on every benchmark. Especially in STEM, coding, and reasoning, Qwen3 dense base models exceed Qwen2.5 base models at higher parameter scales—evidence of the gains from more data and architectural improvements (QK-Norm, global-batch LB, fine-grained MoE).

The post-training pipeline pursues two objectives: Thinking Control—merging non-thinking and thinking modes in one model with user-controllable token budgets—and Strong-to-Weak Distillation—reducing the compute and effort to build small models by leveraging large teachers. Flagship Qwen3-235B-A22B and Qwen3-32B undergo a sophisticated four-stage process: Stage 1 Long-CoT Cold Start, Stage 2 Reasoning RL, Stage 3 Thinking Mode Fusion, Stage 4 General RL. Lightweight models bypass the four stages and are produced by distillation. We find that directly distilling teacher logits into lightweight students improves Pass@1 and Pass@64 while using only ~1/10 of the GPU hours of the four-stage method.

For Stage 1, we curate a dataset spanning math, code, logical reasoning, and general STEM problems, each paired with verified reference answers or code-based test cases. Query filtering uses Qwen2.5-72B-Instruct to remove queries that are hard to verify, include multiple sub-questions, ask for free-form generation, or can be answered correctly without CoT. After reserving a validation set, we generate N candidates per remaining query with QwQ-32B; when QwQ-32B consistently fails, human annotators check accuracy. We then strictly filter out responses with wrong final answers, heavy repetition, evident guessing without reasoning, inconsistency between thinking and summary, inappropriate language mixing, or suspicious similarity to validation items.

A carefully chosen subset of the refined data is used for the initial cold-start training of reasoning patterns. The objective at this stage is to instill foundational reasoning patterns rather than to maximize immediate reasoning performance—so the model's potential is not capped and remains flexible for later RL. To realize this we deliberately keep both the number of training samples and the number of training steps small in this preparatory phase.

The query-verifier pairs used in Reasoning RL must satisfy four criteria: (1) not used in cold-start; (2) learnable for the cold-start model; (3) as challenging as possible; (4) covering broad sub-domains. We ultimately collect 3,995 query-verifier pairs and employ GRPO to update the model. We find that a large batch size and a high number of rollouts per query, combined with off-policy training for sample efficiency, are beneficial. We also address exploration–exploitation balance by controlling the model's entropy to increase steadily or remain stable, which is crucial for stable training.

As a result, training reward and validation performance improve consistently over a single RL run without any manual hyperparameter intervention. For example, the AIME'24 score of Qwen3-235B-A22B rises from 70.1 to 85.1 over 170 RL training steps. This stable monotonic improvement is in part what enables Qwen3 to push reasoning benchmarks substantially without the typical mid-training tuning loops seen in RLHF.

Thinking Mode Fusion integrates the "non-thinking" capability into the previously trained "thinking" model so a single model can serve both modes. We do continual SFT on the Reasoning RL model and design a chat template to fuse the two modes. The SFT dataset combines "thinking" and "non-thinking" data: thinking data is generated via rejection sampling on Stage 1 queries using the Stage 2 model itself (so Stage 2 performance is not compromised); non-thinking data is curated to cover coding, math, instruction following, multilingual tasks, creative writing, QA, and role-playing, with automatically generated checklists for quality and a higher proportion of translation tasks to boost low-resource languages.

For chat-template design, we introduce /think and /no think flags in user queries or system messages so the model selects the appropriate mode. For non-thinking samples we keep an empty <think></think> block in the assistant's response to enforce internal format consistency; developers can also prepend an empty think block to prevent thinking. By default the model is in thinking mode; we add training samples in which user queries omit any flag. In complex multi-turn dialogs we randomly insert multiple /think and /no think flags, and the model's response adheres to the last flag encountered.

Thinking Budget. Once the model learns to respond in both modes, it naturally handles intermediate cases—responding based on incomplete thinking. This lays the foundation for budget control: when the thinking length reaches a user-defined threshold, we manually halt thinking and insert the stop-thinking instruction "Considering the limited time by the user, I have to give the solution based on the thinking directly now. </think>. ". After this insertion the model generates a final response based on its accumulated reasoning so far. This ability is not explicitly trained—it emerges naturally as a result of Thinking Mode Fusion.

The General RL stage broadly enhances the models' capabilities and stability across diverse scenarios. We establish a sophisticated reward system covering 20+ distinct tasks, each with customized scoring criteria. Target capabilities include: Instruction Following (content, format, length, structured output); Format Following (respond to /think and /no think flags, consistently use <think>/</think> tokens); Preference Alignment (helpfulness, engagement, style on open-ended queries); Agent Ability (tool invocation through designated interfaces with real-environment feedback during multi-turn rollouts); and Specialized Scenarios such as RAG (rewards guiding accurate, contextually appropriate responses to minimize hallucination).

We use three reward types: (1) Rule-based reward—widely used in Reasoning RL and also useful for instruction/format adherence; well-designed rule rewards assess correctness precisely and prevent reward hacking. (2) Model-based reward with reference answer—we provide a reference and prompt Qwen2.5-72B-Instruct to score the response; this handles diverse tasks flexibly without strict formatting and avoids false negatives common to purely rule-based rewards. (3) Model-based reward without reference—using human preference data, we train a reward model to assign scalar scores; this covers a broader range of queries and effectively boosts engagement and helpfulness.

The Strong-to-Weak Distillation pipeline is specifically designed for lightweight models—five dense (Qwen3-0.6B, 1.7B, 4B, 8B, 14B) and one MoE (Qwen3-30B-A3B). It enhances performance while transferring robust mode-switching capabilities. The pipeline has two phases. (1) Off-policy Distillation: we combine teacher outputs generated in /think and /no think modes for response distillation, helping the student develop basic reasoning and mode-switching abilities. (2) On-policy Distillation: the student generates on-policy sequences in either mode, and is fine-tuned by aligning its logits with those of the teacher (Qwen3-32B or Qwen3-235B-A22B) to minimize the KL divergence.

We evaluate instruction-tuned models in both thinking and non-thinking modes across several dimensions. General Tasks: MMLU-Redux, GPQA-Diamond (sample 10× and average), C-Eval, LiveBench (2024-11-25). Alignment Tasks: IFEval strict-prompt accuracy, Arena-Hard, AlignBench v1.1, plus Creative Writing V3 and WritingBench for writing. Math & Text Reasoning: MATH-500, AIME'24, AIME'25 (sample 64× and average per question), ZebraLogic, AutoLogi. Agent & Coding: BFCL v3 (FC format with YARN-extended 64K context for multi-turn), LiveCodeBench v5, and CodeForces ratings via CodeElo.

Key findings: Qwen3-32B (Thinking) outperforms QwQ-32B on 17 of 23 benchmarks, making it the new SOTA reasoning model at the 32B "sweet spot"; it competes with closed-source OpenAI o3-mini (medium) with better alignment and multilingual performance. Qwen3-32B (Non-thinking) beats all non-thinking baselines on most benchmarks and exceeds the previous Qwen2.5-32B-Instruct with significantly fewer activated / total parameters. For 8B / 4B / 1.7B / 0.6B, Qwen3 models in thinking mode compete with DeepSeek-R1-Distill-Qwen-14B / 32B; in non-thinking mode they compete with Qwen2.5-7B-Instruct / 14B-Instruct, LLaMA-3.1-8B-Instruct and Gemma-3-12B-IT—often surpassing them at their respective sizes.

The Effectiveness of Thinking Budget. To verify that Qwen3 can enhance its intelligence by leveraging an increased thinking budget, we adjust the allocated thinking budget on four benchmarks across Mathematics, Coding, and STEM. The resulting scaling curves (Figure 2) show that Qwen3 exhibits scalable and smooth performance improvement correlated with the allocated thinking budget. Moreover, if we extend the output length beyond 32K, the model's performance is expected to improve further; we leave this exploration as future work.

The Effectiveness and Efficiency of On-Policy Distillation. Starting from the same off-policy distilled 8B checkpoint, we compare on-policy distillation against direct RL on math/code queries (Table 21). Distillation achieves significantly better performance while requiring only ~1/10 of the GPU hours of RL. Moreover, distillation from teacher logits expands the student's exploration space and improves Pass@64 on AIME'24/'25, while RL does not improve Pass@64. Concretely, on Qwen3-8B: AIME'24 jumps from 67.6 (RL) to 74.4 (on-policy distill); AIME'25 from 55.5 to 65.5; MATH500 from 94.8 to 97.0; LiveCodeBench v5 from 52.9 to 60.3—at 1,800 vs. 17,920 GPU hours.

The Effects of Thinking Mode Fusion and General RL. To evaluate Stages 3 and 4, we evaluate Qwen3-32B at Stage 2 (Reasoning RL), Stage 3 (Thinking Mode Fusion), and Stage 4 (General RL) using extra in-house benchmarks: CounterFactQA (anti-hallucination on counterfactual queries), LengthCtrl (length-controlled creative writing), ThinkFollow (multi-turn dialogs with random /think and /no think flag switches to test mode-switch correctness), and ToolUse (single-/multi-turn tool calling stability, scoring intent recognition, format and parameter accuracy).

Conclusion. Qwen3 features both thinking and non-thinking modes, letting users dynamically manage the number of tokens used for complex thinking. It is pre-trained on 36 trillion tokens, understands and generates 119 languages and dialects, and achieves strong performance across standard pre- and post-training benchmarks for code, math, reasoning and agents. In the near future, we plan to scale up pretraining with higher-quality and more diverse data; improve architecture and training for efficient compression and extremely long contexts; and invest more compute in RL—especially agent-based RL systems that learn from environmental feedback—to build agents capable of complex tasks requiring inference-time scaling.

A.1.1 Long-Context Ability (RULER). To enable length extrapolation we use YARN with a scaling factor = 4. In thinking mode we set the thinking budget to 8,192 tokens to mitigate overly verbose reasoning on extremely long inputs. Results: (1) in non-thinking mode, Qwen3 outperforms similarly sized Qwen2.5 models on long-context tasks; (2) in thinking mode, performance slightly degrades, because thinking content does not provide significant benefits for retrieval tasks and may instead interfere with retrieval. We plan to enhance long-context thinking-mode capability in future versions.

A.1.2 Multilingual Ability. Tables 24-35 detail benchmark scores across many languages including Spanish, French, Portuguese, Italian, Arabic, Japanese, Korean, Indonesian and more. Multilingual benchmarks span Multi-IF (8 langs), INCLUDE (44 langs), MMMLU (14 langs), MT-AIME2024 (55 langs), PolyMath (18 langs), and MLogiQA (10 langs). Combined with 119-language pre-training and a high translation-task proportion during Thinking Mode Fusion, this yields broad cross-lingual capability—especially on low-resource languages where translation-mix during SFT directly improves downstream scores.