2025-01-06

In this report, we introduce Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. Compared to previous iterations, Qwen 2.5 has been significantly improved during both the pre-training and post-training stages. In terms of pre-training, we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens. This provides a strong foundation for common sense, expert knowledge, and reasoning capabilities. In terms of post-training, we implement intricate supervised finetuning with over 1 million samples, as well as multistage reinforcement learning, including offline learning DPO and online learning GRPO. Post-training techniques significantly enhance human preference, and notably improve long text generation, structural data analysis, and instruction following.

To handle diverse and varied use cases effectively, we present Qwen2.5 LLM series in rich configurations. The open-weight offerings include base models and instruction-tuned models in sizes of 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B parameters. Quantized versions of the instruction-tuned models are also provided. Over 100 models can be accessed from Hugging Face Hub, ModelScope, and Kaggle. In addition, for hosted solutions, the proprietary models currently include two mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both available from Alibaba Cloud Model Studio.

Qwen2.5 has demonstrated top-tier performance on a wide range of benchmarks evaluating language understanding, reasoning, mathematics, coding, human preference alignment, etc. Specifically, the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and proprietary models and demonstrates competitive performance to the state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5 times larger. Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness while performing competitively against GPT-4o-mini and GPT-4o respectively. Additionally, as the foundation, Qwen2.5 models have been instrumental in training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and multimodal models.

The sparks of artificial general intelligence (AGI) are increasingly visible through the fast development of large foundation models, notably large language models (LLMs). The continuous advancement in model and data scaling, combined with the paradigm of large-scale pre-training followed by high-quality supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), has enabled large language models (LLMs) to develop emergent capabilities in language understanding, generation, and reasoning. Building on this foundation, recent breakthroughs in inference time scaling, particularly demonstrated by o1, have enhanced LLMs' capacity for deep thinking through step-by-step reasoning and reflection. These developments have elevated the potential of language models, suggesting they may achieve significant breakthroughs in scientific exploration as they continue to demonstrate emergent capabilities indicative of more general artificial intelligence.

Besides the fast development of model capabilities, the recent two years have witnessed a burst of open (open-weight) large language models in the LLM community, for example, the Llama series, Mistral series, and our Qwen series. The open-weight models have democratized the access of large language models to common users and developers, enabling broader research participation, fostering innovation through community collaboration, and accelerating the development of AI applications across diverse domains.

Recently, we release the details of our latest version of the Qwen series, Qwen2.5. In terms of the open-weight part, we release pre-trained and instruction-tuned models of 7 sizes, including 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B, and we provide not only the original models in bfloat16 precision but also the quantized models in different precisions. Specifically, the flagship model Qwen2.5-72B-Instruct demonstrates competitive performance against the state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5 times larger. Additionally, we also release the proprietary models of Mixture-of-Experts (MoE), namely Qwen2.5-Turbo and Qwen2.5-Plus, which performs competitively against GPT-4o-mini and GPT-4o respectively.

In this technical report, we introduce Qwen2.5, the result of our continuous endeavor to create better LLMs. Below, we show the key features of the latest version of Qwen: Better in Size: Compared with Qwen2, in addition to 0.5B, 1.5B, 7B, and 72B models, Qwen2.5 brings back the 3B, 14B, and 32B models, which are more cost-effective for resource-limited scenarios and are under-represented in the current field of open foundation models. Qwen2.5-Turbo and Qwen2.5-Plus offer a great balance among accuracy, latency, and cost. Better in Data: The pre-training and post-training data have been improved significantly. The pre-training data increased from 7 trillion tokens to 18 trillion tokens, with focus on knowledge, coding, and mathematics. The pre-training is staged to allow transitions among different mixtures. The post-training data amounts to 1 million examples, across the stage of supervised finetuning (SFT), direct preference optimization (DPO), and group relative policy optimization (GRPO). Better in Use: Several key limitations of Qwen2 in use have been eliminated, including larger generation length (from 2K tokens to 8K tokens), better support for structured input and output (e.g., tables and JSON), and easier tool use. In addition, Qwen2.5-Turbo supports a context length of up to 1 million tokens.

Basically, the Qwen2.5 series include dense models for opensource, namely Qwen2.5-0.5B / 1.5B / 3B / 7B / 14B / 32B / 72B, and MoE models for API service, namely Qwen2.5-Turbo and Qwen2.5-Plus. For dense models, we maintain the Transformer-based decoder architecture as Qwen2. The architecture incorporates several key components: Grouped Query Attention (GQA) for efficient KV cache utilization, SwiGLU activation function for non-linear activation, Rotary Positional Embeddings (RoPE) for encoding position information, QKV bias in the attention mechanism and RMSNorm with pre-normalization to ensure stable training. Building upon the dense model architectures, we extend it to MoE model architectures. This is achieved by replacing standard feed-forward network (FFN) layers with specialized MoE layers, where each layer comprises multiple FFN experts and a routing mechanism that dispatches tokens to the top-K experts. Following the approaches demonstrated in Qwen1.5-MoE, we implement fine-grained expert segmentation and shared experts routing.

Table 1 reports the architecture and licenses of the open-weight Qwen2.5 dense models. The 0.5B model uses 24 layers, 14/2 Q/KV heads, tie-embedding enabled, 32K context / 8K generation length, under Apache 2.0; the 1.5B uses 28 layers, 12/2 heads, tie-embedding enabled, 32K/8K, Apache 2.0; the 3B uses 36 layers, 16/2 heads, tie-embedding enabled, 32K/8K, under Qwen Research; the 7B uses 28 layers, 28/4 heads, tie-embedding disabled, 128K/8K, Apache 2.0; the 14B uses 48 layers, 40/8 heads, tie-embedding disabled, 128K/8K, Apache 2.0; the 32B uses 64 layers, 40/8 heads, tie-embedding disabled, 128K/8K, Apache 2.0; the 72B uses 80 layers, 64/8 heads, tie-embedding disabled, 128K/8K, under the Qwen license.

For tokenization, we utilize Qwen's tokenizer, which implements byte-level byte-pair encoding (BBPE) with a vocabulary of 151,643 regular tokens. We have expanded the set of control tokens from 3 to 22 compared to previous Qwen versions, adding two new tokens for tool functionality and allocating the remainder for other model capabilities. This expansion establishes a unified vocabulary across all Qwen2.5 models, enhancing consistency and reducing potential compatibility issues.

Qwen2.5 demonstrates significant enhancements in pre-training data quality compared to its predecessor Qwen2. These improvements stem from several key aspects. (1) Better data filtering. High-quality pre-training data is crucial for model performance, making data quality assessment and filtering a critical component of our pipeline. We leverage Qwen2-Instruct models as data quality filters that perform comprehensive, multi-dimensional analysis to evaluate and score training samples. The filtering method represents a significant advancement over our previous approach used for Qwen2, as it benefits from Qwen2's expanded pre-training on a larger multilingual corpus. The enhanced capabilities enable more nuanced quality assessment, resulting in both improved retention of high-quality training data and more effective filtering of low-quality samples across multiple languages.

(2) Better math and code data. During the pre-training phase of Qwen2.5, we incorporate training data from Qwen2.5-Math and Qwen2.5-Coder. This data integration strategy proves highly effective, as these specialized datasets are instrumental in achieving state-of-the-art performance on mathematical and coding tasks. By leveraging these high-quality domain-specific datasets during pre-training, Qwen2.5 inherits strong capabilities in both mathematical reasoning and code generation. (3) Better synthetic data. To generate high-quality synthetic data, particularly in mathematics, code, and knowledge domains, we leverage both Qwen2-72B-Instruct and Qwen2-Math-72B-Instruct. The quality of this synthesized data is further enhanced through rigorous filtering using our proprietary general reward model and the specialized Qwen2-Math-RM-72B model.

(4) Better data mixture. To optimize the pre-training data distribution, we employ Qwen2-Instruct models to classify and balance content across different domains. Our analysis revealed that domains like e-commerce, social media, and entertainment are significantly overrepresented in web-scale data, often containing repetitive, template-based, or machine-generated content. Conversely, domains such as technology, science, and academic research, while containing higher-quality information, are traditionally underrepresented. Through strategic down-sampling of overrepresented domains and up-sampling of high-value domains, we ensure a more balanced and information-rich training dataset that better serves our model's learning objectives. Building on these techniques, we have developed a larger and higher-quality pre-training dataset, expanding from the 7 trillion tokens used in Qwen2 to 18 trillion tokens.

We develop scaling laws for hyper-parameter based on the pre-training data of Qwen2.5. While previous studies primarily used scaling laws to determine optimal model sizes given compute budgets, we leverage them to identify optimal hyperparameters across model architectures. Specifically, our scaling laws help determine key training parameters like batch size $B$ and learning rate $\mu$ for both dense models and MoE models of varying sizes. Through extensive experimentation, we systematically study the relationship between model architecture and optimal training hyper-parameters. Specifically, we analyze how the optimal learning rate $\mu_\text{opt}$ and batch size $B_\text{opt}$ vary with model size $N$ and pre-training data size $D$. Our experiments cover a comprehensive range of architectures, including dense models with 44M to 14B parameters and MoE models with 44M to 1B activated parameters, trained on datasets ranging from 0.8B to 600B tokens. Using these optimal hyper-parameter predictions, we then model the final loss as a function of model architecture and training data scale.

Additionally, we leverage scaling laws to predict and compare the performance of MoE models with varying parameter counts against their dense counterparts. This analysis guides our hyper-parameter configuration for MoE models, enabling us to achieve performance parity with specific dense model variants (such as Qwen2.5-72B and Qwen2.5-14B) through careful tuning of both activated and total parameters.

For optimal training efficiency, Qwen2.5 employs a two-phase pre-training approach: an initial phase with a 4,096-token context length, followed by an extension phase for longer sequences. Following the strategy used in Qwen2, we extend the context length from 4,096 to 32,768 tokens during the final pre-training stage for all model variants except Qwen2.5-Turbo. Concurrently, we increase the base frequency of RoPE from 10,000 to 1,000,000 using the ABF technique. The RoPE rotation angles per dimension can be written as $\theta_i = b^{-2i/d}$, where the base $b$ has been adjusted up from $10^4$ to $10^6$ — yielding lower frequencies and a longer effective receptive field.

For Qwen2.5-Turbo, we implement a progressive context length expansion strategy during training, advancing through four stages: 32,768 tokens, 65,536 tokens, 131,072 tokens, and ultimately 262,144 tokens, with a RoPE base frequency of 10,000,000. At each stage, we carefully curate the training data to include 40% sequences at the current maximum length and 60% shorter sequences. This progressive training methodology enables smooth adaptation to increasing context lengths while maintaining the model's ability to effectively process and generalize across sequences of varying lengths.

To enhance our models' ability to process longer sequences during inference, we implement two key strategies: YARN and Dual Chunk Attention (DCA). Through these innovations, we achieve a four-fold increase in sequence length capacity, enabling Qwen2.5-Turbo to handle up to 1 million tokens and other models to process up to 131,072 tokens. Notably, these approaches not only improve the modeling of long sequences by reducing perplexity but also maintain the models' strong performance on shorter sequences, ensuring consistent quality across varying input lengths. Intuitively, YARN performs per-frequency NTK-aware interpolation on RoPE — scaling low-frequency components while leaving high-frequency ones intact — and DCA partitions the extra-long sequence into chunks, re-mapping inter-chunk relative positions so that every position index falls within the training distribution.

Qwen 2.5 introduces two significant advancements in its post-training design compared to Qwen 2: (1) Expanded Supervised Fine-tuning Data Coverage: The supervised fine-tuning process leverages a massive dataset comprising millions of high-quality examples. This expansion specifically addresses key areas where the previous model showed limitations, such as long-sequence generation, mathematical problem-solving, coding, instruction-following, structured data understanding, logical reasoning, cross-lingual transfer, and robust system instruction. (2) Two-stage Reinforcement Learning: The reinforcement learning (RL) process in Qwen 2.5 is divided into two distinct stages: Offline RL and Online RL.

Offline RL: This stage focuses on developing capabilities that are challenging for the reward model to evaluate, such as reasoning, factuality, and instruction-following. Through meticulous construction and validation of training data, we ensure that the Offline RL signals are both learnable and reliable, enabling the model to acquire those complex skills effectively. Online RL: The Online RL phase leverages the reward model's ability to detect nuances in output quality, including truthfulness, helpfulness, conciseness, relevance, harmlessness and debiasing. It enables the model to generate responses that are precise, coherent, and well-structured while maintaining safety and readability. As a result, the model's outputs consistently meet human quality standards and expectations.

In this section, we detail the key enhancements made during the SFT phase of Qwen2.5, focusing on several critical areas. (1) Long-sequence Generation: Qwen2.5 is capable of generating high-quality content with an output context length of up to 8,192 tokens, a significant advancement over the typical post-training response length, which often remains under 2,000 tokens. To address this gap, we develop long-response datasets. We employ back-translation techniques to generate queries for long-text data from pre-training corpora, impose output length constraints, and use Qwen2 to filter out low-quality paired data. (2) Mathematics: We introduce the chain-of-thought data of Qwen2.5-Math, which encompasses a diverse range of query sources, including public datasets, K-12 problem collections, and synthetic problems. To ensure high-quality reasoning, we employ rejection sampling along with reward modeling and annotated answers for guidance, producing step-by-step reasoning process.

(3) Coding: To enhance coding capabilities, we incorporate the instruction tuning data of Qwen2.5-Coder. We use multiple language-specific agents into a collaborative framework, generating diverse and high-quality instruction pairs across nearly 40 programming languages. We expand our instruction dataset by synthesizing new examples from code-related Q&A websites and gathering algorithmic code snippets from GitHub. A comprehensive multilingual sandbox is used to perform static code checking and validate code snippets through automated unit testing, ensuring code quality and correctness. (4) Instruction-following: To ensure high-quality instruction-following data, we implement a rigorous code-based validation framework. In this approach, LLMs generate both instructions and corresponding verification code, along with comprehensive unit tests for cross-validation. Through execution feedback-based rejection sampling, we carefully curate the training data used for Supervised Fine-Tuning, thereby guaranteeing the model's faithful adherence to intended instructions.

(5) Structured Data Understanding: We develop a comprehensive structured understanding dataset that encompasses both traditional tasks, such as tabular question-answering, fact verification, error correction, and structural understanding, as well as complex tasks involving structured and semi-structured data. By incorporating reasoning chains into the model's responses, we significantly enhance its ability to infer information from structured data. (6) Logical Reasoning: To enhance the model's logical reasoning capabilities, we introduce a diverse set of 70,000 new queries spanning various domains. These queries encompass multiple-choice questions, true/false questions, and open-ended questions. The model is trained to approach problems systematically, employing a range of reasoning methods such as deductive reasoning, inductive generalization, analogical reasoning, causal reasoning, and statistical reasoning. Through iterative refinement, we systematically filter out data containing incorrect answers or flawed reasoning processes.

(7) Cross-Lingual Transfer: To facilitate the transfer of the model's general capabilities across languages, we employ a translation model to convert instructions from high-resource languages into various low-resource languages, thereby generating corresponding response candidates. To ensure the accuracy and consistency of these responses, we evaluate the semantic alignment between each multilingual response and its original counterpart. (8) Robust System Instruction: We construct hundreds of general system prompts to improve the diversity of system prompts in post-training, ensuring consistency between system prompts and conversations. (9) Response Filtering: To evaluate the quality of responses, we employ multiple automatic annotation methods, including a dedicated critic model and a multi-agent collaborative scoring system. Responses are subjected to rigorous assessment, and only those deem flawless by all scoring systems are retained. Ultimately, we construct a dataset of over 1 million SFT examples. The model is fine-tuned for two epochs with a sequence length of 32,768 tokens. The learning rate is gradually decreased from $7 \times 10^{-6}$ to $7 \times 10^{-7}$. To address overfitting, we apply a weight decay of 0.1, and gradient norms are clipped at a maximum value of 1.0.

Compared to Online Reinforcement Learning (RL), Offline RL enables the pre-preparation of training signals, which is particularly advantageous for tasks where standard answers exist but are challenging to evaluate using reward models. In this study, we focus on objective query domains such as mathematics, coding, instruction following, and logical reasoning, where obtaining accurate evaluations can be complex. In the previous phase, we extensively employ strategies like execution feedback and answer matching to ensure the quality of responses. For the current phase, we reuse that pipeline, employing the SFT model to resample responses for a new set of queries. Responses that pass our quality checks are used as positive examples, while those that fail are treated as negative examples for Direct Preference Optimization (DPO) training. To further enhance the reliability and accuracy of the training signals, we make use of both human and automated review processes. This dual approach ensures that the training data is not only learnable but also aligned with human expectations. Ultimately, we construct a dataset consisting of approximately 150,000 training pairs. The model is then trained for one epoch using the Online Merging Optimizer, with a learning rate of $7 \times 10^{-7}$.

The DPO loss is given by $\mathcal{L}_{\text{DPO}} = -\mathbb{E}\bigl[\log \sigma\bigl(\beta\log\tfrac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\tfrac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\bigr)\bigr]$.

To develop a robust reward model for online RL, we adhere to a set of carefully defined labeling criteria. The specific guidelines for data labeling are as follows. Truthfulness: Responses must be grounded in factual accuracy, faithfully reflecting the provided context and instructions. Helpfulness: The model's output should be genuinely useful, addressing the user's query effectively while providing content that is positive, engaging, educational, and relevant. Conciseness: Responses should be succinct and to the point, avoiding unnecessary verbosity. Relevance: All parts of the response should be directly related to the user's query, dialogue history, and the assistant's context. Harmlessness: The model must prioritize user safety by avoiding any content that could lead to illegal, immoral, or harmful behavior. Debiasing: The model should produce responses that are free from bias, including but not limited to gender, race, nationality, and politics.

The queries utilized to train the reward model are drawn from two distinct datasets: publicly available open-source data and a proprietary query set characterized by higher complexity. Responses are generated from checkpoints of the Qwen models, which have been fine-tuned using different methods — SFT, DPO, and RL — at various stages of training. To introduce diversity, those responses are sampled at different temperature settings. Preference pairs are created through both human and automated labeling processes, and the training data for DPO is also integrated into this dataset. We employ Group Relative Policy Optimization (GRPO) for the online RL stage. The query order during training is determined by the variance of response scores — higher-variance queries are prioritized for effective learning. We sample 8 responses per query, with a global batch size of 2048 and 2048 samples per episode.

To extend Qwen2.5-Turbo's context length, we introduce longer SFT examples during post-training. SFT uses a two-stage approach: in the first stage, the model is fine-tuned exclusively on short instructions (each with a length of up to 32,768 tokens). This stage employs identical data and training steps as those used for other Qwen2.5 models, ensuring strong performance on short tasks. In the second stage, the fine-tuning process combines both short instructions (up to 32,768 tokens) and long instructions (up to 262,144 tokens). This hybrid approach effectively enhances the model's instruction-following capabilities in long-context tasks while maintaining its performance on short tasks. During the RL stage, we use a strategy similar to other Qwen2.5 models, focusing solely on short instructions. This design choice is motivated by two main reasons: first, RL training is computationally expensive for long-context tasks; second, there is a scarcity of reward models that provide suitable reward signals for long-context tasks. Notably, we find that conducting RL on short instructions can still significantly enhance the model's alignment with human preferences in long-context tasks.

In this section, we present a thorough assessment of base language models and instruction-tuned models of the Qwen2.5 series. The evaluation employs a comprehensive suite that combines public benchmarks and skill-oriented in-house datasets. The evaluation suite is designed to be primarily automatic with minimal human interaction. To prevent test data leakage, we exclude potentially contaminated data using n-gram matching when constructing the pre-training and post-training datasets. Following the criteria used in Qwen2, a training sequence $s_t$ is removed from the training data if there exists a test sequence $s_e$ such that the length of the longest common subsequence (LCS) between tokenized $s_t$ and $s_e$ satisfies both $|\text{LCS}(s_t, s_e)| \geq 13$ and $|\text{LCS}(s_t, s_e)| \geq 0.6 \times \min(|s_t|, |s_e|)$.

We conduct comprehensive evaluations of the base language models of the Qwen2.5 series. The evaluation of base models primarily emphasizes their performance in natural language understanding, general question answering, coding, mathematics, scientific knowledge, reasoning, and multilingual capabilities. The evaluation datasets cover General Tasks (MMLU 5-shot, MMLU-Pro 5-shot, MMLU-redux 5-shot, BBH 3-shot, ARC-C 25-shot, TruthfulQA 0-shot, Winogrande 5-shot, HellaSwag 10-shot), Mathematics & Science Tasks (GPQA 5-shot, TheoremQA 5-shot, GSM8K 4-shot, MATH 4-shot), Coding Tasks (HumanEval, HumanEval+, MBPP, MBPP+, MultiPL-E across Python/C++/Java/PHP/TypeScript/C#/Bash/JavaScript), and Multilingual Tasks grouped into Exam, Understanding, Mathematics, and Translation categories.

Qwen2.5-72B & Qwen2.5-Plus. We compare the base models of Qwen2.5-72B and Qwen2.5-Plus to other leading open-weight base models: Llama3-70B, Llama3-405B, Mixtral-8x22B, and our previous 72B version, Qwen2-72B. The Qwen2.5-72B base model significantly outperforms its peers in the same category across a wide range of tasks. It achieves results comparable to Llama-3-405B while utilizing only one-fifth of the parameters. Furthermore, when compared to its predecessor, Qwen2-72B, the Qwen2.5-72B shows marked improvements in nearly all benchmark evaluations, particularly excelling in general tasks, mathematics, and coding challenges. With significantly lower training and inference costs, Qwen2.5-Plus achieves very competitive performance results compared to Qwen2.5-72B and Llama3-405B. Moreover, Qwen2.5-Plus achieves 64.0 on MMLU-Pro, which is 5.9 points higher than Qwen2.5-72B.

Qwen2.5-14B/32B & Qwen2.5-Turbo. The evaluation of the Qwen2.5-Turbo, Qwen2.5-14B and 32B models is compared against baselines of similar sizes, including Yi-1.5-34B, Gemma2-27B and Qwen1.5-32B. Qwen2.5-32B surpasses larger models in similar size categories. Qwen2.5-Turbo achieves competitive performance with substantially fewer activated parameters. On benchmarks like MATH and MBPP, Qwen2.5-32B leads its peers. Qwen2.5-7B. Compared with Mistral-7B, Llama3-8B, Gemma2-9B and Qwen2-7B, Qwen2.5-7B shows strong improvements. Notably, despite having only 6.5B non-embedding parameters (vs Gemma2-9B's 8.2B), Qwen2.5-7B delivers very competitive results on MMLU, MATH and HumanEval. Edge-side models (0.5B/1.5B/3B). Qwen2.5-0.5B outperforms Gemma2-2.6B on various math and coding tasks; 1.5B and 3B variants offer some of the strongest open-source options at the resource-constrained scale.

To comprehensively evaluate the quality of instruction-tuned models, we compile automatic and human evaluation to assess the capabilities and human preferences. We evaluate model performance on open and in-house benchmarks covering general knowledge and instruction-following, mathematics, reasoning, coding, alignment to human preferences, and multilingual capabilities. Qwen2.5-72B-Instruct demonstrates very competitive performance compared with leading open-weight and proprietary instruction-tuned models. Notably, on multiple critical benchmarks — including MMLU-redux, MATH, MBPP, MultiPL-E, LiveCodeBench, Arena-Hard, and MTBench — Qwen2.5-72B-Instruct surpasses Llama-3.1-405B-Instruct, despite using a significantly smaller parameter count. Qwen2.5-Plus outperforms Qwen2.5-72B-Instruct on 9 of 13 benchmarks at substantially lower cost, demonstrating the effectiveness of the MoE configuration tuned via scaling laws.

Qwen2.5-14B-Instruct delivers competitive results that rival GPT-4o-mini at a much smaller scale. Qwen2.5-Turbo outperforms Qwen2.5-14B-Instruct on 8 of 10 benchmarks despite significantly lower inference cost. Qwen2.5-7B-Instruct significantly outperforms Gemma2-9B-IT and Llama3.1-8B-Instruct across nearly all evaluated tasks (with IFEval being the only notable exception), achieving 75.5 on MATH and 84.8 on HumanEval. Qwen2.5-3B-Instruct surpasses Phi-3.5-mini-instruct and MiniCPM3-4B-Instruct on mathematics and coding tasks despite having fewer parameters, making it one of the strongest general-purpose choices for resource-constrained scenarios.

We evaluate Qwen2.5-RM-72B against Nemotron-4-340B-Reward, Llama-3.1-Nemotron-70B-Reward, Skywork-Reward-Gemma-2-27B and Athene-RM-70B on Reward Bench, RMB, PPE, and an internally collected Human-Preference-Chinese benchmark. Qwen2.5-RM-72B leads on PPE and Human-Preference-Chinese, and ranks second on RMB. Through extensive practical experience, we observe a critical phenomenon: over-optimization on a specific RM benchmark may trigger Goodhart's law, degrading performance elsewhere. More importantly, current RM evaluation benchmarks do not accurately predict the performance of the RL models trained under their guidance — a higher score on RM benchmarks does not necessarily correlate with superior performance of the resulting RL model. This finding suggests the need for more comprehensive evaluation methods for reward models.

To evaluate the long-context capabilities of Qwen2.5 models, we conduct experiments on three benchmarks: RULER, LV-Eval, and Longbench-Chat. With YARN and DCA enabled, Qwen2.5 models demonstrate strong performance in long-context processing. Among them, Qwen2.5-72B-Instruct shows the strongest performance across all context lengths, significantly outperforming existing open-weight long-context models as well as proprietary models like GPT-4o-mini and GPT-4. On RULER at 128K context length, Qwen2.5-72B-Instruct scores 67.0 without DCA+YARN; once DCA+YARN are enabled, it jumps to 88.4 — a +21.4 improvement. Similar 20+ point jumps are observed for the 7B, 14B and 32B variants, confirming that long-context capability requires more than just adjusting the RoPE base — inference-time interpolation algorithms remain essential.

Qwen2.5-Turbo achieves 100% accuracy on the 1M-token passkey retrieval task (Figure 2), demonstrating its capability to capture detailed information from extremely long contexts. To further improve long-context inference efficiency, we develop a sparse attention mechanism based on Minference. For 1M-token sequences, this approach reduces the attention computation by 12.5 times. Across different hardware configurations (H20 and A100), we observe 3.2× to 4.3× speedups in the Time to First Token (TTFT) compared to using full attention (Figure 3). This sparse attention mechanism enables practical deployment of 1M-token context in production while maintaining retrieval quality.

Qwen2.5 represents a significant advancement in large language models (LLMs), with enhanced pre-training on 18 trillion tokens and sophisticated post-training techniques, including supervised fine-tuning and multi-stage reinforcement learning. These improvements boost human preference alignment, long text generation, and structural data analysis, making Qwen2.5 highly effective for instruction-following tasks. Available in various configurations, Qwen2.5 offers both open-weight from 0.5B to 72B parameters and proprietary models including cost-effective MoE variants like Qwen2.5-Turbo and Qwen2.5-Plus. Empirical evaluations show that Qwen2.5-72B-Instruct matches the performance of the state-of-the-art Llama-3-405B-Instruct, despite being six times smaller. Qwen2.5 also serves as a foundation for specialized models, demonstrating its versatility for domain-specific applications.

We believe that Qwen2.5's robust performance, flexible architecture, and broad availability make it a valuable resource for both academic research and industrial applications, positioning it as a key player of future innovations. In the future, we will focus on advancing robust foundational models. First, we will iteratively refine both base and instruction-tuned large language models (LLMs) by incorporating broader, more diverse, higher-quality data. Second, we will also continue to develop multimodal models. Our goal is to integrate various modalities into a unified framework. This will facilitate seamless, end-to-end information processing across textual, visual, and auditory domains. Third, we are committed to enhancing the reasoning capabilities of our models. This will be achieved through strategic scaling of inference compute resources.