DeepSeek-V3 Technical Report

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks.

Figure 1 (benchmark snapshot). MMLU-Pro 75.9 (EM); GPQA-Diamond 59.1 (Pass@1); MATH-500 90.2 (EM); AIME 2024 39.2 (Pass@1); Codeforces 51.6 (Percentile); SWE-bench Verified 42.0 (Resolved). Compared peers: DeepSeek-V2.5 (66.2 / 41.3 / 74.7 / 16.7 / 35.6 / 22.6), Qwen2.5-72B-Inst (71.6 / 49.0 / 80.0 / 23.3 / 24.8 / 23.8), Llama-3.1-405B-Inst (73.3 / 51.1 / 73.8 / 23.3 / 25.3 / 24.5), GPT-4o-0513 (72.6 / 49.9 / 74.6 / 9.3 / 23.6 / 38.8), Claude-3.5-Sonnet-1022 (78.0 / 65.0 / 78.3 / 16.0 / 20.3 / 50.8). Model checkpoints: github.com/deepseek-ai/DeepSeek-V3.

In recent years, Large Language Models (LLMs) have been undergoing rapid iteration and evolution, progressively diminishing the gap towards Artificial General Intelligence (AGI). Beyond closed-source models, open-source models, including DeepSeek series, LLaMA series, Qwen series, and Mistral series, are also making significant strides, endeavoring to close the gap with their closed-source counterparts. To further push the boundaries of open-source model capabilities, we scale up our models and introduce DeepSeek-V3, a large Mixture-of-Experts (MoE) model with 671B parameters, of which 37B are activated for each token.

With a forward-looking perspective, we consistently strive for strong model performance and economical costs. Therefore, in terms of architecture, DeepSeek-V3 still adopts MLA for efficient inference and DeepSeekMoE for cost-effective training. These two architectures have been validated in DeepSeek-V2, demonstrating their capability to maintain robust model performance while achieving efficient training and inference. Beyond the basic architecture, we implement two additional strategies to further enhance the model capabilities. Firstly, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing, with the aim of minimizing the adverse impact on model performance that arises from the effort to encourage load balancing. Secondly, DeepSeek-V3 employs a multi-token prediction training objective, which we have observed to enhance the overall performance on evaluation benchmarks.

In order to achieve efficient training, we support the FP8 mixed precision training and implement comprehensive optimizations for the training framework. In this work, we introduce an FP8 mixed precision training framework and, for the first time, validate its effectiveness on an extremely large-scale model. Through the support for FP8 computation and storage, we achieve both accelerated training and reduced GPU memory usage. As for the training framework, we design the DualPipe algorithm for efficient pipeline parallelism, which has fewer pipeline bubbles and hides most of the communication during training through computation-communication overlap. We also develop efficient cross-node all-to-all communication kernels to fully utilize InfiniBand (IB) and NVLink bandwidths, and meticulously optimize the memory footprint, making it possible to train DeepSeek-V3 without using costly tensor parallelism.

During pre-training, we train DeepSeek-V3 on 14.8T high-quality and diverse tokens. The pre-training process is remarkably stable: throughout the entire training process, we did not encounter any irrecoverable loss spikes or have to roll back. Next, we conduct a two-stage context length extension for DeepSeek-V3 — first to 32K, then to 128K. Following this, we conduct post-training, including Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on the base model of DeepSeek-V3, to align it with human preferences and further unlock its potential. During the post-training stage, we distill the reasoning capability from the DeepSeek-R1 series of models, and meanwhile carefully maintain the balance between model accuracy and generation length.

Table 1 (training costs, H800 @ $2/GPU·hr). Pre-Training: 2664K GPU·hours = $5.328M; Context Extension: 119K = $0.238M; Post-Training: 5K = $0.01M; Total: 2788K = $5.576M. Training each trillion tokens requires only 180K H800 GPU·hours — i.e., 3.7 days on our 2048-GPU cluster.

Our main contributions. Architecture: an innovative auxiliary-loss-free load-balancing strategy and a Multi-Token Prediction (MTP) training objective, which is also reusable for speculative decoding. Pre-training: an FP8 mixed-precision training framework — the first time FP8 has been validated at this scale — together with co-designed algorithms / frameworks / hardware that overcome the cross-node MoE communication bottleneck and achieve near-full computation-communication overlap, completing DeepSeek-V3 pre-training on 14.8T tokens at only 2.664M H800 GPU·hours. Post-training: knowledge distillation of reasoning capability from a long-CoT R1 model into a standard LLM, elegantly incorporating R1's verification and reflection patterns into DeepSeek-V3 while controlling output style and length.

Summary of evaluations. Knowledge: MMLU 88.5 / MMLU-Pro 75.9 / GPQA 59.1 — best among open-source, competitive with GPT-4o and Claude-3.5-Sonnet; surpasses both on Chinese SimpleQA. Math: state-of-the-art on math benchmarks among non-long-CoT models — beats o1-preview on MATH-500. Code: top on competition coding (LiveCodeBench); slightly behind Claude-3.5-Sonnet on engineering tasks (SWE-Bench) but well ahead of every other open model.

The basic architecture of DeepSeek-V3 is still within the Transformer framework. For efficient inference and economical training, DeepSeek-V3 also adopts MLA and DeepSeekMoE, which have been thoroughly validated by DeepSeek-V2. Compared with DeepSeek-V2, an exception is that we additionally introduce an auxiliary-loss-free load balancing strategy for DeepSeekMoE to mitigate the performance degradation induced by the effort to ensure load balance. Figure 2 illustrates the basic architecture of DeepSeek-V3, and we will briefly review the details of MLA and DeepSeekMoE in this section.

For attention, DeepSeek-V3 adopts the MLA architecture. Let $d$ denote the embedding dimension, $n_h$ the number of attention heads, $d_h$ the dimension per head, and $\mathbf{h}_t \in \mathbb{R}^d$ the attention input for the $t$-th token. The core of MLA is the low-rank joint compression for attention keys and values to reduce the KV cache during inference:

where $\mathbf{c}_t^{KV} \in \mathbb{R}^{d_c}$ is the compressed KV latent ($d_c \ll d_h n_h$); $W^{DKV}$ is the down-projection; $W^{UK}, W^{UV}$ are the up-projections for keys/values; and $W^{KR}$ produces a decoupled key that carries the Rotary Positional Embedding (RoPE). During generation, only the blue-boxed vectors $\mathbf{c}_t^{KV}$ and $\mathbf{k}_t^R$ need to be cached, which yields a significantly reduced KV cache while keeping performance comparable to standard Multi-Head Attention (MHA).

For queries we also do low-rank compression to reduce activation memory during training: $\mathbf{c}_t^Q = W^{DQ}\,\mathbf{h}_t$, $\mathbf{q}_t^C = W^{UQ}\,\mathbf{c}_t^Q$, $\mathbf{q}_t^R = \mathrm{RoPE}(W^{QR}\,\mathbf{c}_t^Q)$, $\mathbf{q}_{t,i} = [\mathbf{q}_{t,i}^C;\,\mathbf{q}_{t,i}^R]$. Finally:

For Feed-Forward Networks (FFNs), DeepSeek-V3 employs the DeepSeekMoE architecture. Compared with traditional MoE architectures like GShard, DeepSeekMoE uses finer-grained experts and isolates some experts as shared ones. Let $\mathbf{u}_t$ denote the FFN input of the $t$-th token; the FFN output $\mathbf{h}'_t$ is:

Here $N_s$ and $N_r$ are the numbers of shared / routed experts; $K_r$ is the number of activated routed experts; $\mathbf{e}_i$ is the centroid of the $i$-th routed expert. Different from DeepSeek-V2, V3 uses the sigmoid function to compute affinities and applies normalization among selected affinities to produce the gating values.

Auxiliary-loss-free load balancing. Unbalanced expert load causes routing collapse and degrades efficiency under expert parallelism. Conventional fixes rely on auxiliary loss, but too large an auxiliary loss hurts model quality. To get a better trade-off, we introduce, for each expert, a bias term $b_i$ added to the affinity only for top-K selection:

The bias is used only for routing — the gating value multiplied with the FFN output is still derived from the original $s_{i,t}$. We monitor the whole-batch expert load each step; at step end we decrease $b_i$ by $\gamma$ if expert $i$ is overloaded and increase it by $\gamma$ otherwise, where $\gamma$ is the bias-update speed. To prevent extreme imbalance within any single sequence we additionally keep a tiny sequence-wise balance loss $\mathcal{L}_\mathrm{Bal} = \alpha \sum_i f_i P_i$ with an extremely small $\alpha$. Also: node-limited routing sends each token to at most $M$ nodes; thanks to balanced load, DeepSeek-V3 drops no tokens during training or inference.

Inspired by Gloeckle et al. (2024), we set a Multi-Token Prediction (MTP) objective for DeepSeek-V3 that extends the prediction scope to multiple future tokens at each position. On one hand, MTP densifies training signals and may improve data efficiency. On the other hand, MTP may enable the model to pre-plan its representations for better prediction of future tokens. Different from Gloeckle et al., who parallelly predict $D$ additional tokens using independent output heads, we sequentially predict additional tokens and keep the complete causal chain at each prediction depth.

MTP modules. Our implementation uses $D$ sequential modules. The $k$-th MTP module consists of a shared embedding $\mathrm{Emb}(\cdot)$, a shared output head $\mathrm{OutHead}(\cdot)$, a Transformer block $\mathrm{TRM}_k(\cdot)$, and a projection matrix $M_k \in \mathbb{R}^{d\times 2d}$. For the $i$-th input token at the $k$-th depth, we first combine the representation at depth $k{-}1$ with the embedding of token $t_{i+k}$ via a linear projection:

where $[\cdot;\cdot]$ is concatenation; for $k=1$, $\mathbf{h}^{k-1}_i$ refers to the main model. Then $\mathbf{h}^k_{1:T-k} = \mathrm{TRM}_k(\mathbf{h}'^{k}_{1:T-k})$ and $P^k_{i+k+1} = \mathrm{OutHead}(\mathbf{h}^k_i)$. The training loss at depth $k$ is a cross-entropy $\mathcal{L}^k_\mathrm{MTP}$ on tokens $t_{2+k:T+1}$, and the total MTP loss is $\mathcal{L}_\mathrm{MTP} = \frac{\lambda}{D}\sum_{k=1}^D \mathcal{L}^k_\mathrm{MTP}$. At inference, MTP modules can be directly discarded — the main model functions normally; alternatively they can be repurposed for speculative decoding to reduce latency.

DeepSeek-V3 is trained on a cluster equipped with 2048 NVIDIA H800 GPUs. Each node has 8 GPUs connected by NVLink and NVSwitch within the node; across nodes, InfiniBand (IB) interconnects are used. DeepSeek-V3 applies 16-way Pipeline Parallelism (PP), 64-way Expert Parallelism (EP) spanning 8 nodes, and ZeRO-1 Data Parallelism (DP). We avoid Tensor Parallelism (TP) entirely through careful memory engineering.

DualPipe. Cross-node expert parallelism makes the computation-to-communication ratio about 1:1. We design DualPipe — a pipeline parallelism algorithm that not only accelerates training by overlapping forward and backward computation-communication phases, but also reduces pipeline bubbles. We split each chunk into four components — attention, all-to-all dispatch, MLP, all-to-all combine — and further split backward attention/MLP into "backward for input" vs "backward for weights" (ZeroBubble-style). DualPipe uses a bidirectional schedule that feeds micro-batches from both ends, ensuring both all-to-all and PP communication are fully hidden.

Cross-node all-to-all kernels. NVLink bandwidth is 160 GB/s (~3.2× IB at 50 GB/s). We limit each token to at most 4 nodes (cutting IB traffic). When routing is decided, a token first travels via IB to the GPU with the same in-node index on the target node, then is immediately forwarded via NVLink to the GPU hosting the target expert. IB and NVLink communications are fully overlapped; on average each token can pick 3.2 experts/node — even though only 8 routed experts are activated, this could scale to 13 (4 nodes × 3.2 experts) with the same communication cost. Warp specialization is used with just ~20 SMs reserved for communication.

Inspired by recent advances in low-precision training, we propose a fine-grained mixed-precision framework using the FP8 data format. Although progress has been made in inference quantization, there are relatively few studies on low-precision pre-training at scale. To address outliers in activations / weights / gradients and effectively extend the dynamic range of FP8, we use a fine-grained quantization strategy: tile-wise grouping of $1\times N_c$ elements (activations) or block-wise grouping of $N_c\times N_c$ elements (weights). The associated dequantization overhead is largely mitigated under an increased-precision accumulation process — a critical aspect for accurate FP8 GEMM. To further reduce memory and communication in MoE training, we cache and dispatch activations in FP8 while storing low-precision optimizer states in BF16. Validated on two model scales similar to DeepSeek-V2-Lite and DeepSeek-V2 over ~1T tokens, our FP8 model's relative loss error vs the BF16 baseline stays consistently below 0.25% — well within training-randomness tolerance.

Mixed precision framework (Figure 6). Most compute-density operations — the three GEMMs of every Linear (Fprop, Dgrad, Wgrad) — run in FP8 with BF16/FP32 outputs, theoretically doubling speed vs BF16. The FP8 Wgrad GEMM also lets us store activations in FP8 for backward, slashing memory. Operators sensitive to low precision (embedding, output head, MoE gating, normalizations, attention) keep BF16/FP32; master weights and optimizer states are kept high precision.

Improved precision from quantization and multiplication. H800 Tensor Cores accumulate FP8 with only ~14-bit mantissa precision. We solve this by promotion to CUDA Core: every $N_C=128$ inner-product elements, partial results are moved from the Tensor Core to the CUDA Core for FP32 accumulation, with two WGMMA stages interleaved to hide overhead. Combined with tile/block-wise scaling, this lets FP8 GEMM match BF16-grade accuracy.

Low-precision storage and communication; inference and deployment. Activation caching, all-to-all dispatch, and gradient accumulation use low precision where safe; optimizer states use BF16. For inference, prefilling and decoding are deployed separately, with redundant expert deployment to absorb domain-shift load imbalance during inference. Finally, we list hardware design suggestions: communication hardware should support fine-grained scaling factors and FP8/BF16 accumulation at full speed; compute hardware should expose high-precision accumulation as a first-class capability rather than a workaround.

Compared with DeepSeek-V2, we optimize the pre-training corpus by enhancing the ratio of mathematical and programming samples, while expanding multilingual coverage beyond English and Chinese. Our data processing pipeline is refined to minimize redundancy while maintaining diversity. We use the document-packing method for data integrity but do not apply cross-sample attention masking during training. The final corpus contains 14.8T high-quality and diverse tokens under our tokenizer.

We adopt the Fill-in-Middle (FIM) strategy following DeepSeekCoder-V2, using the Prefix-Suffix-Middle (PSM) framework: <|fim_begin|> f_pre <|fim_hole|> f_suf <|fim_end|> f_middle <|eos_token|>. FIM is applied document-level during pre-packing at rate 0.1. The tokenizer uses Byte-level BPE with a 128K vocabulary, and its pretokenizer adds tokens that combine punctuations and line breaks; to mitigate the "token-boundary bias" this could cause on few-shot prompts without trailing newlines, we randomly split a fraction of such combined tokens during training.

Model hyper-parameters. Transformer layers $L=61$, hidden dim $d=7168$, init std $0.006$. MLA: $n_h=128$, per-head $d_h=128$, KV-compression $d_c=512$, query-compression $d'_c=1536$, decoupled-RoPE per-head $d^R_h=64$. All FFNs except the first 3 layers are MoE: each MoE has 1 shared expert + 256 routed experts, each expert's intermediate dim is 2048; 8 routed experts activated per token; each token is sent to at most 4 nodes. MTP depth $D=1$ (i.e., predict 1 extra token). Extra RMSNorms after compressed latents and extra scaling factors at width bottlenecks are also used (as in DeepSeek-V2). Total parameters: 671B; activated per token: 37B.

Training hyper-parameters. AdamW: $\beta_1=0.9$, $\beta_2=0.95$, weight_decay $=0.1$. Max sequence length 4K. LR schedule: linear warmup from 0 to $2.2\times10^{-4}$ in 2K steps; hold $2.2\times10^{-4}$ until 10T tokens; cosine-decay to $2.2\times10^{-5}$ over 4.3T tokens; final 500B tokens at $2.2\times10^{-5}$ for the first 333B and $7.3\times10^{-6}$ for the remaining 167B. Grad-clip 1.0. Batch size: ramp from 3072 to 15360 in the first 469B tokens, then constant 15360. Node-limited routing $M=4$. Auxiliary-loss-free bias-update speed $\gamma=0.001$ for the first 14.3T, then $0$. Balance-loss $\alpha=0.0001$. MTP loss weight $\lambda=0.3$ (first 10T) then $0.1$ (remaining 4.8T).

We adopt a DeepSeek-V2-style approach: after pre-training, apply YaRN for context extension in two phases of 1000 steps each, expanding the context window from 4K → 32K → 128K. YaRN is applied exclusively to the decoupled shared key $\mathbf{k}^R_t$. Hyper-parameters: scale $s=40$, $\alpha=1$, $\beta=32$, scaling factor $\sqrt{t} = 0.1\ln s + 1$. Phase 1: seq-len 32K, batch 1920. Phase 2: seq-len 128K, batch 480. Learning rate $7.3\times10^{-6}$ for both. After this two-phase extension, DeepSeek-V3 handles inputs up to 128K while maintaining strong performance; the Needle-In-A-Haystack (NIAH) test (after SFT) shows consistent robustness across all context lengths up to 128K.

We compare DeepSeek-V3-Base with the state-of-the-art open-source base models — DeepSeek-V2-Base, Qwen2.5 72B Base, and LLaMA-3.1 405B Base — using our internal evaluation framework with the same evaluation setting. DeepSeek-V3-Base achieves the best performance on most benchmarks, especially math and code.

Table 3 highlights (DeepSeek-V3 vs Qwen2.5-72B vs LLaMA-3.1-405B): BBH 3-shot 87.5 / 79.8 / 82.9; MMLU 5-shot 87.1 / 85.0 / 84.4; MMLU-Pro 64.4 / 58.3 / 52.8; DROP F1 89.0 / 80.6 / 86.0; HumanEval Pass@1 65.2 / 53.0 / 54.9; MBPP Pass@1 75.4 / 72.6 / 68.4; LiveCodeBench-Base Pass@1 19.4 / 12.9 / 15.5; GSM8K 8-shot 89.3 / 88.3 / 83.5; MATH 4-shot 61.6 / 54.4 / 49.0; CMath 90.7 / 84.5 / 77.3; C-Eval 90.1 / 89.2 / 72.5; MMMLU non-English 79.4 / 74.8 / 73.8. Total params: 671B (MoE, 37B active) vs 72B dense vs 405B dense. Under our efficient architectures and engineering optimizations, training each trillion tokens requires only 180K H800 GPU·hours — much cheaper than 72B or 405B dense.

MTP ablation (Table 4). We validate MTP on two scales: a 15.7B-total small MoE (2.4B active) trained on 1.33T tokens, and a 228.7B-total large MoE (20.9B active) trained on 540B tokens. The MTP module is appended at depth 1 and discarded at inference, so inference cost is identical. MTP consistently improves most benchmarks: e.g., small MoE — MMLU 50.0 → 53.3, HumanEval 20.7 → 26.8, GSM8K 25.4 → 31.4, MATH 10.7 → 12.6; large MoE — HumanEval 44.5 → 53.7, GSM8K 72.3 → 74.0, MATH 38.6 → 39.8.

Auxiliary-Loss-Free ablation (Table 5). Same two scales; baseline uses pure auxiliary loss + sigmoid + top-K affinity normalization with the same aux-loss strength as DeepSeek-V2-Lite / V2. Removing all aux losses and adopting the loss-free balancing yields better results on most benchmarks: e.g., small MoE — BBH 37.3 → 39.3, GSM8K 27.1 → 29.6; large MoE — BBH 66.7 → 67.9, HumanEval 40.2 → 46.3, GSM8K 70.7 → 74.5, MATH 37.2 → 39.6.

Batch-wise vs sequence-wise balancing. Batch-wise balancing imposes a more flexible constraint and allows experts to better specialize across domains. On 1B/3B MoEs the validation losses are nearly indistinguishable between aux-loss-free and a batch-wise auxiliary loss (1B: 2.253 vs 2.253; 3B: 2.080 vs 2.080) — both beat sequence-wise (1B 2.258, 3B 2.085). The aux-loss-free model shows much greater expert specialization on Pile-test domains (Figure 9).

We curate 1.5M instances spanning multiple domains, with each domain using a tailored data-creation method.

Reasoning data. For mathematics, code-competition, and logic puzzles, we generate data by leveraging an internal DeepSeek-R1 model. R1-generated data has strong accuracy but suffers from overthinking, poor formatting, and excessive length, so we balance high accuracy with clarity and conciseness. To this end, we develop domain-specific expert models via a combined SFT + RL pipeline; these expert models serve as data generators. For each instance we produce two types of SFT samples: <problem, original response> and <system prompt, problem, R1 response>. The system prompt is designed to instruct the model to produce responses enriched with reflection/verification. During RL, high-temperature sampling lets the model blend R1-generated and original styles even without explicit system prompts; after hundreds of RL steps, the intermediate RL model learns to incorporate R1 patterns. Finally, rejection sampling against the expert models curates the high-quality SFT data for the final model — keeping R1's strengths while producing concise responses.

Non-reasoning data. For creative writing, role-play, simple QA, we use DeepSeek-V2.5 to generate responses and enlist human annotators to verify correctness. SFT settings. Two epochs over the SFT dataset; cosine-decay LR from $5\times10^{-6}$ to $1\times10^{-6}$; sequence packing with sample masking so examples remain mutually invisible.

Reward Model. We employ both a rule-based RM and a model-based RM. Rule-based: for questions that can be validated by rules (math with a boxed final answer; LeetCode with a compiler over test cases), rule-based reward is resistant to manipulation and highly reliable. Model-based: for free-form answers, the RM judges whether responses match the ground truth; for open-ended creative tasks, the RM gives feedback from question + answer. The RM is trained from DeepSeek-V3 SFT checkpoints; the preference data not only provides the final reward but also the chain-of-thought leading to it, mitigating reward hacking.

Group Relative Policy Optimization (GRPO). Following DeepSeek-V2, we adopt GRPO, which drops the critic (typically of policy-size) and estimates the baseline from group scores. For each question $q$, GRPO samples a group $\{o_1, \ldots, o_G\}$ from $\pi_{\theta_\mathrm{old}}$ and maximizes:

where the KL term uses the unbiased $k_3$ estimator, and the group-normalized advantage is:

RL prompts span coding, math, writing, role-play, QA — this both aligns the model with human preferences and improves benchmark performance, especially where SFT data is scarce.

Standard benchmarks (Table 6, DeepSeek-V3 vs Claude-3.5-Sonnet-1022 vs GPT-4o-0513 vs LLaMA-3.1-405B-Inst vs Qwen2.5-72B-Inst). MMLU 88.5 / 88.3 / 87.2 / 88.6 / 85.3; MMLU-Pro 75.9 / 78.0 / 72.6 / 73.3 / 71.6; DROP F1 91.6 / 88.3 / 83.7 / 88.7 / 76.7; GPQA-Diamond 59.1 / 65.0 / 49.9 / 51.1 / 49.0; LongBench v2 48.7 / 41.0 / 48.1 / 36.1 / 39.4; HumanEval-Mul 82.6 / 81.7 / 80.5 / 77.2 / 77.3; LiveCodeBench Pass@1-COT 40.5 / 36.3 / 33.4 / 28.4 / 31.1; Codeforces percentile 51.6 / 20.3 / 23.6 / 25.3 / 24.8; SWE Verified Resolved 42.0 / 50.8 / 38.8 / 24.5 / 23.8; AIME 2024 Pass@1 39.2 / 16.0 / 9.3 / 23.3 / 23.3; MATH-500 EM 90.2 / 78.3 / 74.6 / 73.8 / 80.0; CNMO 2024 43.2 / 13.1 / 10.8 / 6.8 / 15.9; C-Eval 86.5 / 76.7 / 76.0 / 61.5 / 86.1; C-SimpleQA 64.8 / 51.3 / 59.3 / 50.4 / 48.4. DeepSeek-V3 is the strongest open-source model and is competitive against frontier closed-source models.

Open-ended (Table 7, GPT-4-Turbo-1106 as judge). Arena-Hard 85.5 (vs Claude-3.5-Sonnet 85.2, GPT-4o 80.4, Qwen2.5-72B 81.2, LLaMA-3.1 405B 69.3, V2.5-0905 76.2) — DeepSeek-V3 is the first open-source model to surpass 85%. AlpacaEval 2.0 (length-controlled win rate) 70.0 (vs Claude-3.5-Sonnet 52.0, GPT-4o 51.1, V2.5-0905 50.5) — a 20-point jump over V2.5-0905.

DeepSeek-V3 as judge (Table 8, RewardBench). DeepSeek-V3 hits Average 87.0, on par with the best GPT-4o-0806 (86.7) and Claude-3.5-Sonnet-1022 (88.7). With majority voting (maj@6), the Average lifts to 89.6, exceeding all baselines. This validates DeepSeek-V3-as-judge for the constitutional-AI / self-rewarding loop.

Distillation from DeepSeek-R1 (Table 9). Based on DeepSeek-V2.5, we ablate the contribution of R1 distillation: short-CoT baseline vs distilled-from-R1. LiveCodeBench-CoT Pass@1 31.1 → 37.4; MATH-500 Pass@1 74.6 → 83.2. The trade-off: distillation improves performance but substantially increases average response length (MATH-500 length 769 → 1510). For DeepSeek-V3 we carefully selected settings to balance accuracy vs efficiency. Knowledge distillation from long-CoT reasoning models is a promising direction for post-training — its effectiveness in math/coding suggests broader applicability to other cognitive tasks requiring complex reasoning.

Self-rewarding. In domains without easy external verifiers, we apply the constitutional-AI approach using DeepSeek-V3's own voting evaluations as feedback. This noticeably improves alignment, especially in subjective evaluations. We see LLM-as-feedback-source as a paradigm with broad future potential.

MTP at inference. DeepSeek-V3 predicts 2 tokens through MTP; combined with speculative decoding it accelerates generation. The 2nd-token acceptance rate is 85–90% across generation topics, yielding ~1.8× TPS (tokens-per-second) speedup with no quality loss.

We introduce DeepSeek-V3, a large MoE language model with 671B total parameters and 37B activated parameters, trained on 14.8T tokens. In addition to MLA and DeepSeekMoE, it pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. Training is cost-effective thanks to FP8 training and meticulous engineering optimizations; post-training successfully distills reasoning capability from the DeepSeek-R1 series. Comprehensive evaluations show DeepSeek-V3 is the strongest open-source model currently available, achieving performance comparable to leading closed-source models like GPT-4o and Claude-3.5-Sonnet — at only 2.788M H800 GPU·hours total.

Limitations. (1) The recommended deployment unit is relatively large, which may burden small teams. (2) Although our deployment achieves >2× the end-to-end generation speed of DeepSeek-V2, further gains are still possible. We expect these limitations to be naturally addressed with more advanced hardware.

Future directions. ① Continuously refine architectures to improve training/inference efficiency and approach efficient infinite-context support; explore breaking Transformer's modeling limits. ② Continuously iterate training data quantity and quality, exploring additional training signal sources to drive data scaling along more dimensions. ③ Explore deep-thinking capabilities by expanding reasoning length and depth. ④ Develop more comprehensive, multi-dimensional model evaluation methods to prevent over-optimization on fixed benchmarks.

B.1 FP8 vs BF16 training. We validate our FP8 mixed-precision recipe on two model scales similar to DeepSeek-V2-Lite and DeepSeek-V2. The smaller scale: ~16B-total MoE trained on ~1.33T tokens (Figure 10 upper). The larger scale: ~230B-total MoE trained on ~0.9T tokens (Figure 10 lower). In both, the FP8 loss curve almost completely overlaps the BF16 baseline; the relative loss error stays well below the 0.25% acceptable threshold (zoomed-in plots show ±0.001 ~ ±0.0005 distributed roughly symmetrically around 0). The 230B-scale relative error is in fact tighter than the 16B-scale — large models do not amplify FP8 noise.

B.2 Block-wise quantization. Tile-wise scaling for activations and block-wise scaling for weights are the workhorses that defuse outliers. Pure per-tensor scaling would otherwise let outliers dominate the FP8 range and silently degrade quality. The "promotion-to-CUDA-Core" trick (FP32 accumulation every $N_C=128$ inner-product elements) is what ultimately lets FP8 GEMM match BF16 quality on H800 — H800's native Tensor-Core FP8 accumulation precision is only ~14 bits, insufficient on its own.