DeepSeek-V3

TL;DR#

• What was built: DeepSeek-V3, a Mixture-of-Experts (MoE) Transformer with 671B total parameters, of which 37B are used for each token. It has 61 layers, 1 shared and 256 routed experts per MoE layer, and 8 routed experts active per token (§4.2). It was pre-trained on 14.8T tokens, extended to 128K context, then fine-tuned with SFT and RL.
• Architecture: it keeps DeepSeek-V2's Multi-head Latent Attention (MLA) and DeepSeekMoE, and adds two things. Auxiliary-loss-free load balancing steers routing with a per-expert bias instead of a loss term. Multi-token prediction (MTP) trains the model to also predict the token after next.
• Systems: the first validation of FP8 mixed-precision training at this scale (fine-grained 1×128 / 128×128 scaling plus higher-precision accumulation). DualPipe is a bidirectional pipeline schedule that hides MoE all-to-all communication behind computation. Custom all-to-all kernels need only 20 of the H800's 132 SMs.
• Cost: 2.788M H800 GPU hours in total, about $5.576M at an assumed $2/GPU-hour (Table 1). That covers only the final run, not research or ablations. Each trillion tokens took 180K GPU hours, or 3.7 days on 2,048 GPUs.
• Results: the strongest open base model at release (e.g. MMLU 87.1, MATH 61.6, HumanEval 65.2, Table 3). The chat model scores MMLU 88.5, MATH-500 90.2, AIME 2024 39.2 and Codeforces 51.6 percentile, competitive with GPT-4o-0513 and Claude-3.5-Sonnet-1022 (Table 6).
• Stability: no irrecoverable loss spikes and no rollbacks over the whole run (Abstract, §1).

Why it matters#

By late 2024 the open-model frontier was set by dense models such as LLaMA-3.1 405B and Qwen2.5 72B. Closed models such as GPT-4o and Claude-3.5-Sonnet were still clearly ahead. Training a frontier-class model was assumed to need tens of thousands of GPUs. DeepSeek-V3 reached that level on a 2,048-GPU H800 cluster, a chip whose interconnect is cut down for export. It is mostly a co-design story: almost every model choice is tuned to make the hardware efficient, and almost every systems choice is tuned for this model.

The paper continues a year of work. DeepSeekMoE introduced fine-grained and shared experts. DeepSeek-V2 added MLA and showed that the combination is cheap to train and serve. DeepSeekMath introduced GRPO, the RL algorithm used again here. V3 scales the recipe roughly 3× in total parameters and replaces the parts that hurt quality or throughput at that size. The auxiliary balance loss is the clearest case: it is swapped for a routing bias.

Key idea At this scale, MoE training is limited by communication, not arithmetic. With cross-node expert parallelism, compute and communication time are roughly 1:1. Almost every V3 innovation either hides that communication (DualPipe, node-limited routing, custom kernels), shrinks it (FP8 dispatch), or removes something that fights it (the auxiliary loss, tensor parallelism).

What came next: the reasoning model DeepSeek-R1 (arXiv 2501.12948) was trained on top of DeepSeek-V3-Base. The infrastructure became public in February 2025 as open-source repos (DualPipe, DeepEP, DeepGEMM). A follow-up ISCA 2025 paper discusses the hardware lessons (arXiv 2505.09343). The cluster-level groundwork is in Fire-Flyer AI-HPC.

Background you need#

• Mixture-of-Experts (MoE): the dense feed-forward block is replaced by many small "expert" FFNs. A router sends each token to only a few of them, so total parameters grow while compute per token stays small. See DeepSeekMoE for the fine-grained and shared-expert design.
• Load balancing and routing collapse: if the router favours a few experts, those experts are overloaded and the rest are undertrained. With experts spread across GPUs, imbalance also leaves GPUs idle. The usual fix is an auxiliary loss that penalizes imbalance (Switch Transformer, arXiv 2101.03961).
• KV cache and MLA: during generation, each past token's keys and values are stored for every layer. This cache limits batch size and context length. MLA, from DeepSeek-V2, stores one small latent vector per token and reconstructs keys and values from it.
• Parallelism vocabulary: pipeline parallelism (PP) puts different layers on different GPUs and streams micro-batches through them. Idle time at the start and end is the "bubble". Expert parallelism (EP) puts different experts on different GPUs and moves tokens with all-to-all "dispatch" and "combine" collectives. ZeRO-1 data parallelism shards optimizer state across data-parallel replicas.
• FP8 formats: 8-bit floats. E4M3 has 4 exponent bits and 3 mantissa bits (more precision, less range). E5M2 has more range and less precision. Hopper Tensor Cores run FP8 matrix multiplies at roughly twice the BF16 rate. The narrow range makes outliers dangerous.
• GRPO: Group Relative Policy Optimization, from DeepSeekMath. It is PPO without a critic: sample a group of answers per prompt and use each answer's reward minus the group mean, divided by the group standard deviation, as its advantage.

The core ideas#

1. Architecture recap: MLA + DeepSeekMoE (§2.1)#

The paper re-states both components briefly. Anything not specified follows DeepSeek-V2.

MLA (§2.1.1). For token \(t\) with attention input \(\mathbf{h}_t \in \mathbb{R}^d\), keys and values are not projected directly. MLA first compresses \(\mathbf{h}_t\) into a latent vector:

Here \(\mathbf{c}^{KV}_t \in \mathbb{R}^{d_c}\) is the latent, \(W^{DKV}\) is the down-projection, and \(W^{UK}, W^{UV}\) are up-projections producing all \(n_h\) heads' keys and values. Rotary position embeddings (RoPE) cannot be absorbed into this low-rank path. So a separate, small "decoupled" key \(\mathbf{k}^R_t = \mathrm{RoPE}(W^{KR}\mathbf{h}_t)\) of width \(d_h^R\) is shared across all heads and concatenated onto each head's key. Only \(\mathbf{c}^{KV}_t\) and \(\mathbf{k}^R_t\) are cached. Queries are low-rank compressed too, with latent width \(d_c'\), to save activation memory in training.

With V3's settings (\(d_c = 512\), \(d_h^R = 64\), \(n_h = 128\) heads of \(d_h = 128\); §4.2), each token caches \(512 + 64 = 576\) numbers per layer. Standard multi-head attention with the same heads would cache \(2 \times 128 \times 128 = 32{,}768\). That is roughly 57× less (our arithmetic from the §4.2 config, not a figure the paper states).

DeepSeekMoE (§2.1.2). The FFN output for token input \(\mathbf{u}_t\) is

\(N_s\) shared experts always run. Of the \(N_r\) routed experts, only the top-\(K_r\) by affinity get a nonzero gate \(g_{i,t}\). V3 uses \(N_s = 1\), \(N_r = 256\), \(K_r = 8\), and an expert hidden size of 2048. The first three layers stay dense (§4.2).

Two small changes from V2:

• The affinity is a sigmoid, \(s_{i,t} = \mathrm{Sigmoid}(\mathbf{u}_t^\top \mathbf{e}_i)\), where \(\mathbf{e}_i\) is expert \(i\)'s centroid. V2 used a softmax.
• The gates are the selected affinities normalized to sum to 1 over the chosen \(K_r\) experts (Eq. 13).

Node-limited routing. Each token may be sent to experts on at most \(M = 4\) nodes. Nodes are chosen by the sum of the highest \(K_r/M\) affinities on each node. This caps the expensive cross-node InfiniBand traffic. The 256 routed experts of each layer are spread over 64 GPUs on 8 nodes during training (§4.2).

No token dropping. Balance is good enough that V3 drops no tokens in training or inference (§2.1.2).

2. Auxiliary-loss-free load balancing (§2.1.2, §4.5.2–4.5.3)#

The problem. The standard fix for imbalance is an auxiliary loss added to the training objective. If it is weak, load becomes unbalanced. If it is strong, it pushes routing away from what is best for language modelling and quality drops. You end up trading quality against balance.

The fix, from Wang et al., arXiv 2408.15664: give each expert a bias \(b_i\) that affects only which experts get selected, not how much their outputs are weighted:

After each training step, look at each expert's load over the whole batch. If the expert was overloaded, set \(b_i \leftarrow b_i - \gamma\). If it was underloaded, set \(b_i \leftarrow b_i + \gamma\). The bias update speed \(\gamma\) is 0.001 for the first 14.3T tokens and 0 for the final 500B (§4.2).

Intuition: the bias is a price signal. A popular expert becomes slightly "more expensive", so marginal tokens spill over to the next-best expert. No gradient ever flows through \(b_i\), so it never distorts what the model learns. The gate value multiplied into the output is still the original \(s_{i,t}\).

A safety net. A tiny sequence-wise balance loss is kept, with \(\alpha = 0.0001\), to prevent extreme imbalance inside a single sequence (Eqs. 17–20):

• \(f_i\) is expert \(i\)'s share of routed slots in the sequence, scaled by \(N_r/K_r\) so that 1 means "fair share".
• \(P_i\) is the average normalized affinity for expert \(i\) over the sequence.

Evidence. Table 5 compares pure aux-loss baselines against the aux-loss-free method at two scales. At 15.7B total parameters and 1.33T tokens, and at 228.7B total and 578B tokens, the aux-loss-free version wins on most benchmarks. At the large scale, for example, HumanEval goes from 40.2 to 46.3 and GSM8K from 70.7 to 74.5.

Why it helps (§4.5.3). The bias balances load over the whole batch, not within each sequence. A batch-level constraint lets experts specialize by domain: a math-heavy sequence can lean on math experts as long as the batch as a whole is balanced. Figure 9 shows stronger domain specialization (Wikipedia vs GitHub vs DM Mathematics) in the aux-loss-free model.

The authors check this explanation with a batch-wise auxiliary loss. It matches the aux-loss-free method: validation loss is 2.253 for both versus 2.258 for a sequence-wise loss at 1B parameters, and 2.080 versus 2.085 at 3B. So the scope of balancing matters more than whether a loss is used. The cost of batch-level balance is possible imbalance inside small batches, or under domain shift at inference. Large training micro-batches and redundant experts at serving time handle these.

3. Multi-token prediction (§2.2, §4.5.1, §5.4.3)#

Plain version. Besides predicting token \(t_{i+1}\) from position \(i\), the model also predicts \(t_{i+2}\). That is a denser training signal and may push hidden states to "plan ahead".

How V3 differs from Gloeckle et al. (arXiv 2404.19737). Gloeckle et al. predict future tokens in parallel with independent heads. V3 predicts them sequentially, so each extra prediction still sees the true previous tokens. The paper calls this keeping "the complete causal chain".

The \(k\)-th MTP module has its own single Transformer block \(\mathrm{TRM}_k\) and projection matrix \(M_k \in \mathbb{R}^{d\times 2d}\). It shares the embedding layer and output head with the main model. For position \(i\) at depth \(k\):

Here \(\mathbf{h}^{k-1}_i\) is the previous depth's representation (the main model's output when \(k=1\)), and \([\cdot;\cdot]\) is concatenation. This goes through \(\mathrm{TRM}_k\), and the shared head predicts token \(t_{i+k+1}\). Each depth gets a cross-entropy loss \(\mathcal{L}^k_{\text{MTP}}\). The total extra loss is \(\mathcal{L}_{\text{MTP}} = \frac{\lambda}{D}\sum_{k=1}^D \mathcal{L}^k_{\text{MTP}}\).

V3 uses depth \(D = 1\), so one extra token. The weight is \(\lambda = 0.3\) for the first 10T tokens and \(0.1\) for the remaining 4.8T (§4.2).

Example. For input "The cat sat on", the main model at position "sat" predicts "on". The MTP module takes the main model's hidden state at "sat" plus the embedding of the true next token "on", and predicts the word after "on". In Figure 3's notation, input \(t_1\) yields targets \(t_2\) (main) and \(t_3\) (MTP-1).

Inference. The MTP module can be dropped, leaving an ordinary model. Or it can be used as a built-in draft model for speculative decoding. The second-token prediction is accepted 85–90% of the time, giving 1.8× tokens per second (§5.4.3).

Evidence (Table 4). Adding a depth-1 MTP module helps most benchmarks at the same inference cost. At large scale (228.7B total, 540B tokens), HumanEval goes from 44.5 to 53.7 and GSM8K from 72.3 to 74.0. It is not uniform: MMLU drops from 67.5 to 66.6 at large scale.

4. DualPipe and computation–communication overlap (§3.2.1)#

Training layout (§3.2). HAI-LLM, DeepSeek's in-house framework, runs 16-way PP, 64-way EP across 8 nodes, and ZeRO-1 DP. There is no tensor parallelism, thanks to memory savings described below.

The problem. Cross-node expert parallelism makes the compute-to-communication ratio about 1:1. Run naively, GPUs would wait on the network about half the time.

The trick. Split each chunk of work into attention, all-to-all dispatch, MLP and all-to-all combine. Following ZeroBubble (arXiv 2401.10241), split each backward pass into "backward for input" and "backward for weights". Then pair one forward chunk with one backward chunk from a different micro-batch, and interleave their pieces. While chunk A's tokens travel over the network, chunk B computes, and vice versa (Figure 4). The number of SMs given to communication versus computation is tuned by hand so that both all-to-all and PP communication are fully hidden.

Bidirectional schedule. DualPipe feeds micro-batches into the pipeline from both ends at once (Figure 5). Each device therefore holds layers from both the front and back of the model. This creates many forward/backward pairs to overlap and shrinks the bubble. Table 2 compares the schemes (\(PP\) = pipeline stages; \(F\), \(B\), \(W\) = forward, full backward, backward-for-weights time; \(F\&B\) = an overlapped forward+backward pair):

The bubble count roughly halves, at the price of two copies of parameters. With 64-way EP each GPU holds few expert weights, so this costs little. DualPipe only needs the stage and micro-batch counts to be even. Neither the bubble nor activation memory grows with more micro-batches. The authors argue that if the compute-to-communication ratio is held constant, experts can keep being spread across more nodes at near-zero all-to-all overhead.

5. All-to-all kernels and memory savings (§3.2.2–3.2.3)#

Topology-aware dispatch. On the H800 cluster, NVLink inside a node gives 160 GB/s and InfiniBand between nodes gives 50 GB/s, about 3.2× less.

1. A token first goes over IB to the GPU with the same local index on each of its (at most 4) target nodes.
2. That GPU immediately forwards it over NVLink to the GPUs hosting its experts.

IB carries each token at most once per node and NVLink fans it out. On average a token can reach 3.2 experts per node at no extra NVLink cost. In principle that allows up to 13 experts per token (4 × 3.2) at the same communication cost, although V3 uses 8.

Few SMs. Using warp specialization, 20 SMs are split into 10 communication channels. Separate warps handle IB send, IB→NVLink forward, and NVLink receive. Warp counts adapt to load. Custom PTX instructions and auto-tuned chunk sizes limit L2 cache pollution, which would otherwise slow the computation kernels running alongside.

Memory (§3.2.3). RMSNorm and MLA up-projections are recomputed in backward instead of stored. The weight EMA lives in CPU memory. Because DualPipe puts the embedding and output head on the same PP rank, the MTP module physically shares them with the main model.

6. FP8 mixed-precision training (§3.3, Appendix B)#

The framework (Figure 6). All three GEMMs of each linear layer run in FP8: forward (Fprop), activation gradient (Dgrad) and weight gradient (Wgrad). In theory this doubles speed over BF16, and activations saved for backward can be stored in FP8.

Several parts stay in BF16 or FP32:

• embedding, output head, MoE gating, normalization and attention;
• master weights, weight gradients and optimizer states, which are sharded across DP ranks to limit the memory cost.

Validated against BF16 at about 16B and 230B total parameters on roughly 1T tokens, the relative loss error stays below 0.25% (§3.3, Appendix B.1).

Four ingredients make it work:

(a) Fine-grained quantization. A standard FP8 recipe uses one scale per tensor, so one outlier forces everything else toward zero. V3 instead uses one scale per 1×128 tile for activations (per token, per 128 channels) and one per 128×128 block for weights. An outlier only hurts its own group. The scales lie along the GEMM's inner dimension, which standard FP8 GEMMs don't support, so dequantization is folded into step (b). The authors note this matches the "microscaling" formats that Blackwell Tensor Cores support natively.

(b) Higher-precision accumulation. The authors found that H800 Tensor Cores accumulate FP8 products with only about 14 bits of precision. On a random GEMM with inner dimension \(K = 4096\) this gives up to about 2% relative error. Their fix is to accumulate on Tensor Cores for \(N_C = 128\) elements (4 WGMMA instructions), then copy the partial sum to FP32 registers on the CUDA cores, multiply by the group scales, and add. Two warpgroups alternate, one promoting while the other multiplies, so Tensor Core utilization stays high.

(c) E4M3 everywhere. Earlier FP8 recipes used E4M3 forward and E5M2 for gradients, to get more range. With small scaling groups, the shared scale supplies the range. V3 therefore uses higher-precision E4M3 for all tensors.

(d) Online scaling. Each tile or block's max-abs value is computed on the fly, rather than "delayed scaling" from past iterations' history.

Storage and communication (§3.3.3). AdamW moments are stored in BF16. Activations entering MoE up-projections are quantized to FP8 before dispatch, so the dispatch all-to-all moves 1-byte instead of 2-byte values; combine stays BF16. The input to the linear layer after attention, which is precision-sensitive, uses a custom E5M6 format.

What failed (Appendix B.2). Quantizing activation gradients in 128×128 blocks, like weights, made a 16B MoE model diverge after about 300B tokens. The authors attribute this to token-correlated outliers in gradients. This is why activations use 1×128 tiles.

7. Inference and deployment (§3.4)#

Serving is split into separate prefill (processing the prompt) and decode (generating tokens) clusters, so each can meet its own latency goal.

Redundant experts duplicate the experts most loaded in live traffic, recomputed periodically (e.g. every 10 minutes). This covers the inference-time weakness of batch-level balancing, since real traffic can be domain-skewed. In decode, the shared expert is treated as a routed expert every token picks (9 per token). Per-expert decode batches are small (usually up to 256 tokens), so the MoE part is memory-bound and can run on few SMs.

8. Hardware wishlist (§3.5)#

The authors ask chip vendors for a communication co-processor so SMs stop moving data, one interface across IB and NVLink, full-precision FP8 accumulation, native group-scaled MMA, FP8 casting fused with TMA loads, and transposed GEMM reads.

9. Pre-training details and post-training (§4, §5)#

Pre-training (§4.1–4.3). The 14.8T-token corpus has more math and code than V2's. Fill-in-the-Middle is applied at rate 0.1, and the tokenizer is byte-level BPE with 128K vocabulary. AdamW runs at 4K sequence length with peak learning rate \(2.2\times10^{-4}\). Context is then extended with YaRN in two 1,000-step phases, 4K→32K→128K.

Post-training (§5). SFT uses 1.5M instances. Reasoning data comes from an internal DeepSeek-R1 model, whose outputs are accurate but long. Per-domain expert models trained with SFT+RL turn them into concise answers, filtered by rejection sampling. RL uses GRPO with rule-based rewards (boxed math answers, code tests) plus a model-based reward model. In a DeepSeek-V2.5 ablation, R1 distillation raised MATH-500 from 74.6 to 83.2 but doubled response length, 769 → 1,510 tokens (Table 9).

Key numbers#

How to read the paper#

• 1 Introduction — Read closely. Table 1 (costs) and the contribution list map the whole paper. Note the caveat that costs exclude research and ablations.
• 2.1.1 Multi-Head Latent Attention — Skim if you've read DeepSeek-V2, otherwise read closely. Eqs. 1–11 and the "only blue boxes are cached" remark are what matter.
• 2.1.2 DeepSeekMoE with Auxiliary-Loss-Free Load Balancing — Read closely. Eqs. 12–16 and the bias-update paragraph are the main architectural novelty. Also note node-limited routing and no token dropping.
• 2.2 Multi-Token Prediction — Read closely with Figure 3 open. Trace which embeddings feed which module.
• 3.1 Compute Clusters — Skim (one paragraph).
• 3.2 Training Framework / 3.2.1 DualPipe — Read closely. Figures 4–5 and Table 2. Figure 5 takes a few minutes to decode; follow micro-batch 0 from both ends.
• 3.2.2 Cross-node all-to-all — Read closely if you do systems work, skim otherwise. The IB→NVLink two-hop idea is the key point.
• 3.2.3 Memory saving — Skim. Three short tricks.
• 3.3 FP8 Training — Read closely. Figures 6–7. The 14-bit accumulation finding and the 1×128 / 128×128 scheme are the core. Read Appendix B.2 alongside it.
• 3.4 Inference and Deployment — Read if you serve models. The prefill/decode table above summarizes it.
• 3.5 Suggestions on Hardware Design — Skim, but it shows clearly where the H800 limited them.
• 4.1–4.3 Data, hyper-parameters, long context — Skim on first pass; come back for exact schedules.
• 4.4 Evaluations — Skim Table 3.
• 4.5 Discussion (ablations) — Read closely. Tables 4–5 are the evidence for MTP and aux-loss-free balancing. §4.5.3's batch-wise vs sequence-wise result explains why the method works.
• 5.1–5.2 SFT and RL — Read the reasoning-data paragraph (R1 distillation). Skim GRPO if you know DeepSeekMath.
• 5.3 Evaluations — Skim Tables 6–8.
• 5.4 Discussion — Read 5.4.1 (Table 9) and 5.4.3 (MTP acceptance rate). 5.4.2 is a short note on self-rewarding.
• 6 Conclusion, Limitations — Read (one page).
• Appendix B — Read B.2 (block-wise gradient quantization diverges). Appendix C — Skip (more Figure 9-style plots).

Suggested order: §1 → §2 → §4.5 → §3.2 → §3.3 + App. B → §3.4 → §5.1 → Tables 3 and 6 → §6. Expect about 2.5–3 hours for a careful first read, or 1 hour for §1, §2, §4.5 and the FP8 section alone.

Check your understanding#

Limitations and open questions#

• Deployment unit is large. The authors acknowledge that the recommended serving setup (32 GPUs for prefill, 320 for decode) is a burden for small teams. They also say generation speed, though more than 2× DeepSeek-V2's, can still improve (§6).
• The cost figure is narrow. The $5.576M covers only the final training run at an assumed rental price. It excludes prior research, ablations, data work and owning the cluster, which the paper states explicitly.
• Ablations are at smaller scale. MTP and aux-loss-free balancing are validated at 15.7B and 228.7B total parameters, not 671B. Gains are "most benchmarks", not all: MTP slightly lowers large-scale MMLU and aux-loss-free lowers large-scale MMLU in Tables 4–5.
• Contributions are entangled. There is no ablation that isolates the value of FP8, DualPipe, R1 distillation and data changes in the final model. The headline comparison to V2 changes everything at once.
• Factual knowledge lags. SimpleQA is 24.9 vs GPT-4o's 38.2. The authors attribute this to allocating more tokens to Chinese knowledge.
• Evaluation caveats. Base-model comparisons use DeepSeek's internal framework. The authors note V2-Base scores shifted slightly because of framework changes. Chat evaluations cap outputs at 8K tokens.
• Hardware-specific engineering. Much of the efficiency (accumulation workarounds, SM-based communication, two-hop routing) is tuned to H800 limits and may not transfer to other hardware.
• Dependence on an unreleased model. Post-training distills from an internal DeepSeek-R1 that the paper does not describe.

Further reading#

• Earlier in the series: DeepSeekMoE for fine-grained and shared experts; DeepSeek-V2 for MLA, device-limited routing and the base recipe; DeepSeekMath for GRPO; DeepSeek-Coder-V2 for FIM in a general MoE model; Fire-Flyer AI-HPC for DeepSeek's earlier cluster and HFReduce/HaiScale/3FS stack.
• Prior work it builds on: Auxiliary-loss-free load balancing, Wang et al. (arXiv 2408.15664); Multi-token prediction, Gloeckle et al. (arXiv 2404.19737); Zero Bubble pipeline parallelism, Qi et al. (arXiv 2401.10241); GShard (arXiv 2006.16668); Switch Transformer (arXiv 2101.03961); YaRN (arXiv 2309.00071).
• What came after: DeepSeek-R1 (arXiv 2501.12948), trained from V3-Base; Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures, ISCA 2025 (arXiv 2505.09343).
• Code and weights: github.com/deepseek-ai/DeepSeek-V3, as given in the paper. Related infrastructure released later: DualPipe, DeepEP (all-to-all kernels) and DeepGEMM (FP8 GEMMs).