DeepSeek-V2

TL;DR#

• DeepSeek-V2 is a Mixture-of-Experts (MoE) transformer with 236B total parameters, 21B activated per token, a 128K context, and 8.1T pre-training tokens (Abstract, §3.1). It is followed by SFT and RL (GRPO).
• Multi-head Latent Attention (MLA) caches one shared 512-dim latent vector plus a 64-dim RoPE key per token per layer instead of full per-head keys and values. That is the same cache as GQA with only 2.25 groups, and in the paper's ablations it scores higher than full multi-head attention (Table 1, Table 9).
• DeepSeekMoE (fine-grained routed experts plus always-on shared experts) is scaled up to 2 shared + 160 routed experts, 6 routed active, with three additions for expert parallelism: device-limited routing (each token reaches at most 3 devices), three balance losses, and device-level token dropping (§2.2).
• Against DeepSeek 67B (dense), V2 is stronger on almost every benchmark. It saves 42.5% of training GPU-hours per trillion tokens, cuts the KV cache by 93.3%, and reaches 5.76x the maximum generation throughput, over 50K tokens/s on one 8×H800 node (§3.2.3).
• Base model: MMLU 78.5, with Mixtral 8x22B at 77.6 and LLaMA 3 70B at 78.9 (Table 2). Chat (RL): MT-Bench 8.97, AlpacaEval 2.0 LC 38.9, AlignBench 7.91 (Tables 4–5).
• The paper also releases DeepSeek-V2-Lite (15.7B total, 2.4B active), a small model with the same MLA + MoE design for research (Appendix B).

Why it matters#

Large models face two separate cost problems. Training cost scales with the parameters active per token. Inference cost, during token-by-token generation, is often limited by memory rather than compute: every new token must read the cached keys and values of every earlier token in every layer. At long context and large batch, this KV cache controls how many requests fit on a GPU, and decoding becomes memory-bound. The usual fixes, Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), share KV heads across query heads. They shrink the cache but lose quality. DeepSeek's own 7B ablation shows MHA > GQA > MQA on hard benchmarks (Appendix D.1, Table 8).

DeepSeek-V2 addresses both problems. DeepSeekMoE (January 2024) handles training cost by activating a small fraction of the FFN parameters per token, and V2 is the first time DeepSeek scales it to a flagship model, which requires engineering for cross-device communication. MLA, new in this paper, handles the inference cost: compress K and V into a small shared latent, and use matrix associativity so the full keys and values are never rebuilt at inference.

Key idea Cache a low-rank latent instead of keys and values. Because the up-projections from the latent are linear, they can be folded into the query and output projections, so attention runs directly against the compressed cache. RoPE breaks that folding, so position information is carried by a separate small shared key.

This architecture became DeepSeek's base for the rest of 2024. DeepSeek-Coder-V2 continues pre-training from a V2 checkpoint. ESFT fine-tunes only the relevant experts of V2-style fine-grained MoE models. DeepSeek-VL2 builds on DeepSeekMoE-based language models. DeepSeek-V3 keeps MLA and DeepSeekMoE and changes the load-balancing approach. The RL stage uses GRPO from DeepSeekMath.

Background you need#

• KV cache. During autoregressive generation, each layer stores the key and value vectors of all previous tokens so they are not recomputed. For MHA its size per token is \(2 n_h d_h l\) elements (\(n_h\) heads, head dimension \(d_h\), \(l\) layers). It lives in GPU RAM and is read in full for every generated token.
• MQA / GQA. Multi-Query Attention (Shazeer 2019) uses one K/V head for all query heads. GQA (Ainslie et al. 2023) uses \(n_g\) groups. The cache shrinks by \(n_h\) or \(n_h/n_g\), at some cost in quality. DeepSeek LLM 67B used GQA with 8 groups.
• RoPE. Rotary Position Embedding (Su et al.) rotates query and key vectors by an angle that depends on position, so \(q_t^\top k_j\) depends on the offset \(t-j\). The rotation is applied after projection, and that is what makes it a problem for MLA.
• Mixture-of-Experts and expert parallelism. An MoE layer replaces one FFN with many expert FFNs and a router that sends each token to a few of them. With expert parallelism (EP), experts live on different GPUs, and tokens are exchanged in an all-to-all step. Load imbalance means some GPUs wait while overloaded ones finish.
• DeepSeekMoE. Splits each expert into smaller ones (fine-grained segmentation), so more experts are active per token at the same compute, and adds a few shared experts that every token uses, to hold common knowledge. See DeepSeekMoE.
• GRPO. Group Relative Policy Optimization, a PPO variant without a value network. It samples several responses per prompt and uses each response's reward, normalized within its group, as the advantage. See DeepSeekMath.

The core ideas#

1. Where the KV cache comes from (MHA recap)#

Notation follows §2.1.1. With embedding dimension \(d\), \(n_h\) heads of size \(d_h\), and input \(\mathbf{h}_t \in \mathbb{R}^d\) for token \(t\), MHA computes \(\mathbf{q}_t = W^Q \mathbf{h}_t\), \(\mathbf{k}_t = W^K \mathbf{h}_t\) and \(\mathbf{v}_t = W^V \mathbf{h}_t\), each in \(\mathbb{R}^{n_h d_h}\), and splits them into heads:

To generate token \(t\), you need \(\mathbf{k}_{j,i}\) and \(\mathbf{v}_{j,i}\) for every earlier \(j\) and every head \(i\). That is \(2 n_h d_h\) numbers per token per layer. For DeepSeek-V2's shape (\(n_h = 128\), \(d_h = 128\)), that is 32,768 elements per token per layer. Over 60 layers it is about 1.97M elements per token (our arithmetic). At 128K context that cache is enormous.

2. MLA step one: low-rank joint KV compression#

MLA puts a small bottleneck in front of keys and values (§2.1.2, Eqs. 9–11):

• \(\mathbf{c}^{KV}_t \in \mathbb{R}^{d_c}\) is the compressed latent, shared by keys and values and by all heads. In V2, \(d_c = 512\), while \(n_h d_h = 16{,}384\).
• \(W^{DKV} \in \mathbb{R}^{d_c \times d}\) is the down-projection. \(W^{UK}, W^{UV} \in \mathbb{R}^{d_h n_h \times d_c}\) are the up-projections back to full per-head keys and values.

Only \(\mathbf{c}^{KV}_t\) is cached, so the cache is \(d_c\) elements per token per layer instead of \(2 n_h d_h\).

This differs from GQA. GQA discards information by forcing heads to share identical K/V. In MLA, each head still gets its own key and value, \(W^{UK}_i \mathbf{c}^{KV}_t\) and \(W^{UV}_i \mathbf{c}^{KV}_t\), as different linear readouts of a shared 512-dim code. The constraint is that all heads' keys and values together have rank at most 512. That turns out to cost less than forcing heads to share K/V.

The absorption trick. If inference had to rebuild \(\mathbf{k}^C\) and \(\mathbf{v}^C\) for every past token at every step, compression would save memory but add a lot of compute. It doesn't have to. For head \(i\), write \(W^{UK}_i, W^{UV}_i \in \mathbb{R}^{d_h \times d_c}\) for that head's slice. Then

So you map the query into the 512-dim latent space once per head and take dot products directly with the cached latents. On the value side,

where \(a_{tj}\) are the attention weights. \(W^{UV}_i\) can therefore be applied after the weighted sum, or folded into the output projection \(W^O\). The paper summarizes this: "\(W^{UK}\) can be absorbed into \(W^Q\), and \(W^{UV}\) can be absorbed into \(W^O\)", so keys and values are never computed explicitly (§2.1.2, Appendix C).

One useful way to see the absorbed form (our framing): every head attends over the same cached vectors, as in MQA, but each head has its own learned query map into the latent space and its own output map out of it. The memory traffic is MQA-like and the expressiveness is close to MHA.

Query compression. Queries also go through a bottleneck, \(\mathbf{c}^Q_t = W^{DQ}\mathbf{h}_t\) and \(\mathbf{q}^C_t = W^{UQ}\mathbf{c}^Q_t\), with \(d'_c = 1536\) (Eqs. 12–13). This has nothing to do with the cache, because queries aren't cached. It reduces activation memory during training. V2-Lite skips it (Appendix B.1).

3. MLA step two: decoupled RoPE#

RoPE breaks the absorption trick (§2.1.3). With RoPE on the compressed keys, the score becomes

where \(R_t\) is the rotation for position \(t\). A position-dependent matrix \(R_{j-t}\) now sits between the query side and \(W^{UK}_i\), and it differs for every (query, key) pair. No single fixed matrix can be folded into the query projection, so you would have to rebuild rotated keys for the whole prefix at every step, which is the cost MLA was meant to avoid.

The fix is to split each head's query and key into two parts: a "content" part without RoPE, which goes through the latent and can be absorbed, and a small "rotary" part that carries position (Eqs. 14–19):

• \(\mathbf{q}^R_{t,i} \in \mathbb{R}^{d^R_h}\) is a per-head rotary query. \(\mathbf{k}^R_t \in \mathbb{R}^{d^R_h}\) is a single rotary key shared across all heads, MQA-style, so it costs only \(d^R_h\) cache elements. In V2, \(d^R_h = 64\).
• The score is a sum of two dot products: content (absorbable, through the latent) plus rotary (computed directly on the small cached \(\mathbf{k}^R\)). Values have no rotary part.
• The softmax scale uses \(\sqrt{d_h + d^R_h} = \sqrt{192}\), because the effective query/key dimension per head is 128 + 64.

The cache per token per layer is now \(\mathbf{c}^{KV}_t\) and \(\mathbf{k}^R_t\): \(d_c + d^R_h = 512 + 64 = 576\) elements.

4. How small is the cache?#

Table 1 compares mechanisms in elements per token:

With \(d_c = 4 d_h\) and \(d^R_h = d_h/2\), MLA's 576 elements equal GQA with 2.25 groups (\(2 \times 2.25 \times 128 = 576\)). Compared with full MHA at V2's shape, the cache is \(576 / 32{,}768 \approx 1.8\%\) (our arithmetic).

Where the 93.3% comes from (our reconstruction; the paper does not show the derivation). DeepSeek 67B uses GQA with 8 KV heads, \(d_h = 128\) and 95 layers: \(2 \times 8 \times 128 \times 95 = 194{,}560\) elements per token. DeepSeek-V2: \(576 \times 60 = 34{,}560\) elements. In elements alone that is an 82% reduction. The deployed V2 also quantizes its KV cache to about 6 bits per element (§3.2.3). If 67B's cache is 16-bit, the byte ratio is \((34{,}560 \times 6)/(194{,}560 \times 16) \approx 6.7\%\), a 93.3% reduction, which matches the headline. So the 93.3% likely combines MLA with KV quantization, and MLA alone accounts for about 82%.

Quality. Appendix D.2 (Table 9) compares MLA and MHA in MoE models at two scales:

MLA uses 14% (small) and 4% (large) of MHA's cache and matches or beats it on 7 of 8 numbers. The small models were trained on 1.33T tokens and the large ones on 420B.

5. DeepSeekMoE inside V2#

Every FFN except the first layer's is an MoE layer (§3.1.2). For FFN input \(\mathbf{u}_t\) (Eqs. 20–22):

• \(N_s = 2\) shared experts run on every token with no gate.
• \(N_r = 160\) routed experts. \(\mathbf{e}_i\) is expert \(i\)'s learned centroid, \(s_{i,t}\) is the token-to-expert affinity, and the top \(K_r = 6\) are used, weighted by their (unrenormalized) softmax scores.
• Each expert is a small FFN with intermediate size 1536, the same width as the query latent.

So each token passes through 8 small experts (2 shared + 6 routed), a combined intermediate width of \(8 \times 1536 = 12{,}288\) (our arithmetic). Most of the 236B parameters sit in the 160 routed experts per layer, which is why only 21B are active.

Two stability details: extra RMSNorm layers after the compressed latents, and extra scaling factors at the "width bottlenecks" (the latents and the routed experts' intermediate states). The paper says low-rank compression and fine-grained experts change the layer output scale and that these keep training stable. It does not give the factor values (§3.1.2).

6. Device-limited routing#

Fine-grained experts mean many active experts per token. With 160 routed experts spread over 8 GPUs (20 per GPU, our arithmetic), an unconstrained top-6 could send a token to 6 different GPUs, and all-to-all traffic scales with the number of devices each token reaches.

V2 adds a constraint (§2.2.2). For each token, first choose the \(M\) devices whose experts have the highest affinity scores, then do top-\(K\) only among experts on those devices. V2 uses \(D = 8\) devices for routed experts per layer and \(M = 3\) (§3.1.2). The paper reports that \(M \geq 3\) performs "roughly aligned" with unrestricted top-K. In the example, a token's 6 experts come from at most 60 candidates (3 devices × 20 experts) instead of 160, and its hidden state is sent to at most 3 GPUs instead of up to 6.

7. Three balance losses#

Unbalanced routing has two costs: routing collapse, where a few experts get all the tokens and others never train, and idle devices under EP. V2 uses three auxiliary losses (§2.2.3), all of the form "fraction of load × average affinity", as in Switch Transformer and GShard:

Expert-level (\(\alpha_1 = 0.003\)), Eqs. 23–25:

\(T\) is the number of tokens in a sequence. \(f_i\) is normalized so it equals 1 for every expert under perfectly uniform routing. \(P_i\) is differentiable and carries the gradient. Its gradient lowers the router's affinity most for over-selected experts (large \(f_i\)), which shifts tokens toward under-used experts.

Device-level (\(\alpha_2 = 0.05\)), Eqs. 26–28: the same loss over device groups \(\mathcal{E}_1, \dots, \mathcal{E}_D\), with \(f'_i\) the mean of \(f_j\) within a group and \(P'_i\) the sum of \(P_j\). The coefficient is much larger than \(\alpha_1\). The design accepts some imbalance among experts on the same GPU, but not between GPUs, since only cross-GPU imbalance causes waiting.

Communication (\(\alpha_3 = 0.02\)), Eqs. 29–31: \(f''_i = \frac{D}{MT}\sum_t \mathbb{1}(\text{token } t \text{ is sent to device } i)\), with \(P''_i = \sum_{j \in \mathcal{E}_i} P_j\). Device-limited routing caps what each device sends at \(MT\) hidden states. This loss encourages each device to receive about \(MT\), so no device's inbound link becomes the bottleneck.

The small \(\alpha_1\) reflects a trade-off: balance losses help efficiency but interfere with the language-modeling objective. DeepSeek-V3 later replaces most of this with a bias-based, auxiliary-loss-free method.

8. Device-level token dropping#

The losses encourage balance but cannot guarantee it, so V2 also drops tokens during training (§2.2.4). Each device gets a compute budget equal to the average load (capacity factor 1.0). If it is over budget, the tokens with the lowest affinity on that device are dropped, meaning they skip that expert and keep only the residual and other experts' outputs. Tokens from about 10% of training sequences are never dropped, so the model also sees drop-free computation. That lets inference run without dropping and stay consistent with training. Evaluation never drops tokens (§3.1.2).

9. Training setup and long context#

• Shape: 60 layers, hidden 5120, 128 heads × 128 dims, \(d_c = 512\), \(d'_c = 1536\), \(d^R_h = 64\), init std 0.006 (§3.1.2).
• Data: 8.1T tokens, about 12% more Chinese than English. Same pipeline as DeepSeek 67B but with more recovered web data, more Chinese, better quality filtering, and removal of "contentious content" (§3.1.1, Appendix E). Same 100K BBPE tokenizer.
• Optimizer: AdamW (0.9, 0.95, wd 0.1), peak LR \(2.4 \times 10^{-4}\), 2K warmup steps, ×0.316 at about 60% and again at about 90% of tokens. This is the step schedule from DeepSeek LLM, with the first drop moved earlier. Batch size ramps from 2304 to 9216 over the first 225B tokens. Sequence length 4K.
• Parallelism: HAI-LLM with 16-way zero-bubble pipeline parallelism, 8-way expert parallelism and ZeRO-1 data parallelism. No tensor parallelism, which works because the active parameters are few and some operators are recomputed. Shared-expert compute overlaps with the EP all-to-all. Custom CUDA kernels handle communication, routing and fused expert linears, and MLA runs on an improved FlashAttention-2. Hardware is NVIDIA H800 nodes (8 GPUs over NVLink/NVSwitch, InfiniBand between nodes) (§3.1.3). See Fire-Flyer for DeepSeek's cluster design.
• Long context: YaRN applied only to the shared rotary key \(\mathbf{k}^R_t\), which is the only component carrying RoPE, with \(s = 40\), \(\alpha = 1\), \(\beta = 32\) and a target of 160K. The attention-entropy factor is modified to \(\sqrt{t} = 0.0707 \ln s + 1\). Then 1000 extra steps at 32K sequence length with batch 576. Needle-in-a-Haystack passes up to 128K (Figure 4, §3.1.4).

Decoupled RoPE makes context extension simpler: only the 64-dim rotary path needs rescaling, and the latent path never sees positions.

10. Alignment: SFT, then two-stage GRPO#

• SFT: 1.5M instances (1.2M helpfulness, 0.3M safety), 2 epochs, LR \(5 \times 10^{-6}\) (§4.1). The paper reports that fewer than 10K SFT examples noticeably hurt IFEval, which goes against the "less is more" alignment claims (§4.4).
• RL algorithm: GRPO (Eqs. 32–34). For each prompt, sample \(G\) outputs from the old policy. Each output's advantage is its reward standardized within the group, \(A_i = (r_i - \text{mean})/\text{std}\), inside a PPO-style clipped objective with a KL penalty to a reference model. No critic network is needed, which matters at this model size.
• Two stages: first reasoning alignment with a code/math reward model \(RM_{reasoning}\), because math and code keep improving over longer RL. Then human preference alignment with \(r = c_1 RM_{helpful} + c_2 RM_{safety} + c_3 RM_{rule}\). Code preference data comes from compiler feedback and math preference data from ground-truth labels. Reward models are initialized from V2 Chat (SFT) (§4.2).
• Engineering: a hybrid engine with different parallel layouts for training and generation, vLLM for rollouts, and scheduled CPU offloading. The paper reports that online RL clearly beat offline in their experiments (§4.4), and also an alignment tax, for example on BBH, which drops from 81.3 to 79.7 between SFT and RL (Table 3).

Key numbers#

How to read the paper#

• 1 Introduction: Skim, but study Figure 1 (MMLU vs activated params, and the cost/KV/throughput bars) and Figure 2 (the full block diagram, worth keeping open while reading §2).
• 2.1.1 Standard MHA: Skim if you know attention. It fixes notation and the \(2 n_h d_h l\) cache formula.
• 2.1.2 Low-Rank KV Joint Compression: Read closely. Eqs. 9–11 and the one sentence about absorbing \(W^{UK}\) and \(W^{UV}\) are the core of the paper. Work through the associativity yourself. Figure 3 compares MHA, GQA, MQA and MLA visually.
• 2.1.3 Decoupled RoPE: Read closely. The first paragraph explains why RoPE breaks absorption. Eqs. 14–19 are the fix. Then go to Appendix C for the full formulas in one place.
• 2.1.4 Comparison of KV Cache: Read. Table 1 and its caption carry the "2.25 groups" calculation.
• 2.2.1 Basic Architecture: Read if you skipped DeepSeekMoE. Otherwise skim Eqs. 20–22.
• 2.2.2–2.2.4 Device-limited routing, balance losses, token dropping: Read closely if you build MoE systems, skim otherwise. Compare the normalizations of \(f_i\), \(f'_i\) and \(f''_i\).
• 3.1.1 Data Construction: Skim. Qualitative only.
• 3.1.2 Hyper-Parameters: Read closely. All the real numbers are here (\(d_c\), expert counts, \(D\), \(M\), $\alpha$s, LR and batch schedule).
• 3.1.3 Infrastructures: Read if you care about systems: no tensor parallelism, zero-bubble PP and overlapping shared experts with all-to-all.
• 3.1.4 Long Context Extension: Read. Short, and notable for applying YaRN only to \(\mathbf{k}^R\).
• 3.2.1–3.2.2 Evaluation: Skim the benchmark list. Study Table 2 and the paragraph comparing V2 against Qwen1.5, Mixtral and LLaMA 3, which is honest about where V2 loses.
• 3.2.3 Training and Inference Efficiency: Read closely. It is the source of 42.5%, FP8 weights, 6-bit KV and 5.76x.
• 4.1 SFT / 4.2 RL: Read 4.2 for the two-stage GRPO recipe. Skim 4.1.
• 4.3 Evaluation Results: Skim. Tables 3–5.
• 4.4 Discussion: Read. SFT data size, alignment tax and online vs offline RL.
• 5 Conclusion: Skim. It says the next goal is GPT-4 parity, which became V3.
• Appendix B (V2-Lite): Read if you want a model you can run. Table 6 compares it with DeepSeek 7B and DeepSeekMoE 16B.
• Appendix C (Full MLA formulas): Read closely, together with §2.1.
• Appendix D (Attention ablations): Read closely. Tables 8–9 are the evidence that MLA > MHA > GQA > MQA.
• Appendix E–G: Skip on first pass. E covers the MMLU moral-scenarios gap after debiasing, F has extra math/code results, and G has prompt formats.

Suggested order: Figure 2 → §2.1 → Appendix C → Table 1 → Appendix D → §2.2 → §3.1.2–3.1.4 → §3.2.3 → Table 2 → §4.2 and §4.4. Allow about 2 hours. MLA alone, with pencil and paper, is about 45 minutes.

Check your understanding#

Limitations and open questions#

• Headline efficiency mixes several changes. The 93.3% KV and 5.76x throughput figures compare deployed V2 (MLA, MoE, FP8 weights, ~6-bit KV, 60 layers) against DeepSeek 67B. The contribution of each change is not separated. Appendix D's MLA-vs-MHA ablation is the cleaner comparison.
• MLA ablations are few. Four benchmarks and two scales (Table 9), and the large pair was trained on only 420B tokens. There is no sweep over \(d_c\) or \(d^R_h\), and no comparison against GQA at equal cache size within MoE models.
• The quality gain over MHA is not explained. The paper reports MLA > MHA but gives no analysis of why a low-rank constraint would help. It may act as regularization or come from the extra parameters in the query path, but this is not tested.
• Unspecified details. The scaling factors at the width bottlenecks, reward-model coefficients \(c_1, c_2, c_3\), GRPO hyperparameters (\(G\), \(\varepsilon\), \(\beta\)), RL data sizes and the FFN size of the first dense layer are not given.
• Balance losses vs quality. Three auxiliary losses plus token dropping add hyperparameters and gradients that interfere with the LM loss. The paper does not measure that cost. V3's move to auxiliary-loss-free balancing suggests DeepSeek saw room to improve.
• English knowledge gap. The authors acknowledge V2 trails LLaMA 3 70B on basic English knowledge (TriviaQA, NaturalQuestions, HellaSwag), having seen less than a quarter of its English tokens (§3.2.2). They also report weaker MMLU moral-scenarios scores, which they attribute to debiasing the corpus (Appendix E).
• Alignment tax. RL improves open-ended scores but lowers some standard benchmarks, for example BBH (§4.4, Table 3). The authors leave how to avoid this for future work.
• Inference with absorption has its own costs. Attention in the absorbed form runs over 576-dim keys for every head, which raises per-token attention FLOPs. That is fine when decoding is memory-bound but less so for long prefill. The paper does not discuss this trade-off or its kernel strategy beyond "improved FlashAttention-2".

Further reading#

• In this series: DeepSeek LLM (the 67B baseline and the training recipe V2 reuses), DeepSeekMoE (the MoE design, at smaller scale), DeepSeekMath (origin of GRPO), DeepSeek-Coder-V2 (continued pre-training from a V2 checkpoint), ESFT (expert-specialized fine-tuning on fine-grained MoE models), Fire-Flyer AI-HPC (DeepSeek's training infrastructure), DeepSeek-VL2 (MoE + MLA language backbone for vision-language), and DeepSeek-V3 (scales MLA + DeepSeekMoE further, with auxiliary-loss-free balancing).
• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (MQA): arXiv:1911.02150
• Ainslie et al., GQA: arXiv:2305.13245
• Su et al., RoFormer (RoPE): arXiv:2104.09864
• Peng et al., YaRN: arXiv:2309.00071
• Lepikhin et al., GShard: arXiv:2006.16668, and Fedus et al., Switch Transformers: arXiv:2101.03961 (the balance-loss form)
• Riquelme et al., Scaling Vision with Sparse Mixture of Experts (the inspiration for dropping by lowest affinity): arXiv:2106.05974
• Qi et al., Zero Bubble Pipeline Parallelism: arXiv:2401.10241
• Code and weights (V2, V2-Lite, Chat): github.com/deepseek-ai/DeepSeek-V2