DeepSeekMoE

TL;DR#

• DeepSeekMoE is a new Mixture-of-Experts (MoE) FFN layer built on two changes to the standard top-2 design (GShard): fine-grained expert segmentation (split each expert into \(m\) smaller ones and route to \(m\) times as many) and shared expert isolation (a few experts every token always uses). Total and activated parameters stay the same.
• At 2B total parameters and 100B tokens, DeepSeekMoE (1 shared + 63 routed experts, 7 routed active) beats GShard at the same size on every benchmark in Table 1 (e.g. Pile loss 1.808 vs 1.867). It matches GShard×1.5, which has 1.5× the expert parameters and compute, and comes close to "Dense×16", which the authors treat as the upper bound for MoE (Table 2).
• DeepSeekMoE 16B (16.4B total, 2.8B active, 2T tokens) is about as good as DeepSeek 7B dense, trained on the same data, using 40.5% of the FLOPs (Table 3). It beats LLaMA2 7B on most benchmarks using 39.6% of the FLOPs (Table 4). It runs on a single 40GB GPU without quantization.
• A preliminary DeepSeekMoE 145B (22.2B active, only 245B tokens) beats GShard 137B by a wide margin and is roughly on par with DeepSeek 67B dense at 28.5% of the compute (Table 6).
• A consistent weakness: multiple-choice benchmarks such as MMLU, CEval and CMMLU. The authors blame the small attention budget (about 0.5B attention parameters in the 16B model vs 2.5B in DeepSeek 7B).

Why it matters#

MoE models replace each Transformer FFN with several "expert" FFNs and a router that sends each token to only a few of them. Total parameters grow while compute per token stays flat. By early 2024 the standard recipe was still GShard-style top-2 routing over a small pool (8 or 16 experts), as in Mixtral. The paper argues this leaves a lot on the table for two reasons. With few experts, each one gets a mix of unrelated tokens and has to store mismatched knowledge (knowledge hybridity). And because every expert needs common knowledge such as basic syntax, that knowledge gets copied across many experts (knowledge redundancy).

DeepSeek's answer is architectural, not a new loss or routing algorithm. Make experts small and numerous so knowledge can be split finely and recombined flexibly. Then pull the common knowledge into a few experts that every token visits. Both changes keep FLOPs fixed, so any gain comes from better use of the same parameters.

Key idea At constant compute, many small routed experts plus a few always-on shared experts specialize better than a few large routed experts. The shared experts absorb common knowledge, and the fine split lets each routed expert stay narrow.

This paper is the base of the rest of DeepSeek's 2024 language-model work. DeepSeek-V2 uses DeepSeekMoE as its FFN, adds device-limited routing and communication-aware balance losses for large-scale expert parallelism, and pairs it with a new attention design (MLA). That attention change addresses exactly the attention weakness this paper identifies. DeepSeek-Coder-V2 and DeepSeek-VL2 are built on DeepSeekMoE backbones. ESFT depends on fine-grained experts being specialized enough to fine-tune selectively. DeepSeek-V3 scales the same layer further and replaces most of the auxiliary balance loss with a bias-based scheme.

Background you need#

• Transformer FFN. Each block has an attention sublayer and a position-wise feed-forward network (two or three linear maps with a nonlinearity). In large models most parameters sit in FFNs, and they are widely thought to store factual knowledge (Dai et al., Knowledge Neurons).
• Mixture-of-Experts with top-\(K\) routing. A router scores each expert for each token and keeps the top \(K\). Only those experts run, and their outputs are summed, weighted by the router scores. Shazeer et al. 2017 introduced it at scale; GShard (top-2) and Switch Transformer (top-1) brought it to Transformers.
• Total vs activated parameters. Total parameters set memory footprint and capacity. Activated parameters (those used per token) set FLOPs. MoE separates the two.
• Load balancing and routing collapse. A learned router can end up sending almost all tokens to a few experts, leaving the rest untrained. An auxiliary loss that penalizes uneven load is the usual fix. When experts are spread across GPUs, uneven load also leaves some devices idle.
• Expert parallelism. Different experts live on different devices, and tokens are sent across the network to their experts. This explains the paper's separate device-level balance loss.
• DeepSeek LLM baselines. The dense DeepSeek 7B and 67B models from DeepSeek LLM were trained on the same corpus, so they are clean comparison points here.

The core ideas#

The baseline: a conventional MoE layer#

For token \(t\) at layer \(l\), with input hidden state \(\mathbf{u}_t^l \in \mathbb{R}^d\), a standard MoE layer (§2, Eqs. 3-5) computes

where \(N\) is the number of experts and \(\mathbf{e}_i^l\) is a learned "centroid" vector for expert \(i\). \(s_{i,t}\) is the token-to-expert affinity. The gate \(g_{i,t}\) equals \(s_{i,t}\) if \(s_{i,t}\) is among the top \(K\) affinities for that token and is 0 otherwise. Only \(K\) experts run per token. The residual \(+\mathbf{u}_t^l\) is part of the formula; layer norm is omitted.

Fine-grained expert segmentation#

Take each of the \(N\) experts and cut its intermediate (hidden) dimension by a factor of \(m\). You now have \(mN\) experts, each \(1/m\) the size. To keep compute unchanged, route each token to \(mK\) of them instead of \(K\) (§3.1, Eqs. 6-8). Total expert parameters and per-token FLOPs are the same as before. Only the granularity changes.

Why would this help? Routing becomes much more combinatorial. The paper's example: with \(N=16\) and top-2 routing, a token can use \(\binom{16}{2}=120\) different expert combinations. Split each expert into 4 (\(m=4\), so 64 experts, top-8) and the count becomes \(\binom{64}{8}=4{,}426{,}165{,}368\). A token that needs "Python syntax + numerical reasoning + English prose" can get a closer-fitting mix of small specialists, rather than two big generalists that each cover part of what it needs.

The ablation (§4.4, Figure 3) confirms the trend. Going from 16 experts to 32 (1 shared + 31 routed) to 64 (1 shared + 63 routed), with total and activated parameters held constant, improves performance steadily.

There is a practical lower limit on expert size. For the 16B model the authors stop at experts 0.25× the standard FFN size, "due to the potential reduction in computational efficiency associated with excessively small expert sizes" (§5.1.2): small matrix multiplies use the GPU poorly. At 145B, with a wider model, they go to 0.125×.

Shared expert isolation#

Next, set aside \(K_s\) of the experts as shared experts. Every token goes through them; the router is not involved. To keep compute fixed, route to \(K_s\) fewer experts among the rest (§3.2, Eqs. 9-11):

with \(g_{i,t}\) now nonzero for the top \(mK-K_s\) routed experts only. So the layer has \(K_s\) shared experts, \(mN-K_s\) routed experts, and \(mK-K_s\) active routed experts per token.

The reasoning: knowledge every token needs should live in one place, not be relearned by every routed expert. The authors credit DeepSpeed-MoE (Rajbhandari et al. 2022) with the prototype, noting that it came from an engineering angle and this paper from an algorithmic one.

Worked example (the 2B validation model, §4.2). GShard baseline: 16 experts of standard FFN size, top-2, so 2 FFNs' worth of compute per token. DeepSeekMoE: \(m=4\) gives 64 experts of size 0.25. Make 1 shared, leaving 63 routed, and activate 7 of those. Per token: \((1+7)\times 0.25 = 2\) FFNs of compute, the same as GShard. Total: \(64\times0.25=16\) FFNs of parameters, also the same.

How many shared experts? With 64 experts total and fixed activation, 1, 2 and 4 shared experts reach Pile loss 1.808, 1.806 and 1.811 (§4.4), which is almost no difference. The authors keep the 1:3 ratio of shared experts to activated routed experts (2 shared : 6 routed) when scaling up.

Load balancing: two auxiliary losses#

§3.3 defines two balance losses. Let \(N'=mN-K_s\) be the number of routed experts, \(K'=mK-K_s\) the number activated per token, and \(T\) the number of tokens.

Expert-level (against routing collapse):

\(f_i\) is expert \(i\)'s share of selections, scaled so a perfectly uniform router gives \(f_i=1\). \(P_i\) is its average router probability. The product is smallest when load is spread evenly.

Device-level (against stragglers under expert parallelism). Partition routed experts into \(D\) groups \(\mathcal{E}_1,\dots,\mathcal{E}_D\), one per device, and apply the same form to groups: \(f'_i\) is the mean of \(f_j\) over the group and \(P'_i\) is the sum of \(P_j\), weighted by \(\alpha_2\).

The design choice to notice: a small \(\alpha_1\), a larger \(\alpha_2\). Forcing every expert to have equal load hurts quality. What matters for speed is that devices are balanced. In practice:

For 16B the authors note that a larger \(\alpha_1\) "cannot increase the computation efficiency, but instead, it will compromise the model performance" (§5.1.2). No tokens are dropped in any of the runs.

Evidence that the experts really specialize#

§4.5 runs three probes on the 2B model. They are the most interesting part of the paper.

1. Lower redundancy. For each token, mask the top-ranked routed experts and route to the next best. DeepSeekMoE's Pile loss rises faster than GShard×1.5's (Figure 4; GShard×1.5 is the baseline because both have equal loss with nothing masked). If knocking out an expert hurts more, that expert was harder to replace.
2. The shared expert cannot be replaced. Turn off the shared expert and activate one extra routed expert instead (same compute): Pile loss goes from 1.808 to 2.414.
3. More precise knowledge acquisition. At inference, activating only 4 routed experts (instead of 7) already matches GShard's Pile loss (Figure 5). A model trained from scratch with 1 shared + 3 of 63 routed experts active, which is half of GShard's activated expert parameters, still beats GShard (Figure 6).

Scaling to 16B and 145B#

(Table 7 and §4.1.3, §5.1.2, §7.1.) In the 16B and 145B models every FFN except the first layer is MoE, because the load balance converges especially slowly in layer 1. The 16B model uses a 100K BPE vocabulary and batches of 18M tokens (4.5K sequences × 4K length) over 106,449 steps, with step decay (×0.316 at 80% and at 90% of training). The 145B run uses a constant learning rate after warmup, which is another sign it was a partial run.

Results show a pattern: DeepSeekMoE 16B does well on language modeling and knowledge-heavy tasks (TriviaQA 64.8 vs 59.7 for DeepSeek 7B; NaturalQuestions 25.5 vs 22.2) and worse on multiple choice (MMLU 45.0 vs 48.2; CEval 40.6 vs 45.0). The authors tie this to FFN-as-memory: MoE adds FFN capacity, not attention capacity.

Alignment#

§6 fine-tunes the 16B model on 1.4M in-house SFT examples (8 epochs, batch 1024, constant LR \(10^{-5}\), 4K packed sequences). The same data is also used to fine-tune LLaMA2 7B and DeepSeek 7B. DeepSeekMoE Chat 16B is comparable to both (Table 5). It is ahead on code (MBPP 46.2 vs 39.0 for DeepSeek Chat 7B) and still behind on MMLU-style tasks, though by less than before SFT. This addresses an earlier concern that MoE models gain little from fine-tuning.

Key numbers#

How to read the paper#

• Abstract and §1 Introduction: Read closely. They state the two problems (hybridity, redundancy) and the two fixes. Figure 1 is a marketing plot; note what it shows (Open LLM Leaderboard average vs activated params) and move on.
• §2 Preliminaries: Skim if you know MoE. Learn the notation in Eqs. 3-5, since everything later reuses it.
• §3 DeepSeekMoE Architecture: Read closely. Figure 2 is the clearest picture of the idea: (a) top-2, (b) split into finer experts, (c) add a shared expert, all at equal cost. Work through the combinatorics example in §3.1 and the two balance losses in §3.3.
• §4.1 Experimental Setup: Skim. Note the 100B-token budget, 8K vocabulary and that no device-level loss is used at 2B.
• §4.2 Evaluations (Table 1): Read. This is the main comparison at equal compute.
• §4.3 Upper bound comparison (Table 2): Read closely. The "Dense×16" construction (16 shared experts of full FFN size) is how the authors define the MoE ceiling; understand it before accepting the "near upper bound" claim.
• §4.4 Ablation Studies (Figure 3): Read. The shared/routed ratio numbers are in the text, not the figure.
• §4.5 Analysis on Expert Specialization (Figures 4-6): Read closely. These are the paper's actual evidence for "specialization," as opposed to benchmark gains.
• §5 Scaling up to 16B (Tables 3-4): Read the setup and the observations under Table 3, especially the multiple-choice explanation. Skim the per-benchmark rows.
• §6 Alignment (Table 5): Skim. One takeaway: MoE fine-tunes fine.
• §7 DeepSeekMoE 145B (Table 6): Skim, keeping in mind it is a 245B-token preliminary run. The "half activated" column is worth a look.
• §8 Related Work, §9 Conclusion: Skip on first pass.
• Appendix A (Table 7): Use as a reference for configs. Appendix B (Tables 8-10): Skim; Table 10 is the 13B-total comparison where DeepSeekMoE beats GShard×1.5 more clearly. Appendix C (Figure 7): Skip.

Suggested order: §1 → Figure 2 and §3 → Tables 1-2 → §4.5 → Table 3 and its discussion → Table 6 → the rest as needed. About 60-75 minutes for a careful first read.

Check your understanding#

Limitations and open questions#

• Multiple-choice weakness. The authors acknowledge it and attribute it to limited attention capacity. That explanation is plausible but untested here: there is no ablation that adds attention parameters to the MoE model.
• The "upper bound" is a specific construction. Dense×16 is a model with 16 full-size FFNs all active, not a dense model with the standard attention:FFN ratio. "Near the upper bound" holds at 2B parameters and 100B tokens; the paper does not show it at larger scale.
• 145B is preliminary. It was trained on only 245B tokens with a constant learning rate, so the comparison with DeepSeek 67B is between two models at the same early checkpoint, not between finished models. The "maybe even 18.2%" claim rests on this run. The promised public release of the 145B model is not part of this paper.
• Specialization evidence is indirect. Figures 4-6 measure loss under interventions. They do not show what each expert specializes in (no per-domain routing analysis); ESFT later adds that kind of evidence.
• Inference speed is asserted, not measured in detail. "Nearly 2.5 times the inference speed of a 7B dense model" (§5.2.1) is stated "with appropriate operator optimizations," without a benchmark setup. FLOPs fall to 40%, but MoE inference is often limited by memory bandwidth, and all 16.4B parameters still need to be in memory.
• Balance loss tuning. The auxiliary losses trade quality for balance, and the factors change at each scale (0.01, 0.001, 0.003). The paper gives little guidance on choosing them. DeepSeek-V3 later moves to a mostly auxiliary-loss-free scheme to avoid this trade-off.
• Comparisons with LLaMA2 mix architecture and data. Some of the gap (Chinese, code, math) comes from DeepSeek's corpus, as the authors note. The clean architecture comparison is against DeepSeek 7B (Table 3).

Further reading#

• Series follow-ups: DeepSeek-V2 (DeepSeekMoE at 236B with device-limited routing), DeepSeek-Coder-V2, ESFT (fine-tuning only task-relevant experts), DeepSeek-VL2, DeepSeek-V3 (auxiliary-loss-free balancing).
• Dense baselines: DeepSeek LLM (arXiv 2401.02954).
• Prior MoE work: Shazeer et al., Outrageously Large Neural Networks; GShard; Switch Transformers; Hash Layers; DeepSpeed-MoE (origin of the shared-expert idea); Expert Choice routing; ST-MoE.
• FFNs as memory: Knowledge Neurons in Pretrained Transformers.
• Code and checkpoints: github.com/deepseek-ai/DeepSeek-MoE.