ESFT

TL;DR#

• ESFT (Expert-Specialized Fine-Tuning) adapts a Mixture-of-Experts (MoE) model to a task by training only the routed experts most relevant to that task. Shared experts, attention, routers and embeddings stay frozen.
• Relevance comes from a short probe: run about 32 samples of 4096 tokens from the task data through the model and score each expert by its average gate value (ESFT-Gate) or by how often it is selected (ESFT-Token). In each layer, keep the top experts until their scores add up to a threshold \(p\).
• On DeepSeek-V2-Lite (66 experts per layer), ESFT-Gate averages 50.2 across 8 downstream tasks vs 51.0 for full fine-tuning (FFT) and 44.9 for LoRA (Table 1). General-ability scores drop much less: 61.5 (ESFT-Token) vs 58.8 (FFT) and 59.1 (LoRA), from a vanilla 62.4 (Table 2).
• Usually only 2-15 of 66 experts per layer are trained (Figure 4). Trained-parameter storage falls from 28.6 GB (FFT) to 2.57-3.20 GB, and training time from 28.5 to about 20 minutes (§5.2).
• Fine granularity is required. When experts are grouped into coarser units to simulate a GShard-style model, ESFT's advantage shrinks and its cost rises (Figure 7).

Why it matters#

Parameter-efficient fine-tuning (PEFT) methods such as LoRA, adapters and prompt tuning were designed for dense models, where every token uses every weight. MoE models are different: a token uses only the experts its router picks. If those experts really specialize, a downstream task mostly uses a small, consistent subset of them, and that subset is a natural place to concentrate fine-tuning. The paper calls this a "selection dimension exclusive to the sparse MoE architecture."

The argument rests on DeepSeekMoE. That paper claimed fine-grained experts plus shared experts produce real specialization. ESFT tests the claim directly, showing per-task routing is concentrated and differs between tasks (§3.2), and turns it into a practical method. The paper is also a check on DeepSeekMoE's design: if experts were redundant, picking a few of them would not work.

Key idea In a fine-grained MoE, each task already routes to a small set of experts. Fine-tune only that set and leave the rest of the network alone. The task gets dedicated capacity, and the shared parameters that general ability depends on are untouched.

ESFT is built on the DeepSeek-V2 family (V2-Lite) and is DeepSeek's main 2024 study of post-training for MoE models. The same backbone family also underlies DeepSeek-Coder-V2 and DeepSeek-V3.

Background you need#

• Mixture-of-Experts routing. Each FFN is replaced by many expert FFNs. A router computes token-to-expert affinities and sends each token to the top-\(K\) experts, weighting their outputs by gate values. See DeepSeekMoE for the full formulation.
• Shared vs routed (non-shared) experts. In DeepSeekMoE-style layers a few shared experts process every token and routed experts are chosen per token. DeepSeek-V2-Lite has 66 experts per MoE layer: 2 shared and 64 routed, with 6 routed experts active per token (§2.2, Appendix B).
• Full fine-tuning (FFT) and catastrophic forgetting. Updating all weights on a narrow task often lowers performance on everything else. Here, "general ability" means benchmarks such as MMLU, TriviaQA and IFEval, measured after task fine-tuning.
• LoRA. Freeze the pretrained weights and learn a low-rank update \(W + BA\), with \(B \in \mathbb{R}^{d\times r}\), \(A \in \mathbb{R}^{r\times k}\) and small rank \(r\) (Hu et al. 2021). It is the main PEFT baseline here.
• Selective (structured) PEFT. Instead of adding parameters, train a chosen subset of existing ones (e.g. only biases or certain layers). ESFT belongs to this family, with "experts" as the unit of selection.

The core ideas#

Probing: routing is task-specific and concentrated#

Before proposing a method, §3.2 checks two facts on six tasks (math, code, intent recognition, summarization, legal judgment, translation):

1. Within a task, routing is concentrated. Compute each expert's normalized gate value: the sum of its gate values over all tokens divided by the total across experts. Sorted from high to low, a few experts take most of the mass, far above a uniform distribution (Figure 2).
2. Across tasks, the experts differ. Take the top-6 experts from two independent samples and count the overlap. For two samples from the same task the overlap is close to 6. For samples from different tasks it is close to 0 (Figure 3, averaged over layers).

Together these say each task has its own small set of experts. This is the premise of ESFT.

Step 1: sample a small probe set#

Randomly draw a subset \(D_s=\{(x_i,y_i)\}_{i=1}^{N_s}\) from the task's training data. The paper finds 32 concatenated samples of length \(L=4096\), about \(2^{17}\) tokens, are enough: at this size, two independent samples agree on nearly all of their top-6 experts for every task (Appendix C, Figure 8). The probe only needs forward passes, so it costs very little compared with training.

Step 2: score each expert's relevance#

Let \(g^l_{i,k}\) be the gate value of expert \(i\) at layer \(l\) for token \(k\), which is zero if the expert was not selected. Two scores are proposed (§3.3):

Average gate score (ESFT-Gate), the average affinity of expert \(i\) to the probe tokens:

Token selection ratio (ESFT-Token), the fraction of routing slots that go to expert \(i\):

Here \(L_j\) is the length of sample \(x_j\) and \(K\) is the number of routed experts selected per token (6 for V2-Lite). Each token fills exactly \(K\) slots, so the \(r^l_i\) in a layer add up to 1. \(r^l_i\) is then just expert \(i\)'s share of that layer's routing decisions. Gate-score totals are on a different scale, which is one reason the two variants use different thresholds.

Step 3: pick experts per layer by cumulative score#

For each MoE layer \(l\), choose the top-scored experts \(E^l_s\) whose scores together reach a threshold \(p \in (0,1]\):

where \(R^l_i\) is either \(g^l_i\) or \(r^l_i\). The defaults are \(p=0.1\) for ESFT-Gate and \(p=0.2\) for ESFT-Token (§4.3).

Worked example. Using token selection ratio with \(p=0.2\) on a layer with 64 routed experts: if routing were uniform, each expert would hold \(1/64 \approx 0.016\) of the slots and you would need 13 experts to reach 0.2. Because task routing is concentrated, a few top experts often reach 0.2 together, so only those few are trained in that layer. The threshold therefore adapts per layer and per task. Figure 4 shows the result: on average 2-15 of 66 experts per layer, or 5-15% for most tasks and layers. Narrower tasks such as Math and Translation select fewer experts, and middle layers tend to need fewer than early or late ones.

Step 4: fine-tune only the selected experts#

Training is otherwise normal. Routing is not restricted: during training and inference, tokens can still go to any expert. Only the selected experts in each layer receive gradient updates. Everything else stays frozen: unselected routed experts, shared experts, attention, routers, norms and embeddings. A customized model is stored as a set of replacement expert weights.

Why freezing shared parameters matters (§6.3)#

Table 3 varies which parameter groups are trained, using ESFT-Token with \(p=0.2\) for the "relevant" subset:

Two patterns stand out. Specialized ability rises with the number of trained parameters, from 47.4 at 450M to 51.0 at 15.7B. General ability falls as more shared parameters are trained, meaning shared experts and attention, which every input uses. Every configuration that trains the selected relevant subset of routed experts reaches at least 55.3 on average. Every other one reaches at most 54.9, including FFT with about 11× the trainable parameters. The authors therefore offer two recipes. To maximize task performance, train the relevant routed experts plus all shared parameters. For the best balance of task performance, general ability and cost, train only the relevant routed experts, which is standard ESFT.

Ablations: the scores matter, and so does granularity (§6.4)#

• Random experts. Replacing the selected experts with the same number of random experts per layer lowers the downstream average by 2.8 (Token) and 4.4 (Gate) points. Translation drops the most: -13.5 and -20.4 (Table 4).
• Coarser experts. To simulate coarser MoE models, the authors cluster V2-Lite's 64 routed experts into groups of 2 or 4 by co-activation similarity (greedy search, Appendix B). Each group is treated as one expert sharing an averaged score, and 1/8 of experts are selected per token. As group size grows, ESFT's MATH and GSM8K scores fall faster than FFT's while it trains more experts (Figure 7). Selective tuning only works when experts are narrow.

Key numbers#

How to read the paper#

The paper is a short ACL-format paper (about 9 pages plus appendices), so a full read is realistic.

• Abstract and §1 Introduction: Read. The three contributions (probing, the method, the granularity analysis) map directly onto §3.2, §3.3 and §6.4.
• §2 Related Work: Skim. §2.2 has the useful framing of coarse-grained (Mixtral, Grok: 2 of 8 experts) vs fine-grained (DeepSeekMoE, DeepSeek-V2) MoE.
• §3.1 Preliminaries: Skim if you have read DeepSeekMoE. It restates the shared + routed MoE equations.
• §3.2 Probing Task-Specific Expert Specialization: Read closely. Figures 2 and 3 are the evidence the method depends on.
• §3.3 ESFT: Read closely. Equations 6-8 are the whole method. Figure 1 compares FFT, LoRA and ESFT visually.
• §4 Experiment Setup: Read §4.1 to learn the two task types (enhancement: Math, Code; adaptation: Intent, Summary, Law, Translation). Note §4.3's details: the "vanilla" model is V2-Lite after an in-house alignment stage with math and code removed, and task data is mixed 1:1 with alignment data for all methods.
• §5.1 Benchmark Performance (Tables 1-2): Read closely. Table 2's ± values are standard deviations across the tasks trained on.
• §5.2 Computational Efficiency: Skim. The figure is Figure 5 (the text refers to it as Figure 6).
• §6.1 Expert selection (Figure 4): Read. It shows how many experts are actually trained, per layer and task.
• §6.2 Training resources (Figure 6): Read. ESFT vs LoRA across \(p\) and rank, a fairer comparison than a single LoRA setting.
• §6.3 Shared vs non-shared (Table 3): Read closely. This is the most informative analysis in the paper.
• §6.4 Ablations (Table 4, Figure 7): Read.
• Limitations: Read; it is short.
• Appendices: C (probe size) and H (Table 8, forgetting of math and code) are worth a look. F (Tables 9-10, effect of mixing alignment data) is useful if you plan to use ESFT. Skip A, B, D, E and G on first pass.

Suggested order: §1 → §3.2-3.3 → Tables 1-2 → Table 3 → Figure 7 and Table 4 → Figure 6 → Appendix H. About 40-50 minutes.

Check your understanding#

Limitations and open questions#

• One backbone. The authors note that ESFT was tested only on DeepSeek-V2-Lite because few other fine-grained MoE models were available. Whether it transfers to other fine-grained MoEs, or to much larger ones like DeepSeek-V2 and V3, is not shown here.
• Simulated coarse-grained models. The granularity comparison groups experts of one model instead of training separate coarse- and fine-grained models, as the authors acknowledge. Grouped experts trained as fine-grained units are not the same as experts trained coarse from scratch.
• GPT-4 as judge. Summary, Law and Translation are scored 0-10 by GPT-4 (gpt-4-1106-preview) against a reference. That is a noisy metric, and some differences in Table 1 are small.
• Short training. All runs are capped at 500 steps of 32×4096 tokens. Behavior with longer fine-tuning, larger datasets or RL-based post-training is not covered.
• Narrow LoRA setup in the main table. The main LoRA baseline uses rank 8. Figure 6 widens the comparison on Math across ranks, which helps, but other PEFT variants are not compared.
• Multi-task and serving questions. Each task gets its own expert subset. The paper does not study combining adaptations for several tasks in one model, or serving many customized expert sets together, though the per-expert storage format seems suited to it.
• Probe cost is small but not zero. The method assumes task data routes in a stable way. For very diverse tasks, a 32-sample probe may miss experts that are needed; Appendix C only tests six fairly homogeneous tasks.

Further reading#

• Series: DeepSeekMoE (the fine-grained and shared-expert architecture ESFT depends on), DeepSeek-V2 (the V2-Lite backbone), DeepSeek-Coder-V2, DeepSeek-V3.
• PEFT background: LoRA; Adapters (Houlsby et al. 2019); Prefix-Tuning; P-Tuning v2.
• MoE background: GShard, Switch Transformers, Mixtral of Experts.
• Code: github.com/deepseek-ai/ESFT.