DeepSeek-Coder-V2

TL;DR#

• DeepSeek-Coder-V2 is not trained from scratch. It continues pre-training an intermediate DeepSeek-V2 checkpoint (already trained on 4.2T tokens) with 6T more tokens, for 10.2T in total (§3.3). There are two sizes: 236B total / 21B active parameters (DeepSeek-V2 shape) and 16B / 2.4B ("Lite", DeepSeek-V2-Lite shape) (Table 2).
• The new data is 60% code, 10% math and 30% natural language. The code corpus has 1,170B tokens covering 338 languages (up from 86), mined partly with the DeepSeekMath fastText pipeline. The math corpus has 221B tokens, about double DeepSeekMath's (§1, §2).
• Context grows from 16K to 128K using YaRN plus two stages of long-sequence training (§3.4). Alignment is SFT followed by GRPO with a learned reward model trained on compiler and test-case feedback, which beat using the raw pass/fail signal (§3.5, Figure 3).
• Headline results for the 236B Instruct model: 90.2% HumanEval, 76.2% MBPP+, 43.4% LiveCodeBench (tied with GPT-4o), 73.7% Aider (best in the table), 12.7% SWE-bench (the first open model above 10%), 75.7% MATH against GPT-4o's 76.6%, and 4/30 on AIME 2024 (Tables 3, 4, 7 and 9).
• General ability is kept: 79.2% MMLU against DeepSeek-V2 Chat's 78.1%, and a much higher Arena-Hard (65.0 vs 41.6). Knowledge-heavy QA such as TriviaQA drops (Table 10).

Why it matters#

DeepSeek-Coder ended with a lesson: the best code models come from strong general LLMs, and a dense model trained from scratch on code runs out of room. By mid-2024 open code models (CodeLlama, StarCoder2, Codestral, DeepSeek-Coder 33B) were good but clearly behind GPT-4-Turbo, Claude 3 Opus and Gemini 1.5 Pro, especially on competitive programming and real-repo repair.

Coder-V2 combines three earlier pieces of the series. The architecture comes from DeepSeek-V2: a DeepSeekMoE-style sparse model with Multi-head Latent Attention, so a 236B model costs about 21B parameters of compute per token. The data machinery comes from DeepSeekMath: iterative fastText recall from Common Crawl, now applied to code as well. The RL algorithm is GRPO, also from DeepSeekMath. Branching from an intermediate general checkpoint, rather than a finished chat model, lets the code and math data shape the model while 30% general text in the mix limits forgetting.

Key idea Don't build a code model from scratch. Take a strong general MoE model partway through pre-training, feed it a code- and math-heavy mix for trillions more tokens, and align it with GRPO against a reward model trained on test outcomes.

The result is the first open model to roughly match GPT-4-Turbo-class systems on standard code and math benchmarks. The Lite model, with 2.4B active parameters, is a fast option for FIM-style completion. The authors' own closing note is that instruction following, not raw coding skill, is the remaining gap. That shows up in SWE-bench and points toward the general-purpose DeepSeek-V3.

Background you need#

• Mixture-of-Experts (MoE). Each feed-forward layer is split into many experts, and a router sends each token to a few of them. Total parameters (capacity) and active parameters (compute per token) separate: here 236B vs 21B. See DeepSeekMoE and DeepSeek-V2.
• Continued pre-training. Resuming next-token training of an existing checkpoint on a different data mix. The risk is catastrophic forgetting, and mixing general data back in is the usual guard against it.
• Fill-in-the-Middle (FIM). Training on documents rearranged as prefix, suffix, middle, so the model can infill at the cursor. See the DeepSeek-Coder page.
• YaRN. A RoPE rescaling method that interpolates rotary frequencies non-uniformly, with an attention-temperature tweak, to extend the context window with little extra training. See Peng et al. 2023.
• GRPO. A PPO variant without a critic. It samples several answers per prompt and uses each answer's reward z-score within the group as its advantage. See DeepSeekMath.
• The benchmarks. HumanEval and MBPP+ test single functions. LiveCodeBench and USACO are competitive programming, collected after the training cutoff. SWE-bench resolves real GitHub issues with a patch. Aider tests file edits. RepoBench tests repo-level line completion. CRUXEval tests predicting a function's input or output.

The core ideas#

1. A bigger, mined code corpus#

The GitHub part reuses DeepSeek-Coder's rule filters and near-dedup on repos created before November 2023. That yields 821B tokens of code in 338 languages plus 185B tokens of code-related text such as Markdown and issues (§2).

The new part applies the DeepSeekMath recall loop to code:

• Seeds: StackOverflow, library documentation (for example PyTorch docs) and math StackExchange.
• Classifier: a fastText classifier recalls similar pages from Common Crawl. Tokenization uses the DeepSeek-V2 BPE tokenizer instead of whitespace splitting, which "significantly improves the recall accuracy" for languages such as Chinese.
• Domain expansion: domains where more than 10% of pages were recalled are marked code- or math-related. Their relevant URL paths are annotated, and uncollected pages are added to the seed.
• Yield: three iterations produce 70B code-related web tokens and 221B math tokens.
• The same loop run on GitHub: seeded with hand-picked high-quality source code (for example, files with detailed descriptions), two iterations add 94B tokens of source code.

In total: $821 + 185 + 70 + 94 = 1{,}170$B code-related tokens.

Does the new corpus help? A 1B ablation (Table 1) trains the same small model on the old and new code corpora. At 1T tokens, HumanEval rises from 30.5% to 36.0% and MBPP from 44.6% to 49.0%. At 2T tokens on the new corpus, the scores reach 37.2% and 54.0%. (The introduction's "+6.7% / +9.4%" compares the 2T run with the 1T baseline.)

The full mix for the 6T continued tokens is 60% code, 10% math, 30% natural language, and the language part is sampled from DeepSeek-V2's own corpus (§2). A back-of-the-envelope check: 60% of 6T is 3.6T code tokens, about three times the 1,170B code corpus. The paper doesn't state epoch counts.

2. Architecture: DeepSeek-V2, with one change#

The 236B and 16B models use the same hyperparameters as DeepSeek-V2 and DeepSeek-V2-Lite (§3.2). Read the DeepSeek-V2 page for MLA and the MoE layout. The paper reports one deviation: training was unstable, with gradient spikes, which the authors "attributed to the exponential normalization technique," so they "reverted to the conventional normalization method." The paper doesn't explain what exponential normalization refers to.

Training follows DeepSeek-V2: AdamW with \(\beta_1 = 0.9\), \(\beta_2 = 0.95\), weight decay 0.1, and batch size and learning rate per V2. The schedule is cosine decay with 2000 warmup steps, down to 10% of the peak (§3.3). Note the change from the multi-step schedule of earlier DeepSeek papers.

(Table 2.)

3. FIM only for the small model#

Both models train with next-token prediction. Only the 16B Lite model also uses FIM at rate 0.5 in PSM mode, the same recipe as DeepSeek-Coder, applied per document before packing (§3.1):

<｜fim_begin｜> f_pre <｜fim_hole｜> f_suf <｜fim_end｜> f_middle <|eos_token|>

The split follows the intended uses: the Lite model, with 2.4B active parameters, is the fast completion model for editors, and the 236B model is for chat and generation. The Lite base reaches a mean of 86.4% on single-line infilling, tied with DeepSeek-Coder-Base 33B and ahead of Codestral 22B at 83.0% (Table 6). On RepoBench (Table 5) it matches DeepSeek-Coder-Base 33B in Python (38.9% vs 39.1%) but trails Codestral (46.1%). The authors argue that its much smaller active-parameter count makes it faster in practice.

4. Long context: YaRN plus two stages#

Following DeepSeek-V2, the context is extended with YaRN using scale \(s = 40\), \(\alpha = 1\), \(\beta = 32\) (§3.4). Training then continues in two stages, with long-context data upsampled:

The Needle-in-a-Haystack test (Figure 2) passes across 1K to 128K tokens at all depths. This is the evidence behind the 128K claim, and it is a retrieval test, not a test of reasoning over long code.

5. Alignment: SFT, then GRPO with a learned code reward#

SFT (§3.5.1) uses about 20K code and 30K math instructions from DeepSeek-Coder and DeepSeekMath, plus general instructions sampled from DeepSeek-V2's data, for a 300M-token dataset. Training runs for 1B tokens at a batch of 1M tokens, with a cosine schedule, 100 warmup steps and a learning rate of 5e-6.

RL (§3.5.2) uses about 40K prompts for code and math. Each code prompt comes with test cases.

• Math rewards: preference labels come from ground-truth answers.
• Code rewards: the obvious signal is the compiler's 0/1 (all tests pass or not). But many prompts have few tests, so pass/fail is noisy: a wrong program can pass weak tests. The authors train a reward model on the compiler-labelled data and use its scores during RL. On in-house LeetCode and LeetCode-zh sets, RL with the reward-model signal beats RL with the raw compiler signal (Figure 3). All later experiments use the reward model.
• Algorithm: GRPO, the same as DeepSeek-V2's alignment. The paper's stated reason is cost: there is no critic to maintain. For the mechanics, see the DeepSeekMath page. For each prompt, sample a group of answers, score them with the reward model, z-score the scores within the group and use those as advantages.

Why a learned reward model can beat ground truth: the reward model is trained on many programs and test outcomes, so it can generalize past a single prompt's thin test suite and give a graded signal instead of a binary one. The paper shows this empirically. It doesn't analyse it.

6. Where it lands#

Instruct models, greedy decoding:

The pattern: Coder-V2 matches or beats GPT-4-Turbo on self-contained generation (HumanEval, MBPP+) and on edit-style tasks (Aider). It trails on code reasoning (CRUXEval) and on agent-like, multi-file repair (SWE-bench). The authors attribute the reasoning gap to having only 21B active parameters, and the SWE-bench gap to weaker instruction following (§4.4, §5).

Against its own parent, DeepSeek-V2 Chat (Table 10), Coder-V2 Instruct gains on reasoning-heavy benchmarks: BBH 79.7 → 83.9, Arena-Hard 41.6 → 65.0. It loses on knowledge recall (TriviaQA 86.7 → 82.3, NaturalQuestions 53.4 → 47.5) and slightly on chat-style evaluations (MT-Bench 8.97 → 8.77). These are the expected effects of shifting the data mix toward code and math.

Key numbers#

How to read the paper#

• Abstract, §1 Introduction, §1.1 Contributions: Read. The introduction's second and third paragraphs are a compact summary of the whole recipe.
• §1.2 Summary of Evaluations: Skim. It is the headline numbers.
• §2 Data Collection: Read closely. Follow how 821B + 185B + 70B + 94B adds up to 1,170B and how the DeepSeekMath pipeline was adapted (BPE tokenization for fastText, GitHub seeding). Table 1 is the only data ablation.
• §3.1 Training Strategy: Read. It is short. Note that FIM is used only for the 16B model.
• §3.2 Model Architecture: Read the two sentences about the normalization change. Everything else is in DeepSeek-V2.
• §3.3 Training Hyper-Parameters: Skim. Table 2 and the "intermediate checkpoint at 4.2T" detail are what matter.
• §3.4 Long Context Extension: Skim. It gives the YaRN settings, the two-stage table and Figure 2.
• §3.5 Alignment: Read closely, especially §3.5.2 on reward modeling and Figure 3. This is the only new RL finding.
• §4.1 Code Generation: Read Tables 3 and 4.
• §4.2 Code Completion: Skim Tables 5 and 6, unless you care about IDE completion.
• §4.3 Code Fixing: Read Table 7. It shows the SWE-bench gap.
• §4.4 Code Understanding and Reasoning: Skim Table 8.
• §4.5 Mathematical Reasoning: Read Table 9.
• §4.6 General Natural Language: Read Table 10 and the two paragraphs after it. They show what was gained and lost relative to DeepSeek-V2.
• §5 Conclusion: Read the second paragraph, the instruction-following limitation.
• Appendix A (language list): Skip.

Suggested order: §1 → §2 → §3.1, §3.2 → §3.5 → Tables 3, 4, 7, 9, 10 → §5. About 45 minutes, assuming you have read DeepSeek-V2 and DeepSeekMath.

Check your understanding#

Limitations and open questions#

• Instruction following is the authors' main stated weakness, and it shows up in agent-like tasks such as SWE-bench (§5).
• Ablations are few. There is one 1B data ablation (Table 1) and one RL-signal comparison (Figure 3). Nothing isolates the effect of the 60/10/30 mix, the intermediate-checkpoint choice, or GRPO compared with SFT alone.
• The normalization change is unexplained. "Exponential normalization" is not defined, and there are no curves showing the instability.
• Long-context evidence is mostly Needle-in-a-Haystack, which tests retrieval, not reasoning over 128K tokens of code. RepoBench is capped at 16K.
• Contamination is handled by date filtering. LiveCodeBench (Dec 2023 to Jun 2024) and RepoBench (December 2023 only) are filtered against the November 2023 cutoff. Classic benchmarks such as HumanEval have been public for years, and the paper doesn't describe n-gram decontamination for the new web data.
• Reward-model details are sparse. The paper doesn't give the reward model's architecture, size, training data volume or accuracy.
• Small text issues. §4.1 says "seven additional languages" but lists thirteen. §4.4 describes CRUXEval-I and CRUXEval-O the opposite way from the original benchmark, where -I means predicting the input and -O predicting the output. Table 6 FIM numbers aren't comparable with DeepSeek-Coder's Table 6, because only the first generated line is now scored (footnote 5).

Further reading#

• Series: DeepSeek-Coder (data filters, FIM recipe, v1.5 lesson), DeepSeekMath (fastText recall pipeline, GRPO), DeepSeek-V2 (architecture, MLA, base checkpoint), DeepSeekMoE (expert design), ESFT (fine-tuning only task-relevant experts of DeepSeek-V2-Lite-class models), DeepSeek-V3 (next general model).
• YaRN context extension: Peng et al., arXiv:2309.00071.
• Benchmarks: LiveCodeBench arXiv:2403.07974, SWE-bench arXiv:2310.06770, EvalPlus / MBPP+ arXiv:2305.01210, RepoBench arXiv:2306.03091, CRUXEval arXiv:2401.03065.
• StarCoder2 (main open baseline family): arXiv:2402.19173.
• Code and models: github.com/deepseek-ai/DeepSeek-Coder-V2.