DeepSeek-Coder

TL;DR#

• DeepSeek-Coder is a series of decoder-only code models at 1.3B, 6.7B and 33B parameters, each trained from scratch on 2 trillion tokens. The data is 87% source code in 87 programming languages, 10% code-related English (GitHub Markdown, StackExchange) and 3% Chinese text (§2).
• The main data idea is repository-level ordering. Files in a repo are topologically sorted by their import/include dependencies and joined into one training sample, so a file's dependencies appear before it in context (§2.2, Algorithm 1). Deduplication also works on whole repos, not single files (§2.3).
• Training mixes next-token prediction with Fill-in-the-Middle (FIM) at a 50% rate in PSM order. The context is then extended to 16K by rescaling RoPE (§3.1.2, §3.6).
• Headline results: DeepSeek-Coder-Base 33B averages 50.3% on multilingual HumanEval and 66.0% on MBPP, against 41.0% and 55.2% for CodeLlama-Base 34B. The 6.7B model (44.7% / 60.6%) already beats CodeLlama 34B (Table 3). DeepSeek-Coder-Instruct 33B scores 79.3% on Python HumanEval and 27.8% on a fresh LeetCode Contest set, ahead of GPT-3.5-Turbo's 23.3% (Tables 3 and 5).
• A follow-up, DeepSeek-Coder-v1.5 7B, continues training from the general DeepSeek-LLM 7B instead of starting from scratch. It gives up a little coding score for much better math and language results (§5, Table 10). That checkpoint became the starting point for DeepSeekMath.

Why it matters#

In late 2023 the best open code models were StarCoder (15B, trained on The Stack) and CodeLlama (Llama 2 with 500B more tokens of code). Both still trailed GPT-3.5 and GPT-4 by a wide margin. Both also trained on code mostly one file at a time, which ignores how real code works: a function in train.py calls classes defined in model.py, and a file-level model never sees the two together.

DeepSeek-Coder's answer is mostly about data, not architecture. The model is the standard DeepSeek LLM Transformer. What changes is how the corpus is built (filtered, dependency-ordered, repo-deduplicated, decontaminated) and the training objective (FIM, for editor-style completion). The paper's main evidence that the data matters is that a 6.7B model matches or beats a 34B model on most benchmarks.

Key idea Train on whole repositories, with files ordered so that dependencies come first. The model then learns to use cross-file context, the way an engineer reads a codebase.

This paper starts the code thread of DeepSeek's 2024 work. Its v1.5 checkpoint (code continued from a general LLM) was the starting point for DeepSeekMath. Its data pipeline, FIM recipe and instruction data carry over to DeepSeek-Coder-V2, which moves to the DeepSeek-V2 MoE architecture. The paper's closing lesson, that the best code models are built on strong general LLMs, is exactly what Coder-V2 does.

Background you need#

• Decoder-only Transformer with RoPE. The standard GPT-style language model. Rotary Position Embedding (RoPE) encodes position by rotating query and key vectors by position-dependent angles. Changing RoPE's base frequency or scaling factor is the usual cheap way to stretch the context window. See RoFormer.
• Grouped-Query Attention (GQA). Several query heads share one key/value head. This shrinks the KV cache and speeds up inference. The 33B model uses groups of 8 (§3.3). See GQA.
• Fill-in-the-Middle (FIM). Split a document into prefix, middle and suffix, then move the middle to the end, so a left-to-right model learns to generate a span given the text on both sides. This is what an IDE needs when your cursor is inside a function. See Bavarian et al. 2022.
• pass@1. The fraction of problems where one generated program passes all the hidden unit tests. With greedy decoding, as used here, that one sample is deterministic.
• Near-deduplication. Removing documents that are almost the same, typically with MinHash. This matters a lot on GitHub, which is full of forks and vendored copies. See The Stack.
• Benchmark contamination. Test problems such as HumanEval leaking into the training data through GitHub. This inflates scores, so the paper filters by n-gram overlap (§2.4).

The core ideas#

1. A filtered, decontaminated GitHub corpus#

The pipeline (Figure 2) runs crawl → rule filtering → dependency parsing → repo-level dedup → quality screening.

• Crawl: public GitHub repos created before February 2023, restricted to 87 languages (§2.1).
• Rule filters borrowed from StarCoder: drop files with average line length above 100 characters or any line above 1000 characters, files with less than 25% alphabetic characters, and XML-headed files (except XSLT). HTML needs at least 20% visible text and at least 100 characters. JSON and YAML must be 50 to 5000 characters long. These rules alone cut the data to 32.8% of its raw size (§2.1).
• Quality screening: a compiler and a quality model, plus heuristics, remove code with syntax errors, poor readability or low modularity (§2.4).
• Decontamination: any piece of code that shares a 10-gram with HumanEval, MBPP, GSM8K or MATH is removed. For test strings between 3 and 10 grams long, exact match is used instead (§2.4).

The cleaned code is 798 GB in 603M files (Table 1). Java (18.63%), Python (15.12%) and C++ (11.39%) dominate. Many niche languages sit around 0.1% each.

2. Repository-level ordering with a cycle-tolerant topological sort#

The model should see utils.py before the file that does from utils import .... The authors extract dependencies between files with regular expressions (import in Python, using in C#, #include in C) and build a graph with an edge B→A whenever A depends on B (Algorithm 1).

Plain topological sort needs a DAG, which means repeatedly taking a node with in-degree 0. Real repos have import cycles, so a strict sort would get stuck. Instead, the algorithm picks the node with the minimum in-degree among the files not yet placed, appends it to the order, and decrements its neighbours' in-degrees. On a DAG this reduces to ordinary topological sort. On a cycle it still makes progress.

Each disconnected subgraph (a group of files that reference each other) becomes one training sample: its files are concatenated in sorted order, and each file starts with a comment giving its path (§2.2). For example, a repo with config.py, model.py (imports config) and train.py (imports both) is serialized as # config.py ..., then # model.py ..., then # train.py ....

Repo-level dedup follows from this. Near-dedup is run on the concatenated repo as one unit, so the method never deletes a single file out of the middle of a repo and breaks its dependency structure (§2.3).

Does this help? Table 7 runs the cross-file benchmark CrossCodeEval with BM25 retrieval of cross-file context. The 6.7B model trained without repo-level data ("w/o Repo Pre-training") does worse in Java (17.72% → 16.64% exact match), TypeScript (14.03% → 13.23%) and C# (16.23% → 14.48%). Python is roughly unchanged (16.14% vs 16.02%). The gains are real but modest.

3. Fill-in-the-Middle, and how much of it#

Each document is split into \(f_{pre}\), \(f_{middle}\), \(f_{suf}\) and rearranged in PSM (prefix-suffix-middle) order with three sentinel tokens (§3.1.2):

<｜fim_start｜> f_pre <｜fim_hole｜> f_suf <｜fim_end｜> f_middle <|eos_token|>

The model reads the prefix and the suffix, then generates the middle with the ordinary next-token loss. No architecture change is needed. FIM is applied per document before packing, as in Bavarian et al.

How often to apply FIM is a trade-off. The ablation (1.3B model, Python subset, Figure 3) compares 0%, 50% and 100% FIM rates, plus a 50% rate of T5-style Masked Span Prediction (MSP):

• 100% FIM scores best on HumanEval-FIM (single-line infilling) but worst on ordinary left-to-right completion (HumanEval, MBPP).
• 50% PSM keeps most of the infilling ability, keeps completion strong, and beats MSP.

So the final models use a FIM rate of 0.5 in PSM mode. On single-line infilling (Table 6), the base models reach a mean line exact-match of 70.4% (1.3B), 80.7% (6.7B) and 81.2% (33B). The paper recommends the 6.7B model for code-completion tools as the best balance of speed and accuracy (§4.2).

4. Architecture and optimization (unchanged from DeepSeek LLM)#

(Table 2.) All three models use SwiGLU, RoPE, FlashAttention-2, a 32K BPE vocabulary (§3.2) and AdamW with \(\beta_1=0.9\), \(\beta_2=0.95\). The learning-rate schedule is the DeepSeek LLM multi-step one: 2000 warmup steps, then three stages, each at \(1/\sqrt{10}\) of the previous rate, ending at 10% of the peak (§3.4). Training ran on the HAI-LLM framework on A100 and H800 clusters (§3.5).

5. Stretching the context to 16K#

After pre-training, the RoPE scaling factor is raised linearly from 1 to 4 and the base frequency from 10000 to 100000. Training continues for 1000 steps at a 16K sequence length with batch size 512 (§3.6). In theory the model could then handle 64K tokens, but the authors report reliable behaviour only up to 16K.

6. Instruction tuning and the v1.5 detour#

DeepSeek-Coder-Instruct is supervised fine-tuning on Alpaca-format instruction data, with an <|EOT|> token ending each turn. It uses a cosine schedule, a learning rate of 1e-5, a batch of 4M tokens and 2B tokens in total (§3.7). There is no RLHF.

DeepSeek-Coder-v1.5 7B (§5) tests the opposite design. It continues pre-training the general DeepSeek-LLM-7B Base for 2T tokens on a mix of 70% code, 10% Markdown/StackExchange, 7% code-related natural language, 7% math-related natural language and 6% Chinese-English text (Table 9). It uses next-token prediction only, with a 4K context and no FIM. Compared with the 6.7B Coder-Base (Table 10), HumanEval drops slightly (44.7% → 43.2%). GSM8K jumps from 43.2% to 62.4%, MMLU from 36.6% to 49.1%, and HellaSwag from 53.8% to 69.9%. This is the evidence for the conclusion that "the most effective code-focused LLMs are those built upon robust general LLMs."

Key numbers#

How to read the paper#

• Abstract and §1 Introduction: Skim. The contributions list is a fair summary.
• §2 Data Collection: Read closely. This is the actual contribution. Study Algorithm 1 in §2.2 and the minimum-in-degree trick, then §2.3 (repo-level dedup) and §2.4 (decontamination rules). Glance at Table 1 for the language mix. You don't need to memorize it.
• §3.1 Training Strategy: Read closely. Figure 3 and the paragraph after it explain the FIM-rate trade-off and why they chose 0.5 PSM.
• §3.2 to §3.5 (Tokenizer, Architecture, Optimization, Environments): Skim. Table 2 is the only part you need. Everything else is inherited from DeepSeek LLM.
• §3.6 Long Context: Read. It is short, and it is a clean example of RoPE rescaling.
• §3.7 Instruction Tuning: Skim. It is plain SFT with few details about the data.
• §4.1 Code Generation: Read Tables 3 and 5. Note the CoT-prompting result and the contamination caveat for LeetCode. Skim Table 4 (DS-1000).
• §4.2 FIM Completion: Skim Table 6.
• §4.3 Cross-File Completion: Read Table 7. Its last row is the only ablation of repo-level pre-training in the paper.
• §4.4 Program-based Math: Skim Table 8.
• §5 Continue Pre-Training From General LLM: Read closely. Table 10 motivates DeepSeekMath and Coder-V2.
• §6 Conclusion: Skim.
• Appendix A (chat cases) and B (training curves): Skip on first pass.

Suggested order: §1 → §2 → §3.1 → Table 2 → §3.6 → Tables 3, 5, 7 → §5. About 45 to 60 minutes.

Check your understanding#

Limitations and open questions#

• The repo-level ablation is thin. It is a single row in Table 7 at 6.7B, measured with retrieval. The gains are about 1 to 2 points in three languages and none in Python. It is unclear how much of the benefit comes from ordering versus simply having longer, related context.
• Dependency parsing is regex-based (import, using, include). It will miss dynamic imports, build-system relationships and many languages' module systems.
• Little detail on the quality model used in §2.4: no architecture, training data or thresholds.
• Long context is only partly validated. The authors say outputs are most reliable within 16K, although RoPE scaling theoretically allows 64K. There is no long-context benchmark.
• Instruction tuning is lightly described. The data source and size are not given beyond "2B tokens," and there is no RL stage.
• Contamination remains possible on LeetCode, as the authors acknowledge.
• Small inconsistencies. The conclusion says v1.5 was trained on "2 billion tokens," while §5 says 2 trillion. Figure 4's text says "Instruct 34B" for the 33B model. Table 6 labels the models "1B/7B."
• The FIM ablation is small: a 1.3B model on Python only, read from training curves.

Further reading#

• Series follow-ups: DeepSeekMath, which starts from DeepSeek-Coder-v1.5 7B, and DeepSeek-Coder-V2, which reuses this pipeline on a much larger corpus and moves to MoE.
• Foundation: DeepSeek LLM for the architecture, learning-rate schedule and scaling-law hyperparameters.
• Fill-in-the-Middle: Bavarian et al., Efficient Training of Language Models to Fill in the Middle, arXiv:2207.14255.
• StarCoder (filtering rules, FIM baseline): arXiv:2305.06161.
• CodeLlama (main open baseline): arXiv:2308.12950.
• Positional interpolation for RoPE: Chen et al., arXiv:2306.15595.
• CrossCodeEval: Ding et al., arXiv:2310.11248.
• Code and models: github.com/deepseek-ai/DeepSeek-Coder, including the LeetCode Contest benchmark under Evaluation/LeetCode.