DeepSeek LLM

TL;DR#

• DeepSeek trains two dense, LLaMA-style models from scratch, 7B and 67B, on 2 trillion tokens of mostly English and Chinese text (§2, Table 2), then aligns them with SFT and DPO to make DeepSeek Chat.
• Before training, they fit scaling laws for hyperparameters: the near-optimal learning rate and batch size as power laws of the compute budget \(C\) (Eq. 1). The fitted formulas land on the hyperparameters actually used for 7B and 67B.
• They measure model size as non-embedding FLOPs per token \(M\) instead of parameter count, so that \(C = MD\) exactly includes attention cost (Eq. 2). This changes the fitted model/data allocation, especially at small scale.
• They find that the optimal split of compute between model and data depends on data quality: cleaner data favors spending more compute on a bigger model (Table 4). That may explain why OpenAI (Kaplan) and Chinchilla reached different exponents.
• DeepSeek LLM 67B Base beats LLaMA-2 70B on most benchmarks, by wide margins on code and math (HumanEval 42.7 vs 28.7, GSM8K 63.4 vs 58.4, MATH 18.7 vs 13.5; Table 5). The 67B Chat DPO model scores 8.76 on MT-Bench, above GPT-3.5-turbo's 8.39 (Table 8).
• The paper also reports useful negative results: adding 20M multiple-choice questions inflates MMLU/C-Eval without helping generative tasks (Table 13), so they leave that data out.

Why it matters#

In late 2023 the open-model landscape was organized around LLaMA-2: teams trained fixed sizes (7B, 13B, 34B, 70B) and compared at matched size. Few published how they chose model size, data size, learning rate or batch size. The two reference scaling-law papers disagreed. Kaplan et al. (2020) said to put most new compute into parameters (\(N_{opt} \propto C^{0.73}\)), and Chinchilla (Hoffmann et al., 2022) said to split it about evenly (\(N_{opt} \propto C^{0.49}\)). Neither fully described how hyperparameters were tuned at each scale, so you could not tell whether their small models were trained well.

This paper is DeepSeek's answer to "how should we scale, and how do we know the big run will work?" It fits laws on runs from \(10^{17}\) to \(3 \times 10^{20}\) FLOPs and predicts the final loss of the 7B and 67B models, which use about 1000x more compute (§3.2, Figure 5). It also explains its engineering choices, such as a multi-step learning-rate schedule (instead of cosine) that lets a finished run be extended without starting over.

Key idea Get the small runs right before you extrapolate. First find how the optimal learning rate and batch size scale with compute, then fit the model/data allocation, and measure model size in FLOPs per token rather than parameters. The result also depends on your data, so fit on your own corpus.

Later DeepSeek papers build directly on this one. DeepSeekMoE uses the same 2T-token corpus and compares against DeepSeek 7B and 67B. DeepSeek-Coder and DeepSeekMath promised in §6 continue the code and math work. DeepSeek-V2 reuses this paper's tokenizer, initialization, optimizer settings, multi-step LR schedule and evaluation framework, and calls DeepSeek 67B "our previous release" in all its comparisons. The "longtermism" in the title refers to this habit of keeping the infrastructure reusable.

Background you need#

• Scaling laws. Empirical power laws relating loss to compute \(C\), parameters \(N\) and tokens \(D\). With \(C \approx 6ND\), the question is how to pick \(N\) and \(D\) for a fixed \(C\). See Kaplan et al. 2020 and Chinchilla, Hoffmann et al. 2022.
• IsoFLOP profiles. Chinchilla's method: fix a compute budget, train about 10 models of different sizes on correspondingly different token counts, and find the size with the lowest loss. Repeat across budgets and fit a power law through the minima.
• LLaMA architecture. A pre-norm decoder-only transformer with RMSNorm, a SwiGLU feed-forward block and rotary position embeddings (RoPE). See LLaMA.
• Grouped-Query Attention (GQA). Several query heads share one key/value head, which shrinks the KV cache kept during generation. See Ainslie et al. 2023. DeepSeek-V2 replaces it with MLA.
• DPO. Direct Preference Optimization trains on pairs of preferred and rejected responses with a classification-style loss and no separate reward model or RL loop. See Rafailov et al. 2023.
• Bits-per-byte (BPB). Loss normalized by raw bytes rather than tokens, so models with different tokenizers can be compared. The scaling-law plots use it.

The core ideas#

1. Data and tokenizer#

The data pipeline has three stages: deduplication, filtering, remixing (§2.1). The notable detail is how far deduplication goes. Deduplicating across 91 Common Crawl dumps removes 89.8% of documents, against 22.2% within a single dump (Table 1). Much of the web repeats from crawl to crawl, so deduplicating each dump separately leaves a lot of repeated text.

The tokenizer is byte-level BPE with 100,000 regular tokens plus 15 special tokens, padded to a vocabulary of 102,400 for efficiency and future special tokens (§2.1). Numbers are split into single digits, as in LLaMA. Pre-tokenization keeps newlines, punctuation and CJK characters from merging with other character classes.

2. Architecture: LLaMA, but deeper#

(Table 2.) The micro design is LLaMA's: pre-norm RMSNorm, SwiGLU with an FFN intermediate size of \(\tfrac{8}{3} d_{model}\), and RoPE. The 67B model uses GQA to make inference cheaper.

The macro design is different. Other GQA models usually add back the lost attention parameters by widening the FFN. DeepSeek instead spends them on depth: 95 layers, against 80 for LLaMA-2 70B. The layer counts were also chosen to split evenly across pipeline-parallel stages (§2.2).

3. Multi-step learning-rate schedule#

Training uses AdamW (\(\beta_1 = 0.9\), \(\beta_2 = 0.95\), weight decay 0.1), initialization std 0.006, and gradient clipping at 1.0 (§2.3). The schedule is a step function rather than cosine:

• linear warmup over 2000 steps to the peak LR;
• hold at the peak until 80% of training tokens;
• drop to 31.6% of peak (that is \(1/\sqrt{10}\)) until 90%;
• drop to 10% of peak for the final 10%.

At 1.6B parameters and 100B tokens, final loss matches cosine (Figure 1a). The practical benefit is reuse. With cosine, the LR at every step depends on the total length of the run, so a longer run has to start over. With a step schedule, the first stage (constant at peak LR) is the same for any run length, so you can branch from it. The scaling-law experiments take advantage of this: models for budgets from \(10^{17}\) to \(2 \times 10^{19}\) share their first stage (§3.1). Figure 1b shows other stage proportions do slightly better. The authors keep 80/10/10 as a balance between reusability and final loss.

4. Scaling laws for hyperparameters#

They first grid-search batch size and LR at \(10^{17}\) FLOPs and find a wide flat basin of near-optimal settings (Figure 2a). Any setting whose loss is within 0.25% of the minimum counts as "near-optimal", and power laws are fitted to those points (Figure 3):

Here \(\eta_{opt}\) is the peak learning rate, \(B_{opt}\) is the batch size in tokens, and \(C\) is compute in FLOPs. More compute means larger batches and a smaller learning rate.

Worked check (our arithmetic, not the paper's). For the 7B model, Eq. 2 below gives \(M \approx 72 \cdot 30 \cdot 4096^2 + 12 \cdot 30 \cdot 4096 \cdot 4096 \approx 4.2 \times 10^{10}\) FLOPs/token. With \(D = 2 \times 10^{12}\) tokens, \(C \approx 8.5 \times 10^{22}\). Then \(\eta_{opt} \approx 0.3118 \cdot 10^{-2.87} \approx 4.2 \times 10^{-4}\), which is the LR in Table 2. \(B_{opt} \approx 9.2\text{M}\) tokens, about 2250 sequences of 4096 tokens, close to the 2304 used.

They validated the formulas at \(10^{20}\) FLOPs (Figure 2b). The fitted points fall in the middle of the good region. They also note the formulas ignore everything except \(C\): at a fixed budget, different model/data splits have slightly different optimal regions (§3.1).

5. Measuring model size as FLOPs per token#

\(C = 6ND\) counts only parameter multiplies. It leaves out attention over the sequence and, depending on whether \(N\) includes embeddings, may count the vocabulary projection, which costs FLOPs but adds little capacity. The paper instead uses non-embedding FLOPs per token, \(M\), which counts attention but not the vocabulary (Eq. 2):

\(N_1\) is non-embedding parameters (Kaplan's choice), \(N_2\) is all parameters (Chinchilla's choice), and \(l_{seq}\) is the sequence length. The extra term \(12\, n_{layer} d_{model} l_{seq}\) is the cost of computing attention scores and weighted sums over the context, forward and backward. Then \(C = MD\).

The difference is large for small models. For an 8-layer, \(d_{model} = 512\) model with a 4096 context, \(6N_1/M = 0.43\) and \(6N_2/M = 1.32\) (Table 3). The ratio approaches 1 as models get wider. Most scaling-law fits rely on small models, so these errors bias the fitted exponents. Appendix A.2 (Figure 6) shows that \(6N_1\) overestimates large-model performance and \(6N_2\) underestimates it, while \(M\) predicts it best.

6. Optimal model/data allocation, and why it depends on data#

Using IsoFLOP profiles over 8 budgets from \(10^{17}\) to \(3 \times 10^{20}\), with about 10 model/data splits per budget and hyperparameters from Eq. 1, they fit (Eq. 4, Figure 4):

That puts slightly more than half of each extra order of magnitude of compute into model size. The loss curve fitted on these small runs predicts the bits-per-byte of the 7B and 67B models (Figure 5).

Then they refit on three different datasets (§3.3, Table 4):

The authors judge data quality in the order early < current < OpenWebText2, which is small and heavily curated. The model exponent rises in the same order. Their reading is that higher-quality data justifies a larger model for the same compute, possibly because clean text is "less predictive difficulty after sufficient training". This is offered as a possible reason Kaplan and Chinchilla disagreed. They also suggest the fitted exponent could serve as an indirect measure of data quality.

7. Alignment: SFT, repetition, and DPO#

The SFT set has about 1.5M instances: 1.2M "helpful" (31.2% general language, 46.6% math, 22.2% code) and 300K safety (§4). The 7B model trains for 4 epochs at LR 1e-5. The 67B model trains for only 2 epochs at 5e-6 because it overfits sooner.

The paper also tracks the repetition ratio: the fraction of responses to 3868 prompts that fall into an endless loop. It rises as more math SFT data is added. The 7B fix is two-stage SFT. Stage 1 uses all data, and stage 2 drops math and code. Repetition falls from 2.0% to 1.4% (0.020 to 0.014), HumanEval stays at 48.2, GSM8K moves from 63.9 to 63.0, and IFEval rises from 38.0 to 41.2 (Table 12). The 67B model was already below 1% repetition after stage 1, and stage 2 hurt its benchmarks, so it gets only one stage.

DPO then runs for one epoch (LR 5e-6, batch size 512, cosine schedule) on helpfulness and harmlessness preference pairs whose candidate responses come from DeepSeek Chat itself. It mainly helps open-ended chat: MT-Bench goes from 8.35 to 8.76 and AlignBench from 6.43 to 6.69 (Tables 7–8), while standard benchmarks barely move (Table 17).

8. Findings on benchmark gaming#

§5.5 is useful for anyone who builds evaluations.

• Multiple-choice data. Adding 20M Chinese MC questions to SFT raises 7B Chat MMLU from 49.4 to 60.9, C-Eval from 47.0 to 71.3 and CMMLU from 49.7 to 73.8, but TriviaQA stays at 57.9 and in-house ChineseQA slips from 75.0 to 74.4 (Table 13). The model learns the MC format, not new knowledge. They exclude MC data from both pre-training and SFT.
• Instruction data in pre-training. Adding 5M instruction examples in the last 10% of pre-training raises base-model benchmarks, but after SFT the result is the same as adding that data during SFT. They leave it out.
• System prompts slightly hurt the 7B model (MT-Bench 7.15 to 7.11) and help the 67B (8.35 to 8.58) (Table 14).
• Held-out tests separate the models more than standard benchmarks do. On LeetCode contest problems from July to November 2023, the Hungarian 2023 math exam and IFEval, 67B Chat scores 17.5 / 58 / 55.5, against 4.7 / 28.5 / 41.2 for 7B Chat. The authors note that ChatGLM3 6B matches strong scores on MBPP and GSM8K but falls well behind on these newer tests (Table 9).

Key numbers#

How to read the paper#

• 1 Introduction: Skim. It lists the contributions. The note that the multi-step scheduler "facilitates continual training" is worth remembering.
• 2.1 Data: Skim. Look at Table 1 (dedup across dumps) and the tokenizer paragraph. The filtering and remixing stages are not described in detail.
• 2.2 Architecture: Read closely, but it is short. Table 2 is the spec sheet, and the "depth instead of FFN width" choice is the main design decision.
• 2.3 Hyperparameters: Read closely. It defines the multi-step schedule (Figure 1) that later DeepSeek papers reuse.
• 2.4 Infrastructures: Skim, unless you train models. It covers the HAI-LLM framework, ZeRO-1, bf16 training with fp32 gradient accumulation, in-place cross-entropy and checkpoints every 5 minutes. Fire-Flyer covers the hardware side.
• 3 Scaling Laws (intro): Read closely. The three bulleted contributions summarize the section.
• 3.1 Scaling Laws for Hyperparameters: Read closely. Look at Figures 2–3 and Eq. 1, then check the formulas against Table 2 yourself.
• 3.2 Estimating Optimal Model and Data Scaling: Read closely. Look at Eq. 2 and Table 3 for why \(M\) is better than \(6N\), and at Figures 4–5 for the IsoFLOP fits and the loss prediction.
• 3.3 Scaling Laws with Different Data: Read closely. Table 4 is the paper's most cited finding.
• 4 Alignment: Read. Note the SFT data mix, the repetition-ratio metric and the DPO setup.
• 5.1 Public Benchmark Evaluation: Skim the benchmark list. Study Table 5 (base vs LLaMA-2) and Table 6 (base vs chat), and read the paragraph on which tasks drop after SFT.
• 5.2 Open-Ended Evaluation: Skim. Tables 7 (AlignBench) and 8 (MT-Bench).
• 5.3 Held-Out Evaluation: Read. Table 9 makes a good point about contamination and small models.
• 5.4 Safety Evaluation: Skim. Tables 10–11.
• 5.5 Discussion: Read closely. It holds the MC-data, instruction-in-pretraining and system-prompt results (Tables 12–14).
• 6 Conclusion: Skim. It previews Coder, MoE and RL work.
• Appendix: Skip on first pass, except A.2 (Figure 6, model-scale representations) if you care about scaling laws. A.4 compares the model with code- and math-specific models.

Suggested order: §2.2–2.3, then §3 in full, then §4 and §5.5, then the results tables in §5.1. Allow 60–75 minutes, or about 30 minutes if you only want the scaling-law results.

Check your understanding#

Limitations and open questions#

• Data pipeline is underspecified. Filtering and remixing are described only in general terms: no classifier details, domain mixture or language split. The scaling-law finding depends on data quality, and quality is judged by an internal assessment that is not described.
• Data-quality explanation is correlational. It rests on three datasets ordered by the authors' own quality judgment. Other differences between the datasets, such as size, domain or language mix, could also change the exponent.
• Hyperparameter laws depend only on \(C\). The authors admit the optimal region also shifts with the model/data split at fixed compute, and that earlier work modeled optimal batch size through loss instead (§3.1).
• Small fitting range. The laws are fit up to \(3 \times 10^{20}\) FLOPs and extrapolated about 1000x. The loss prediction matches for the two models trained, but that is two points.
• Internal evaluation framework. All baselines are rerun in DeepSeek's framework (Table 5). Comparisons are consistent but may not match other groups' published numbers.
• SFT is heavy on math and code. 46.6% of helpful SFT data is math. The authors note the gains may reflect "code completion and algebraic questions" more than broad skill (§5.1.2). The large GSM8K and HumanEval jumps after SFT should be read with that in mind.
• Language coverage. The authors say the Chinese data is "not exhaustive" and that skill in languages other than Chinese and English is weak (§6).

Further reading#

• In this series: DeepSeekMoE (sparse models trained on the same corpus), DeepSeek-Coder and DeepSeekMath (the code and math follow-ups promised in §6), DeepSeek-V2 (the next general model, which reuses this recipe and compares against DeepSeek 67B), and Fire-Flyer AI-HPC (the cluster and HAI-LLM stack).
• Kaplan et al., Scaling Laws for Neural Language Models: arXiv:2001.08361
• Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla): arXiv:2203.15556
• Touvron et al., Llama 2: arXiv:2307.09288
• Ainslie et al., GQA: arXiv:2305.13245
• Rafailov et al., Direct Preference Optimization: arXiv:2305.18290
• Model weights and code: github.com/deepseek-ai/DeepSeek-LLM