DeepSeekMath

TL;DR#

• DeepSeekMath Corpus: 120B tokens (35.5M web pages) of math content, pulled from Common Crawl by an iterative fastText classifier plus human URL annotation. It is about 9 times the size of OpenWebMath (§1.1, §2.1). In a controlled test at 1.3B scale it beats MathPile, OpenWebMath and Proof-Pile-2 on all 8 benchmarks (Table 1).
• DeepSeekMath-Base 7B continues training DeepSeek-Coder-Base-v1.5 7B for 500B tokens, 56% of them from the new corpus. It scores 36.2% on MATH and 64.2% on GSM8K, ahead of Minerva 540B's 33.6% and 58.8% (Table 2).
• GRPO (Group Relative Policy Optimization) removes PPO's value network. For each question it samples a group of answers, scores them, and uses each answer's z-score within its group as the advantage. The KL penalty moves from the reward into the loss (§4.1).
• RL with GRPO on only about 144K GSM8K/MATH chain-of-thought questions lifts the instruct model from 82.9% to 88.2% on GSM8K and from 46.8% to 51.7% on MATH. It also improves benchmarks it never trained on, such as CMATH (84.6% → 88.8%) (Table 5). Self-consistency over 64 samples reaches 60.9% on MATH (abstract).
• Two pre-training findings: code training first helps math (Table 6), and arXiv papers did not help at 1.3B or 7B (Tables 8 and 9).
• A "unified paradigm" (§5.2) writes SFT, RFT, DPO, PPO and GRPO as one gradient with different data sources and gradient coefficients. The paper also finds that RL improved Maj@K but not Pass@K (Figure 7).

Why it matters#

In early 2024 open models were far behind on competition math. On MATH, GPT-4 scored about 53% with chain-of-thought and Gemini Ultra 53.2%, while the best open models were in the 30s (Table 5). Earlier math-specialized models such as Minerva (closed, 540B) and Llemma (open, 34B) leaned on arXiv papers and small curated web sets. The open question was whether a small model could close the gap.

DeepSeekMath attacks the problem in two places. On the data side, it shows that Common Crawl holds far more usable math than previous corpora captured, if you mine it iteratively. On the RL side, standard PPO needs a value model about as large as the policy, which roughly doubles the memory you train. For math, where the reward is essentially "is the final answer right," a learned per-token value function is also hard to fit, because only the last token gets a score. GRPO replaces the value model with a simple statistic: how this answer scored compared with other answers to the same question.

Key idea You don't need a learned critic to get a baseline. Sample several answers to the same prompt and use their average reward as the baseline. Answers better than their siblings get pushed up, and worse ones get pushed down.

GRPO is the most widely reused idea from DeepSeek's 2024 papers. It is the RL algorithm in DeepSeek-V2, DeepSeek-Coder-V2 and DeepSeek-Prover-V1.5, and it continues into DeepSeek-V3 and, in 2025, DeepSeek-R1 (arXiv:2501.12948). DeepSeekMath-Base itself is the starting model for DeepSeek-Prover.

Background you need#

• Policy-gradient RL for LLMs. The model (the "policy" \(\pi_\theta\)) generates an answer, a reward scores it, and you raise the log-probability of tokens in high-reward answers. The core estimator is \(\nabla_\theta J = \mathbb{E}[A_t \nabla_\theta \log \pi_\theta(o_t \mid \cdot)]\), where the advantage \(A_t\) says how much better than expected the action was. Subtracting a baseline from the reward lowers variance without adding bias.
• PPO. Proximal Policy Optimization limits each update by clipping the probability ratio between the new and old policy to \([1-\varepsilon, 1+\varepsilon]\). In RLHF it uses a learned value model \(V_\psi\) and Generalized Advantage Estimation (GAE) to compute per-token advantages. See Schulman et al. 2017 and InstructGPT.
• KL penalty to a reference model. RLHF keeps the policy close to the SFT model \(\pi_{ref}\) by penalizing KL divergence. This limits reward hacking and keeps the language fluent.
• Outcome vs process reward models. An outcome reward model (ORM) scores the whole answer. A process reward model (PRM) scores each reasoning step. See Let's Verify Step by Step and Math-Shepherd, which this paper follows for reward-model training data.
• Chain-of-thought (CoT), program-of-thought (PoT), tool-integrated reasoning. In CoT the model writes reasoning in natural language. In PoT it writes a Python program whose output is the answer. Tool-integrated reasoning mixes the two (ToRA, arXiv:2309.17452).
• Pass@K vs Maj@K. Pass@K counts a problem as solved if any of K samples is correct. Maj@K (self-consistency) takes the majority-vote answer among K samples.

The core ideas#

1. Mining 120B math tokens from Common Crawl#

The pipeline (Figure 2, §2.1) is a loop:

1. Train a fastText classifier. Use 500K positives from OpenWebMath (the seed) and 500K random Common Crawl pages as negatives. The settings are vector dimension 256, learning rate 0.1, word n-grams up to 3, minimum word count 3, and 3 epochs.
2. Recall pages from a URL-deduplicated and near-deduplicated Common Crawl of 40B HTML pages. Rank them by classifier score and keep the top ones. The cutoff was chosen by pre-training on the top 40B, 80B, 120B and 160B tokens. The first iteration keeps 40B tokens.
3. Find math domains. Group pages by base URL. Any domain where more than 10% of pages were recalled counts as math-related, for example mathoverflow.net.
4. Human annotation. Annotators mark which URL paths in those domains are math (for example mathoverflow.net/questions). Pages under those paths that were missed get added to the seed, which makes the next classifier more diverse.

After four iterations the result is 35.5M pages and 120B tokens. The fourth iteration found that about 98% of its data had already been collected in the third, so they stopped. Decontamination drops any text that shares a 10-gram with GSM8K, MATH, CMATH or AGIEval (exact match for test strings of 3 to 10 grams).

Is it good? §2.2 trains the same DeepSeek-LLM 1.3B for 150B tokens on each corpus (Table 1). DeepSeekMath Corpus gives 23.8% on GSM8K and 13.6% on MATH, against Proof-Pile-2's 14.3% and 11.2%. The corpus is also multilingual, mainly English and Chinese, which is why CMATH jumps to 41.5% while the English-centric corpora barely move it. Figure 3 shows that the gap holds even at matched token counts, so the gain is from quality and not only size.

2. DeepSeekMath-Base 7B: start from code#

The base model is initialized from DeepSeek-Coder-Base-v1.5 7B and trained for 500B tokens: 56% DeepSeekMath Corpus, 4% AlgebraicStack, 10% arXiv, 20% GitHub code, 10% English and Chinese natural language. The peak learning rate is 4.2e-4, the batch is 10M tokens, the context is 4K, and the schedule is the multi-step one from DeepSeek LLM (§2.3).

Why code? §5.1.1 tests it on 1.3B models (Table 6). Under two-stage training, 400B code tokens then 150B math tokens beats 400B general tokens then the same math, both without tools (MATH 15.3% vs 14.4%) and with Python (MATH+Python 9.4% vs 6.7%). Mixing code and math in a single stage helps tool use most (MATH+Python 13.5%) and avoids forgetting how to code (Table 7), but it slightly hurts math without tools. The authors guess that a 1.3B model lacks the capacity to absorb both at once.

The 20% code in the mix keeps the base model's HumanEval near its starting point (40.9% vs 43.2% for Coder-v1.5). MMLU rises from 49.1% to 54.9% and BBH from 55.2% to 59.5% (Table 4). So math training also improves general reasoning benchmarks.

3. From PPO to GRPO#

This is the section to read most carefully.

PPO's objective (Eq. 1), for a question \(q\) and output \(o\) sampled from the old policy:

Here \(\rho_t\) is the probability ratio between the current and old policy for token \(t\), and \(\varepsilon\) is the clip range. The advantage \(A_t\) comes from GAE, which needs a learned value function \(V_\psi\) trained alongside the policy. The reward includes a per-token KL penalty (Eq. 2):

where \(r_\varphi\) is the reward model, \(\pi_{ref}\) the reference (initial SFT) model and \(\beta\) the KL coefficient.

The paper names two problems with this (§4.1.1):

1. \(V_\psi\) is usually a model about as large as the policy, which adds a lot of memory and compute.
2. The reward model scores only the last token, so a value function that must be accurate at every token is hard to train.

GRPO's objective (Eq. 3). For each question, sample a group of \(G\) outputs \(\{o_1, \dots, o_G\}\) from \(\pi_{\theta_{old}}\) and maximize:

with \(\rho_{i,t} = \pi_\theta(o_{i,t} \mid q, o_{i,<t}) / \pi_{\theta_{old}}(o_{i,t} \mid q, o_{i,<t})\). The clipped surrogate has the same form as PPO's. The differences are in where \(\hat{A}\) comes from and where the KL goes.

The KL term sits directly in the loss rather than inside the reward, so it doesn't complicate the advantage. It uses an unbiased estimator (Eq. 4, from Schulman 2020):

This is always \(\ge 0\), because \(x - \log x - 1 \ge 0\) for \(x > 0\). It is computed per token from probabilities you already have.

Outcome-supervised advantage (§4.1.2). Score each of the \(G\) outputs with the reward model to get \(\mathbf{r} = \{r_1, \dots, r_G\}\). Then give every token of output \(i\) the same advantage:

Worked example. The paper's rewards come from a learned reward model, but for intuition take a 0/1 correctness score. Suppose \(G = 64\) and 16 samples are correct. The mean is 0.25 and the standard deviation is \(\sqrt{0.25 \times 0.75} \approx 0.433\). Each correct answer gets \(\hat{A} = 0.75 / 0.433 \approx +1.73\) on every token, and each wrong answer gets \(-0.25 / 0.433 \approx -0.58\). On a hard question where only 2 of 64 are correct, the correct ones get about \(+5.5\). Rare successes earn large updates, and on a question the model already solves 60 times out of 64, the few failures are pushed down hard. Normalizing within the group automatically adjusts for question difficulty, which is the job a value function does in PPO.

Process-supervised advantage (§4.1.3). A PRM scores the end of each reasoning step \(j\) in each output. All step rewards across the group, \(\mathbf{R}\), are normalized together: \(\tilde{r}_i^{index(j)} = (r_i^{index(j)} - \mathrm{mean}(\mathbf{R})) / \mathrm{std}(\mathbf{R})\). A token's advantage is the sum of normalized rewards for the steps that end at or after it:

Tokens early in a solution collect credit or blame from all later steps, much like a return-to-go.

PPO vs GRPO at a glance:

Iterative GRPO (§4.1.4, Algorithm 1). As the policy improves, the old reward model falls behind it. Each outer iteration does three things: generate new reward-model training data from current policy samples, continue training the reward model with a replay buffer that keeps 10% of historical data, and reset \(\pi_{ref}\) to the current policy. Two iterations were run. The first gave most of the gain (Figure 6).

Settings for DeepSeekMath-RL (§4.2): it starts from DeepSeekMath-Instruct 7B and trains on about 144K chain-of-thought questions from GSM8K and MATH. Other SFT questions are held out on purpose, to measure out-of-domain transfer. The reward model starts from DeepSeekMath-Base 7B with a learning rate of 2e-5. The policy learning rate is 1e-6, the KL coefficient \(\beta = 0.04\), \(G = 64\) samples per question, the maximum length is 1024 tokens and the batch size is 1024. The policy makes a single update after each exploration stage. That means \(\pi_{\theta_{old}} = \pi_\theta\) at update time, so \(\rho = 1\) and the clip never activates in practice. The objective then reduces to a REINFORCE-style gradient with a group baseline plus the KL term.

4. The unified paradigm: every method is a gradient coefficient#

§5.2.1 (Eq. 5) writes the gradient of many post-training methods as:

Three things vary from method to method: the data source \(\mathcal{D}\), the reward function \(\pi_{rf}\) (a rule or a model) and the gradient coefficient \(GC\) (Table 10, Appendix A.1):

Two experiments on DeepSeekMath-Instruct 1.3B (Figure 5):

• Online beats offline. Online RFT matches RFT early on and pulls ahead later, as the policy drifts away from the SFT model whose samples RFT keeps reusing.
• Signed, graded coefficients beat 0/1. GRPO beats Online RFT, because RFT never penalizes wrong answers and rewards every correct answer equally. GRPO with process supervision beats GRPO with outcome supervision.

5. What RL actually changed#

Figure 7 plots Pass@K and Maj@K for the Instruct and RL models at temperature 0.7. RL raises Maj@K but not Pass@K. The authors' reading: RL makes the output distribution more robust by moving correct answers that were already in the top K up to the top. It does not give the model new abilities. They attribute this partly to their simple setup (SFT questions only, plain nucleus sampling) and list directions for improvement in §5.2.3: out-of-distribution prompts, tree-search sampling, algorithms robust to noisy rewards, and better PRMs. For context, PRM800K is estimated to have about 20% incorrect labels (§5.2.3).

Key numbers#

How to read the paper#

• §1 Introduction (with 1.1 Contributions, 1.2 Evaluations): Read. It is a compact summary of everything, and the contribution bullets point to the main findings.
• §2.1 Data Collection and Decontamination: Read closely with Figure 2. The domain-level "more than 10% recalled" trick is the non-obvious step.
• §2.2 Validating the Quality of the Corpus: Read Table 1 and Figure 3. Skim §2.2.1 (training settings).
• §2.3 Training and Evaluating DeepSeekMath-Base 7B: Read the data mix in the first paragraph and Table 2. Skim Tables 3 and 4.
• §3 Supervised Fine-Tuning: Skim. Note the 776K examples and the three solution formats. Table 5 matters, but read it together with §4.2.
• §4.1 Group Relative Policy Optimization: Read closely, with pen and paper. Compare Eq. 1 with Eq. 3 term by term, study Figure 4 (the PPO vs GRPO diagram) and Eq. 4, then §4.1.2 to §4.1.4 and Algorithm 1.
• §4.2 Training and Evaluating DeepSeekMath-RL: Read closely. The hyperparameters and the in-domain vs out-of-domain split are here.
• §5.1 Lessons Learnt in Pre-Training: Read Table 6 (code helps math). Skim §5.1.2 (arXiv), but note its caveats.
• §5.2 Insights of Reinforcement Learning: Read closely: Table 10, Figure 5 and Figure 7. §5.2.3 is a useful short research agenda.
• §6 Conclusion, Limitation, and Future Work: Read the limitations paragraph.
• Appendix A.1: Read if you want the derivations behind Table 10. The GRPO gradient (Eqs. 19 to 21) shows exactly how the KL term enters the coefficient.

Suggested order: §1 → §4.1 → §4.2 → §5.2 → §2.1 → Tables 1 and 2 → §5.1 → Appendix A.1. About 90 minutes, two-thirds of it on the RL sections.

Check your understanding#

Limitations and open questions#

• RL mostly sharpens existing ability (Figure 7). The authors link this to using only SFT-stage questions and naive sampling. Whether GRPO with better prompts and rewards can raise Pass@K is left open.
• Geometry and theorem proving are weak. The authors say the model can't handle triangle and ellipse problems in their dry runs, which they attribute to data selection bias. It is also worse than GPT-4 at using few-shot examples (§6).
• The arXiv conclusion is narrow. The authors only tested arXiv-only training at 1.3B and 7B. They did not test arXiv mixed with other data, at larger scale, or for tasks such as informalization (§5.1.2).
• The group std can be zero. If all \(G\) answers get the same reward, the normalization in §4.1.2 divides by zero. The paper doesn't say how this is handled. Implementations typically add a small epsilon, and such groups give no learning signal anyway.
• Length normalization. The \(1/|o_i|\) factor means each token of a long answer gets a smaller share of the update than each token of a short one. The paper doesn't discuss what this does to response length.
• The reward-model details are thin. The reward model's training data follows Math-Shepherd, but the paper doesn't report the reward model's accuracy, and outcome vs process supervision is compared only at 1.3B (Figure 5).
• No PPO baseline. There is no direct, compute-matched GRPO vs PPO comparison on the same setup. The efficiency claim rests on dropping the value model, not on a measured head-to-head.
• The \(G = 64\) cost. Removing the value model saves training memory, but sampling 64 completions per prompt makes generation the dominant cost. The paper points to efficient inference as important (§5.2.3) but doesn't quantify it.

Further reading#

• Series: DeepSeek-Coder (source of the v1.5 initialization), DeepSeek-Prover (built on DeepSeekMath-Base), DeepSeek-V2 and DeepSeek-Coder-V2 (GRPO for alignment), DeepSeek-Prover-V1.5 (GRPO with proof-assistant feedback), DeepSeek-V3.
• PPO: Schulman et al., arXiv:1707.06347. GAE: arXiv:1506.02438.
• RLHF with PPO for LLMs (InstructGPT): arXiv:2203.02155.
• KL estimators: John Schulman, Approximating KL Divergence.
• Process supervision: Lightman et al., arXiv:2305.20050. Math-Shepherd: arXiv:2312.08935.
• Math corpora: OpenWebMath arXiv:2310.06786, Llemma / Proof-Pile-2 arXiv:2310.10631, Minerva arXiv:2206.14858.
• DPO: arXiv:2305.18290. RFT: Yuan et al., arXiv:2308.01825.
• Later: DeepSeek-R1, pure-RL reasoning with GRPO: arXiv:2501.12948.
• Code and models: github.com/deepseek-ai/DeepSeek-Math.