DeepSeek 2024
/

15 papers · Jan–Dec 2024 · ≈4 hours of guides

The year DeepSeek caught up.

In 2024, DeepSeek published fifteen papers that, read in order, explain how a small lab trained a frontier model cheaply. This site has a reading guide for each: the core ideas in plain English, key numbers, a section-by-section map of the paper, and questions to check yourself.

Start with paper 1Pick a reading path

Reading paths

The road to V3

How a dense 67B model became a 671B-parameter MoE trained for a fraction of the usual cost.

  1. DeepSeek LLM
  2. DeepSeekMoE
  3. DeepSeek-V2
  4. DeepSeek-V3

Reasoning with RL

GRPO, then proof search with verifier feedback: the ideas that later fed into R1.

  1. DeepSeekMath
  2. DeepSeek-Prover
  3. DeepSeek-Prover-V1.5
  4. DeepSeek-V3

Seeing and drawing

From an image-understanding model to one model that both understands and generates images.

  1. DeepSeek-VL
  2. Janus
  3. JanusFlow
  4. DeepSeek-VL2

Systems first

The hardware and training-efficiency choices behind the cheap frontier model.

  1. Fire-Flyer AI-HPC
  2. DeepSeek-V2
  3. DeepSeek-V3

Timeline

Foundation modelsCode & mathTheorem provingVision & multimodalInfrastructure
Jan
DeepSeek LLMDeepSeek's first LLM report: it re-derives scaling laws (including how batch size and learning rate should scale with compute) and uses them to train 7B and 67B bilingual dense models on 2T tokens, with the 67B beating LLaMA-2 70B on code, math and reasoning.Foundation models · IntroDeepSeekMoERedesigns the Mixture-of-Experts layer with many small experts plus a few always-on shared experts, so each expert learns more focused knowledge. A 16B-parameter model matches dense 7B models at about 40% of the compute, and this layer became the FFN of DeepSeek-V2 and V3.Foundation models · IntermediateDeepSeek-CoderA family of open code models (1.3B, 6.7B, 33B) trained from scratch on 2T tokens of repository-ordered code with a fill-in-the-middle objective. The 33B model beats every open code model of its time, and the instruct version beats GPT-3.5-Turbo on most coding benchmarks.Code & math · Intro
Feb
DeepSeekMathA 7B model continued from DeepSeek-Coder on a 120B-token math corpus mined from Common Crawl, then improved with GRPO, a critic-free variant of PPO. It reaches 51.7% on MATH without tools or voting. GRPO went on to become DeepSeek's standard RL algorithm.Code & math · Advanced
Mar
DeepSeek-VLDeepSeek's first vision-language model family (1.3B and 7B). It pairs a SigLIP+SAM hybrid encoder that reads 1024×1024 images in 576 tokens with a pretraining recipe that keeps about 70% text data, so the model learns to see without losing much of its language ability.Vision & multimodal · Intermediate
Apr
May
DeepSeek-V2A 236B-parameter Mixture-of-Experts model that activates only 21B parameters per token, introducing Multi-head Latent Attention (MLA) to shrink the KV cache and scaling DeepSeekMoE with communication-aware routing; it beats DeepSeek 67B while cutting training cost by 42.5% and raising generation throughput 5.76x.Foundation models · AdvancedDeepSeek-ProverDeepSeek turns about 870k informal competition problems into Lean 4 theorems, proves them with its own model in a self-improving loop, and fine-tunes DeepSeekMath 7B on the resulting 8 million verified proofs. The 7B prover beats GPT-4 and earlier tree-search systems on miniF2F.Theorem proving · Intermediate
Jun
DeepSeek-Coder-V2DeepSeek-V2's MoE models (236B total / 21B active, and a 16B Lite) continued for 6T more tokens of code, math and text, then aligned with SFT and GRPO. It was the first open model to roughly match GPT-4-Turbo on coding and math benchmarks, with 338 languages and a 128K context.Code & math · Intermediate
Jul
ESFTA parameter-efficient fine-tuning method for Mixture-of-Experts models: measure which experts a task's tokens route to, then train only those experts and freeze everything else. On DeepSeek-V2-Lite it nearly matches full fine-tuning, clearly beats LoRA, and loses much less general ability.Foundation models · Intermediate
Aug
DeepSeek-Prover-V1.5A 7B Lean 4 prover trained with continued pre-training, thought-annotated fine-tuning and GRPO reinforcement learning with the Lean checker as the reward, plus RMaxTS, a curiosity-driven tree search built on top of whole-proof generation. It reaches 63.5% on miniF2F-test and 25.3% on ProofNet.Theorem proving · AdvancedFire-Flyer AI-HPCDeepSeek describes Fire-Flyer 2, the 10,000-GPU PCIe A100 cluster behind its 2024 models, and the software that makes cheap hardware perform. The authors claim about 80% of DGX-A100 performance at roughly half the cost and 40% less energy.Infrastructure · Intermediate
Sep
Oct
JanusA 1.3B model that both reads and draws images with one autoregressive transformer, using a semantic encoder (SigLIP) to see and a separate VQ tokenizer to draw. Splitting the two visual pathways fixes the understanding penalty that single-encoder unified models pay.Vision & multimodal · Intermediate
Nov
JanusFlowJanusFlow keeps Janus's split visual encoders but generates images with rectified flow instead of discrete tokens, running the flow inside the same 1.3B LLM that answers questions. It adds a representation-alignment loss that ties the generation path to the understanding encoder.Vision & multimodal · Intermediate
Dec
DeepSeek-VL2DeepSeek-VL2 rebuilds DeepSeek-VL around dynamic image tiling and a DeepSeekMoE language model with MLA. Three models with 1.0B, 2.8B and 4.5B activated parameters match or beat dense open VLMs of similar or larger active size on OCR, document and chart understanding, and visual grounding.Vision & multimodal · IntermediateDeepSeek-V3A 671B-parameter Mixture-of-Experts model (37B active per token) trained on 14.8T tokens for 2.788M H800 GPU hours. It adds bias-based load balancing, multi-token prediction, FP8 training and a communication-hiding pipeline schedule to the DeepSeek-V2 design, and matches leading closed models on many benchmarks.Foundation models · Advanced