DeepSeek-VL

TL;DR#

• DeepSeek-VL is a pair of open vision-language chat models, 1.3B and 7B, built on top of DeepSeek LLM checkpoints. Both are LLaVA-style: an image encoder, a small adaptor, and a decoder-only LLM.
• The 7B model uses a hybrid vision encoder. SigLIP-L reads a 384×384 view for semantics and SAM-B reads a 1024×1024 view for fine detail. The two are fused into a fixed 576 visual tokens (§3.1). The 1.3B model uses SigLIP alone (Table 4).
• The main training finding: if you continue pretraining an LLM on pure multimodal data, its language skills collapse. Keeping roughly 70% text-only data in the mix, and ramping the multimodal share up gradually ("modality warm-up"), keeps them (§3.2.2, Figure 4, Figure 9).
• Instruction data is organized around a use-case taxonomy built from real GPT-4V and Gemini prompts found online (Table 3). The same taxonomy drives a 100-question human evaluation.
• Headline results: DeepSeek-VL-7B scores 73.2 MMBench, 70.4 SEED-Bench (GPT-4V: 71.6), 36.1 MathVista (GPT-4V: 47.8) (Table 5). On text benchmarks it stays close to DeepSeek-LLM-7B-Chat: 68.4 vs 68.5 HellaSwag, 52.4 vs 49.4 MMLU, but 55.0 vs 63.0 GSM8K (Table 7).

Why it matters#

By early 2024, open vision-language models (VLMs) such as LLaVA-1.5, Qwen-VL and CogVLM did well on academic benchmarks but felt clearly worse than GPT-4V and Gemini in real use. The paper names four causes (§1). Most compute went into instruction tuning rather than pretraining. Instruction data was a pile of academic datasets. Encoders ran at low resolution, typically 336×336 or 448×448, which is too coarse for OCR or small objects. And the models that did pretrain often lost language ability along the way.

DeepSeek-VL treats a VLM as an LLM that has learned to see, not a vision model with a text head bolted on. That framing explains the choices. Pretrain jointly on text and images instead of only training a projector. Keep most of the training stream as text. Measure language benchmarks throughout. Spend the visual token budget carefully, because every image token costs context and inference compute.

Key idea Multimodal pretraining and language ability compete for the same weights. Keep the language share high (about 70%), and start the run on text before ramping images in, and you can add vision without paying much in language skill.

In the DeepSeek arc, this is the first multimodal paper. Janus and JanusFlow then took on both understanding and image generation. DeepSeek-VL2 replaced the fixed-resolution hybrid encoder with dynamic tiling and the dense LLM with a DeepSeekMoE model. The paper's last paragraph already announces that plan: "scale up DeepSeek-VL to larger sizes, incorporating Mixture of Experts" (§5).

Background you need#

• LLaVA-style VLM. A pretrained image encoder turns an image into a grid of feature vectors. A small adaptor (an MLP) maps each vector into the LLM's embedding space. The LLM then reads those "visual tokens" like ordinary text tokens, and the loss is next-token prediction on the text. See LLaVA.
• CLIP / SigLIP encoders. These are vision transformers trained contrastively so that image and caption embeddings line up. They encode semantics ("this is a dog") well, but run at low resolution, and they can map visually different images to similar embeddings ("CLIP-blind pairs", Tong et al. 2024). SigLIP replaces the softmax contrastive loss with a sigmoid loss.
• SAM encoder. The image encoder from Segment Anything, a ViTDet-style ViT trained for segmentation at 1024×1024. It is vision-only, so it captures low-level layout and edges rather than language-aligned meaning.
• Catastrophic forgetting. When you keep training a model on a new data distribution, it can lose what it learned before. Here the old distribution is text and the new one is image-text.
• Perplexity-based multiple-choice evaluation. Instead of having the model generate "B", you score each answer option by its likelihood and pick the most likely one. This gives a smoother signal for small models that know the answer but cannot yet follow the answer format.

The core ideas#

1. The hybrid vision encoder: 1024×1024 in 576 tokens#

The authors wanted high resolution for OCR, documents and small objects, but without a large number of visual tokens. Their answer is two encoders that see the same image at different resolutions, fused into one token per spatial cell.

The shapes, from §3.1:

The trick is to bring both paths to the same 24×24 spatial grid, so each of the 576 positions can hold the SigLIP vector and the SAM vector for the same image region side by side. Interpolating 64→96 first means two stride-2 convolutions (96→48→24) land exactly on 24.

For context, a plain SigLIP-L at 384×384 also gives 576 tokens. The hybrid design keeps that token cost while adding a view at roughly 2.7× the linear resolution (1024 vs 384). That leaves room in a 4096-token context for multi-turn chat and interleaved images.

2. The vision-language adaptor: a "hybrid MLP"#

The adaptor has two layers (§3.1). First, two separate single-layer MLPs process the high-resolution (SAM) and low-resolution (SigLIP) features. Their outputs are concatenated along the feature dimension, and a second, shared MLP layer maps the result into the LLM's input space.

The ablation in Table 10 explains the choice. Separate MLPs let each branch adapt to its own feature scale and distribution. A shared MLP mixes the two better. The hybrid version gets the best average, 55.9, against 55.2 for shared and 54.5 for separate. It also beats concatenating the two feature sets along the sequence axis with token pooling (55.5 and 54.2). Ablations like this one are often what decides the design, and Table 10 is worth reading closely.

3. The LLM and where it starts#

The language model is DeepSeek LLM: a LLaMA-style pre-norm transformer with RMSNorm, SwiGLU feed-forward layers and rotary position embeddings, using the same tokenizer (§3.1). The training does not start from a finished base model. It starts from an intermediate pretraining checkpoint, so that joint vision-language training reads as a continuation of LM pretraining. The 1B model starts from a DeepSeek-LLM-1B trained on about 500B text tokens, and the 7B from DeepSeek-LLM-7B trained on about 2T tokens.

4. Three training stages#

Figure 3 shows the pipeline. In every stage the loss is next-token prediction on text tokens only.

Hyperparameters are from Table 4. With full packing, stage 2 for the 7B processes at most about 42,000 × 2,304 × 4,096 ≈ 400B tokens. That is my arithmetic, not a figure the paper reports.

Why not just scale stage 1? Because it does not help. Table 8 trains stage 1 for 2K, 8K, 20K and 80K steps and then runs SFT directly. Average scores go from 57.5 to 55.1, 55.5 and 55.6. A two-layer MLP has too little capacity to absorb more knowledge, so the real learning has to happen in the LLM, in stage 2. Table 9 confirms that stage 2 matters: stage 1 + 3 averages 57.4, stage 2 + 3 averages 61.7, and all three average 62.4.

5. Balancing modalities: the 7:3 ratio#

This is the paper's most reusable result (§3.2.2, Figure 4). On the 1.3B model, the authors sweep the multimodal:language ratio over 10:90, 25:75, 60:40, 75:25 and 100:0, tracking SEED-Bench, MMBench and MMBench-CN alongside MMLU, HellaSwag and Pile perplexity.

• At 100% multimodal, the multimodal metrics rise, but the language metrics drop sharply.
• Adding text data largely fixes the language decline and does not cost much on multimodal metrics.
• Each modality's score tracks its share of the data, which is direct evidence that the two compete.

The authors attribute the forgetting to two things: multimodal corpora are linguistically simple compared with real text, and the modalities compete for capacity. They settle on language:multimodal ≈ 7:3, and Table 1 lists the text-only corpus (the DeepSeek-LLM corpus) at 70.0%.

Modality warm-up (§4.4, Figure 9). Starting the fixed mix at step 0 destabilizes the model. Instead, the language ratio starts at 1.0 and decays to the target (for example 0.7). In Figure 9 this avoids an early drop in language ability and ends equal or better on both Pile-test and MMBench.

Modality group training (§4.4, Figure 8). This is a systems detail that matters. If one batch mixes text-only and image samples, the fast text samples wait for the slow image samples. Making each global step either all-text or all-multimodal raises training efficiency by 20% without hurting the metrics.

6. Making small-model experiments informative#

The team iterated on the 1.3B model and then scaled up to 7B. During stage 2, though, the 1.3B model's generative benchmark scores swung wildly. They traced this to the model's small capacity and to the lack of instruction-following data: it sometimes "knew" the right option but could not produce it in the expected format. This is the measurement artifact described by Schaeffer et al.. They used two fixes (§3.2.2):

1. Multi-choice PPL (MCPPL). Feed the prompt, the image and all the options, compute the perplexity at each answer position (A, B, C, D), and choose the best. The y-axes in Figures 4, 8 and 9 are "MCPPL Accuracy".
2. A small share of SFT data mixed into pretraining, so instruction following is not the bottleneck.

7. Data built around real use#

Pretraining (Table 1) mixes interleaved image-text (13.1%, including MMC4, Wikipedia, WikiHow and in-house textbooks), captions (11.1%), table and chart data (2.1%), web code (0.4%), scene-text OCR (1.2%), document OCR (2.1%) and text (70.0%). Two in-house sources stand out (§2.1):

• Plot-to-code: 1.46M Jupyter notebooks from The Stack produce 2M image-code pairs, filtered to 1.1M pairs with one image and at least 5 lines of code.
• Document OCR: 1.4M arXiv papers rendered with Nougat's tools, plus 860K English and 180K Chinese e-books and millions of K-12 exam questions, rendered from HTML.

SFT (Table 2) is 47.9% text-only SFT and 35.5% open GPT-4V-distilled sets (ShareGPT4V, LAION-GPTV and others), with smaller shares of table/chart and web-code data. The remaining 10.5% is in-house data organized by the taxonomy in Table 3. The taxonomy covers Recognition, Conversion (for example UI→code), Analysis, Commonsense Reasoning, Logical Reasoning, Evaluation, Multi-graph and Safety, each split into secondary and tertiary categories. The authors built it by collecting real GPT-4V and Gemini test cases posted online, so the SFT prompts reflect what people actually ask.

8. Training infrastructure#

The models were trained with HAI-LLM, DeepSeek's in-house framework (§3.3). The vision encoders plus the text embedding are treated as one heterogeneous "first layer". It does not fit standard tensor parallelism, but it is cheap, so it is simply recomputed on every tensor-parallel rank. Layers are re-split across pipeline stages to balance the uneven cost. After that, standard Megatron-style 3D parallelism applies. The 7B took 5 days on 64 nodes × 8 A100s, and the 1B took 7 days on 16 nodes. The later Fire-Flyer paper describes the cluster and software stack.

Key numbers#

How to read the paper#

• 1 Introduction (pp. 3–5): Read closely. The four bullet points on why open VLMs underperform frame every later decision. Skim the Figure 1 examples.
• 2 Data Construction
• 2.1 VL pretraining data: Skim. Look at Table 1's ratios, especially the 70% text. Read the Web Code and Document OCR paragraphs for the in-house pipelines.
• 2.2 SFT data: Skim, but spend a minute on the Table 3 taxonomy, which drives both SFT and human evaluation.
• 3 Approach
• 3.1 Architecture: Read closely. Trace the tensor shapes yourself, from 64×64×256 to 576×2048.
• 3.2.1 Stage 1: Read. It is short, and it sets up the negative result in Table 8.
• 3.2.2 Stage 2: Read closely. This is the heart of the paper: Figure 4's ratio sweep, the forgetting hypothesis, and the MCPPL and SFT-mixing fixes.
• 3.2.3 Stage 3: Skim. Note which modules train and that SAM stays frozen.
• 3.3 Hyperparameters and Infrastructures: Skim Table 4. Read the pipeline-parallelism paragraph if you work on training systems.
• 4 Evaluation
• 4.1 Multimodal benchmarks: Skim Tables 5 and 6. Compare against the 7B open models, not the 13B or closed rows.
• 4.2 Language benchmarks: Read Table 7. It is the test of the paper's central claim.
• 4.3 Human evaluation: Skim. It is small (100 questions) and depends on an in-house setup.
• 4.4 Ablation study: Read closely. Tables 8–10 and Figures 8–10 hold most of the evidence for the design choices.
• 5 Conclusion: Skim. Note the MoE plan.
• Appendix A: Skip on first pass. It is qualitative examples only.

Suggested order: §1 → §3.1 → §3.2 → §4.4 → Table 1 and Table 7 → §4.1. Budget about 60–75 minutes for a careful read, or about 25 minutes for this order alone.

Check your understanding#

Q1How does the hybrid encoder make SAM and SigLIP features line up position by position?

SAM-B outputs a 64×64×256 map for a 1024×1024 image. The adaptor interpolates it to 96×96 and applies two stride-2 convolutions, giving 24×24×1024. SigLIP-L at 384×384 already gives a 24×24 grid (576 tokens) of 1024-dim features. Because both are 24×24, they can be concatenated along channels into 576 tokens of 2048 dims, each describing one image region at two resolutions.

Q2Why didn't the authors simply train the adaptor longer on more data in stage 1?

They tried. Table 8 shows that going from 2K to 80K stage-1 steps does not improve, and slightly lowers, post-SFT scores (57.5 → 55.6 average). The adaptor is a two-layer MLP with very little capacity, so it can align the embedding spaces but cannot store world knowledge. That knowledge has to go into the LLM, which is why stage 2 unfreezes it.

Q3What happens in stage 2 if you train on 100% multimodal data, and what are the two proposed causes?

Multimodal metrics improve, but language metrics (MMLU, HellaSwag, Pile perplexity) decline sharply (Figure 4). The authors propose two causes. Multimodal corpora are simple and distributionally far from real text, and the two modalities compete for model capacity, which produces catastrophic forgetting of language.

Q4Why use multi-choice perplexity instead of generated answers to monitor the 1.3B model?

The 1.3B model's generative scores fluctuated sharply during stage 2. It often had the knowledge but could not reliably produce the option letter in the expected format, partly because pretraining had no instruction data. Scoring each option's perplexity removes the formatting bottleneck and gives a smooth, monotonic signal, so small-scale experiments say something reliable about design choices.

Q5Modality group training changes nothing about the data mix. Why does it speed training by 20%?

In a mixed batch, each step waits for the slowest sample to finish, and image samples (vision encoders, long visual sequences) are slower than text-only ones. Making each global step homogeneous, either all text or all multimodal, removes that waiting while keeping the overall ratio over many steps the same.

Q6DeepSeek-VL-7B beats its base LLM on MMLU but loses 8 points on GSM8K. What does that suggest?

Joint training does not simply erase language ability. Knowledge-heavy and multiple-choice skills survive or even improve (MMLU 52.4 vs 49.4, AGIEval 27.8 vs 19.3). Multi-step math reasoning is more fragile (55.0 vs 63.0). The authors read this as residual competition between modalities at 7B capacity and expect larger models to suffer less.

Limitations and open questions#

• Math and reasoning lag. MathVista is 36.1 against GPT-4V's 47.8, and in the human evaluation all open models trail GPT-4V badly on logical reasoning. The authors attribute this to base model size (§4.1, §4.3).
• GSM8K drops 8 points relative to DeepSeek-LLM-7B-Chat (Table 7), so the "preserve language" goal is only partly met.
• Resolution is still fixed. Every image is resized to 1024×1024 and 384×384 squares, which distorts extreme aspect ratios such as long documents and infographics. This is the main limitation DeepSeek-VL2 addresses.
• Different encoders at the two sizes. The 1.3B uses SigLIP only (Tables 4 and 7), so 1.3B vs 7B comparisons mix model scale with encoder design.
• Ablations run at small scale. The ratio sweep, warm-up and grouping experiments are on the 1.3B, over about 16K steps, with MCPPL metrics. How well they transfer to 7B is assumed, not shown in full.
• The ratio is described inconsistently. The text and Table 1 say 70% language, but the Figure 4 caption calls the "suitable ratio" "multimodal:language=70%:30%". The text and Table 1 are presumably right.
• Evaluation details. MMBench and MMBench-CN are reported on dev splits (§4.1). The human evaluation has 100 questions, and the pairwise judging uses GPT-4V, including when GPT-4V is one of the contestants.
• Heavy reliance on GPT-4V-distilled SFT data (35.5% of SFT, Table 2) means part of the "real-world" behavior is inherited from GPT-4V.

Further reading#

• Next in the series: DeepSeek-VL2 (dynamic tiling plus MoE). Also Janus and JanusFlow, on unified understanding and generation.
• Base LLM and its training stack: DeepSeek LLM, Fire-Flyer AI-HPC.
• LLaVA, the template architecture: arXiv:2304.08485.
• SigLIP: arXiv:2303.15343. Segment Anything (SAM): arXiv:2304.02643.
• "Eyes Wide Shut", on CLIP-blind pairs and adding vision-only encoders: arXiv:2401.06209.
• Nougat, the source of the arXiv rendering tools: arXiv:2308.13418.
• Schaeffer et al., "Are Emergent Abilities of LLMs a Mirage?": arXiv:2304.15004.
• Code and models: github.com/deepseek-ai/DeepSeek-VL.