DeepSeek-VL2

TL;DR#

• Three open MoE vision-language models: DeepSeek-VL2-Tiny, -Small and DeepSeek-VL2, built on 3B, 16B and 27B DeepSeekMoE LLMs. Counting the vision encoder, they have 1.0B, 2.8B and 4.5B activated parameters (Tables 1–2, §6).
• Vision: the fixed two-resolution hybrid encoder of DeepSeek-VL is replaced by dynamic tiling. The image is resized to the best-fitting grid of up to 9 tiles of 384×384, plus a global thumbnail, all encoded by one SigLIP-SO400M-384 (§2).
• Language: the dense LLM is replaced by DeepSeekMoE with MLA, the compressed-KV attention from DeepSeek-V2, for cheaper inference. The largest model uses sigmoid routing with an expert bias term for load balancing, as in DeepSeek-V3 (Table 1).
• Data: about 800B tokens of VL pretraining (Table 2), with recaptioned images, much more OCR and document data, and new visual grounding data that has the model emit bounding boxes as text (§3).
• Headline: DeepSeek-VL2 (4.5B active) reaches DocVQA 93.3, ChartQA 86.0, OCRBench 811, MMStar 61.3, MathVista 62.8, and RefCOCO val 95.1. Comparable dense models have 8B+ active (Tables 3–5). OCRBench goes from DeepSeek-VL-7B's 456 to 811.

Why it matters#

DeepSeek-VL proved the recipe: an LLaVA-style model, a pretraining mix that protects language skills, and data built around real use. It had two structural limits. Every image was squashed to a 1024×1024 square, which is bad for long documents, infographics and wide screenshots. And it used a dense 7B LLM, so every token paid for every parameter.

By late 2024 the open VLM field had largely settled on tiling (LLaVA-NeXT, InternVL, NVLM) to handle arbitrary resolutions and aspect ratios. Meanwhile, DeepSeek's own LLM line had moved to MoE with MLA (DeepSeek-V2) and auxiliary-loss-free balancing. DeepSeek-VL2 combines the two: a standard, well-tested tiling front end, and DeepSeek's own sparse back end.

Key idea Spend parameters where they are cheap. A small shared ViT handles any resolution by tiling, and a sparse MoE LLM supplies capacity while activating only a few billion parameters per token. Together they let a 4.5B-active model compete with dense 7–8B VLMs.

In the arc, this is the last multimodal paper of 2024. It reuses DeepSeekMoE (January), MLA (May), and the sigmoid-gated, bias-balanced router that DeepSeek-V3 describes two weeks later. It also adds grounding (outputting box coordinates), which earlier DeepSeek VLMs could not do.

Background you need#

• LLaVA-style VLM and the DeepSeek-VL recipe. Vision encoder → MLP adaptor → decoder LLM, trained in three stages (align, pretrain, SFT), with text-only data mixed in to avoid forgetting. See DeepSeek-VL.
• Dynamic tiling (also called AnyRes or slicing). A ViT pretrained at a fixed resolution (here 384×384) can still process a large image if you cut the image into tiles of that size and encode each tile separately. A downscaled "thumbnail" of the whole image keeps the global context. See LLaVA-NeXT and InternVL 1.5.
• Mixture-of-Experts with shared experts. Each FFN layer has many small "routed" experts, of which a router picks the top-K per token, plus a few "shared" experts that every token uses. See DeepSeekMoE.
• Multi-head Latent Attention (MLA). Keys and values are compressed into a low-rank latent vector (rank 512 here), so the KV cache is much smaller and inference throughput higher. See DeepSeek-V2.
• Auxiliary-loss-free load balancing. Instead of relying only on an extra loss to spread tokens across experts, each expert gets a bias added to its routing score, nudged up or down depending on whether the expert is under- or over-loaded. See Wang et al. 2024 and DeepSeek-V3.
• Visual grounding / referring expressions. Given a phrase such as "the giraffe at the back", output the bounding box of that object. RefCOCO, RefCOCO+ and RefCOCOg are the standard benchmarks.

The core ideas#

1. Dynamic tiling: picking a grid for the image#

SigLIP-SO400M-384 works natively at 384×384. To handle any size and aspect ratio, the paper defines a set of candidate resolutions (§2):

Here \(m\) and \(n\) are the tile counts along the two axes, so \(m{:}n\) is the aspect ratio and \(mn \le 9\) caps the number of local tiles at nine. For an input of size \((H, W)\), the model computes, for each candidate, the padding needed after resizing: the long side is scaled to fit, the aspect ratio is kept, and the short side is padded. It picks the candidate \((m_i, n_i)\) with the least padding. The resized image is cut into \(m_i \times n_i\) local tiles, and one global thumbnail tile is added, so the encoder sees \(1 + m_i n_i\) tiles.

A wide screenshot might land on a 1×3 or 2×3 grid, and a tall receipt on a 3×1. This is the main improvement over DeepSeek-VL, which squashed every image into a square.

Each tile yields \(27 \times 27 = 729\) SigLIP embeddings of dimension 1152. Two practical rules apply. Tiling is disabled when there are more than 2 images, to keep the context manageable. And for InfoVQA, whose images are huge and extremely elongated, the cap is raised to \(mn \le 18\) at evaluation time (footnote 2).

2. The adaptor: pixel shuffle, newline tokens and a view separator#

729 tokens per tile is too many, so a 2×2 pixel shuffle merges each 2×2 neighborhood of tokens into one, reducing a tile from 27×27 to 14×14 = 196 tokens (§2). Pixel shuffle (space-to-depth) stacks the four neighbors' channels into one vector instead of averaging them, so no information is thrown away; the MLP decides what to keep. Going from 27 to 14 implies the grid is padded to 28 first, though the paper does not say this.

The tokens are then laid out with spatial markers so the LLM can tell where rows end and which view is which:

• Global thumbnail: 14 rows of 14 tokens, with a <tile_newline> after each row, giving \(14 \times 15 = 210\) tokens.
• <view_separator>: 1 token between the thumbnail and the local tiles.
• Local tiles: arranged as one big 2D grid of shape \((m_i \cdot 14,\; n_i \cdot 14)\), with a <tile_newline> at the end of each full row, giving \(m_i \cdot 14 \times (n_i \cdot 14 + 1)\) tokens.

Total visual tokens:

Worked examples:

Compare DeepSeek-VL's fixed 576. VL2 trades more tokens on detailed images for much better OCR and documents, and with a 4,096-token training context (Table 2), that is one reason tiling is turned off for multi-image inputs. A two-layer MLP projects all these tokens into the LLM's embedding space.

3. The MoE language models#

Configurations from Table 1:

A few details are easy to miss. Tiny does not use MLA. The largest model's router matches the DeepSeek-V3 style: sigmoid gating plus a per-expert bias for load balancing, with a small auxiliary loss on top (weight 0.0001 vs 0.001 for the smaller two, Table 2). Its "expert bias correction step" is 0.001 in stage 2 and 0 in stages 1 and 3, so the biases only adapt during pretraining. The vocabularies also differ, which suggests the base LLMs come from different checkpoints. The vision encoder (about 0.4B, Table 3) is the same for all three.

Why MoE matters here: on a token-by-token basis, DeepSeek-VL2 does about the compute of a 4–5B dense model while holding 27B parameters of knowledge. MLA also shrinks the KV cache, which matters more for VLMs because images add hundreds or thousands of tokens to the context. The paper says the three models fit on a single GPU with 10GB, 40GB and 80GB of memory respectively (§6).

4. Training: three stages, encoder unfrozen#

The stages are the same as DeepSeek-VL's, but what trains changes (§4.1, Table 2):

There are two departures from DeepSeek-VL. First, the vision encoder trains, even in stage 1, because a ViT pretrained only at fixed resolution needs to adapt to tiles and thumbnails. It trains with a 0.1× learning-rate multiplier throughout. Second, the stated pretraining ratio is about 70% VL to 30% text. That reverses DeepSeek-VL's 70% text, and the paper presents it without comment as "following DeepSeek-VL". All stages use sequence length 4,096. Stage 2 uses a step LR schedule (divide by √10 at 50% and 75% of training), and SFT uses a constant LR.

Infrastructure (§4.2). The models were trained on HAI-LLM with pipeline, tensor and expert parallelism. Dynamic tiling creates two new balancing problems. The vision encoder at the head of the pipeline is split at fine granularity across stages to avoid pipeline bubbles. And because different samples have very different tile counts, image tiles are load-balanced across data-parallel ranks. Text-only and image batches also get two different pipeline strategies, switched on demand. Training took 7, 10 and 14 days on 16, 33 and 42 nodes of 8×A100 (Tiny, Small, VL2 respectively).

5. Data: recaptioning, OCR and grounding#

The pretraining data (§3.2) keeps DeepSeek-VL's categories but improves their quality:

• Interleaved: WIT, WikiHow and a 30% random sample of OBELICS (a ratio chosen with Tiny experiments), plus Chinese data from Wanjuan and in-house data.
• Captions, regenerated: open caption sets varied widely in quality. An in-house captioner rewrites them, prompted with OCR hints, metadata and the original caption (in the style of PixelProse). DeepSeek Chat then scores the writing quality of every caption, which filters out repetitive outputs.
• OCR: LaTeX-OCR, 12M RenderedText and a large in-house English and Chinese document set.
• VQA: general VQA from DeepSeek-VL; PubTabNet, FinTabNet and Docmatix for documents; Websight and plot-to-code, partly regenerated with DeepSeek-V2.5; and visual-prompt QA (arrows and boxes drawn on images, following ViP-LLaVA).
• Grounding, which is new. Boxes are emitted as text inside special tokens, with coordinates normalized to the range 0–999:

Prompt:   Locate <|ref|>the leftmost person<|/ref|> in the given image.
Response: <|ref|>the leftmost person<|/ref|><|det|>[[x1, y1, x2, y2]]<|/det|>

Negative samples, where the object is absent, teach the model to say so. Grounded conversation uses a <|grounding|> trigger token, so the model interleaves boxes into free-form answers, for example "Two <|ref|>dogs<|/ref|><|det|>[[...]]<|/det|> are running on the grass."

The SFT data (§3.3) is mostly cleaned and regenerated public data. Some lessons worth noting:

• Regenerate responses. Public VQA answers tend to be short, garbled by poor OCR, or hallucinated, so answers are rewritten using the question, the image and the OCR text.
• An early Tiny model mixed English words into Chinese answers, which larger models did not do. An in-house Chinese QA set fixed it.
• For math and reasoning, Tiny does better with concise answers, and detailed chains of reasoning helped less at that size.
• SFT adds in-context grounding: find an object of the same category in a second image, given a box drawn on the first.

Key numbers#

How to read the paper#

• Abstract and Figure 1: Read. Figure 1 (average score vs activated parameters) is the paper's thesis in one plot.
• 1 Introduction: Skim. It lists the three upgrades: tiling, MoE with MLA, and data.
• 2 Model Architecture: Read closely. Work through the candidate-resolution rule, the token formula and Figure 3 yourself. Table 1 holds the MoE configs. Notice Tiny's MHA and the largest model's sigmoid router.
• 3 Data Construction
• 3.1 Alignment data: Skim. It is a single sentence.
• 3.2 Pretraining data: Read the captioning pipeline and the grounding format. Skim the dataset lists.
• 3.3 SFT data: Skim, but note the practical lessons: language mixing in Tiny, concise answers for small models, regenerated responses.
• 4 Training Methodology
• 4.1 Training Pipelines: Read. Note what is frozen in each stage, and that the ViT trains.
• 4.2 Hyperparameters and Infrastructures: Read Table 2 (token counts, LR multiplier, aux loss, bias step). Read the parallelism paragraph if you build training systems.
• 5 Evaluation
• 5.1 Multimodal Performance: Read Tables 3–5. Compare rows by activated parameters, and watch for CoT and "different setting" markers.
• 5.2 Qualitative Study: Skim. Figures 8–10 (grounding, grounded conversation, in-context grounding) are the interesting ones.
• 6 Conclusion: Read. It is short and includes the stated limitations.

Suggested order: Figure 1 → §2 → Table 2 and §4.1 → §3.2 grounding format → Tables 3–5 → §6. About 45–60 minutes for a full read, 20 minutes for this order.

Check your understanding#

Limitations and open questions#

• Stated by the authors (§6): only a few images fit per session because of the context window; the model struggles with blurry images and unseen objects; and reasoning is weaker than perception.
• No ablations. Unlike DeepSeek-VL, the paper does not ablate tiling against the old hybrid encoder, MoE against dense, the data mix, or the captioning pipeline. The gains cannot be attributed to individual changes.
• No language benchmarks. Given the reversed 70/30 ratio, the absence of text-only evaluations (DeepSeek-VL's Table 7 equivalent) is a notable gap.
• Activated parameters are not the whole cost. Memory scales with total parameters (27B needs 80GB), and tiled images can cost about 2K tokens. Figure 1's x-axis flatters MoE when memory or tokens per image are the constraint.
• Benchmark comparisons are mixed. Some baselines use CoT or "different settings" (marked in Tables 3–4), several cells are missing, and Qwen2-VL-7B is still ahead on DocVQA, OCRBench and MMMU.
• Tiling is disabled for more than 2 images, so fine-grained multi-image tasks fall back to low resolution.
• The base LLMs (3B, 16B, 27B) are not otherwise documented in the series. The paper gives their configs but not their pretraining data or text-only scores.
• Storytelling safety is acknowledged as a tension. The model inherits DeepSeek Chat's alignment, which limits darker genres (§5.2).

Further reading#

• Predecessor: DeepSeek-VL, for the three-stage recipe and the modality-balance study.
• Language side: DeepSeekMoE (fine-grained and shared experts), DeepSeek-V2 (MLA), DeepSeek-V3 (sigmoid routing with bias-based balancing at scale).
• Sibling multimodal work: Janus, JanusFlow.
• Auxiliary-loss-free load balancing: arXiv:2408.15664.
• Tiling prior work: LLaVA-NeXT, InternVL 1.5, NVLM.
• SigLIP: arXiv:2303.15343. ViP-LLaVA, for visual prompts: arXiv:2312.00784. PixelProse, for dense recaptioning: arXiv:2406.10328.
• Code and models: github.com/deepseek-ai/DeepSeek-VL2.