Janus

TL;DR#

• Janus is a single autoregressive transformer (DeepSeek-LLM 1.3B) that does both multimodal understanding (image + text in, text out) and text-to-image generation (text in, image tokens out).
• The key idea is to decouple visual encoding: images going into the model for understanding pass through SigLIP (a semantic encoder), while images the model must produce are represented as discrete codes from a VQ tokenizer. The transformer in the middle is shared.
• Earlier unified models (Chameleon, Show-o) used one visual encoder for both jobs. The ablation in Table 5 shows that this forces a trade-off that mostly hurts understanding: a shared VQ tokenizer gets 35.0 on MMBench, versus 69.4 for Janus's split design.
• Headline results at 1.3B (Tables 2–4): MMBench 69.4, SEED 63.7, POPE 87.0, beating LLaVA-v1.5 (7B) on those three; GenEval 0.61 overall (SDXL 0.55, DALL-E 2 0.52); COCO-30K FID 8.53.
• Training is three stages (adaptors → unified pretraining → SFT), plain next-token cross-entropy everywhere, and took 7 days on 16 nodes of 8×A100-40GB (§4.1).

Why it matters#

By late 2024 there were two mature, separate recipes. For understanding, LLaVA-style models put a vision encoder (CLIP or SigLIP) in front of an LLM. For generation, diffusion models dominated, and autoregressive models over discrete image tokens (e.g. LlamaGen) were catching up. A natural goal is one model that does both, so that a single set of weights reasons about images and creates them.

Two families tried this. "Tool-use" systems (Emu, DreamLLM, SEED-X) let the LLM emit conditioning vectors for a separate pretrained diffusion model, so the LLM never really generates pixels itself. "Truly unified" models (Chameleon, Show-o, VILA-U) used one transformer and one visual tokenizer for both directions. These were elegant but noticeably weaker at understanding than understanding-only models of the same size. Show-o, for example, scores 948 on MME-Perception versus 1338 for Janus (Table 2).

Janus argues the encoder is the bottleneck. Understanding needs high-level semantics: what objects are present, what the text in the image says, how things relate. Generation needs low-level detail: textures and local structure that a decoder can turn back into pixels. One representation cannot be optimal for both, so Janus gives each direction its own encoder and shares only the transformer.

Key idea The model reads images through a semantic encoder and writes images through a pixel-faithful tokenizer. Only the transformer is shared, so neither visual representation has to compromise.

In DeepSeek's 2024 arc, Janus belongs to the vision track that began with DeepSeek-VL, and it reuses DeepSeek-VL's chart and table data. One month later, JanusFlow kept the decoupled design but replaced discrete-token generation with rectified flow. DeepSeek-VL2 continued the understanding-only branch. In January 2025, Janus-Pro scaled the Janus recipe to 7B with more data.

Background you need#

• Vision-language models (LLaVA recipe). A pretrained image encoder turns an image into a grid of feature vectors. A small adaptor (usually an MLP) projects them into the LLM's embedding space, and the LLM treats them as extra "words". See DeepSeek-VL and LLaVA (2304.08485).
• SigLIP. A CLIP-style image encoder trained to match images to captions with a sigmoid loss. Its features are strongly semantic, which makes it a good "eye" for an LLM, but it cannot reconstruct pixels. 2303.15343.
• VQ tokenizer. A convolutional autoencoder whose bottleneck snaps each spatial feature to the nearest entry in a learned codebook. An image becomes a grid of integer IDs, and a decoder maps IDs back to pixels. Janus uses the tokenizer from LlamaGen, with a 16,384-entry codebook and 16× downsampling. LlamaGen (2406.06525).
• Autoregressive image generation. Once an image is a sequence of IDs, a GPT-style model can generate it one token at a time, in raster order, exactly as it generates text.
• Classifier-free guidance (CFG). Run the model with and without the text prompt, then push the prediction away from the unconditional one: \(l_g = l_u + s(l_c - l_u)\). Larger \(s\) follows the prompt more closely at some cost in diversity.
• Chameleon. Meta's early-fusion model that tokenizes images with one VQ tokenizer for both input and output. It is the main "single encoder" design that Janus argues against. 2405.09818.

The core ideas#

Two visual pathways, one transformer#

Figure 2 shows the whole architecture. Each input type has its own encoder, and all of them feed one sequence into the LLM:

All images are 384×384 (§4.1). With 16-pixel patches for SigLIP and 16× downsampling for the VQ tokenizer, both pathways yield a 24×24 grid, which is 576 positions per image. That figure is my arithmetic from the stated settings, not a number the paper prints. Each grid is flattened to 1-D in raster order before entering the LLM.

Generation works like text decoding. The prompt goes in, and the model predicts a distribution over the 16,384 codebook IDs for the next image position, samples one, embeds it through the generation adaptor, and repeats until all positions are filled. The VQ decoder then turns the grid of IDs into pixels.

The design adds no special attention masks or task-specific blocks. It is plain causal attention throughout (§3.1).

Why one encoder is not enough#

The paper's argument is about information granularity. A good understanding encoder throws away pixel detail and keeps semantic meaning, so that "a red jacket" and the text in a meme are easy to read off. A good generation tokenizer must keep enough local detail to rebuild the image, and its codes need not carry semantic meaning.

Force one encoder to do both and you get either semantically poor tokens, which hurt understanding, or tokens that lose detail, which hurt reconstruction. The ablation (Table 5, below) shows that in practice the loss falls mostly on understanding. A side benefit of decoupling is modularity: you can swap in a stronger understanding encoder (EVA-CLIP, InternViT) or a finer generation tokenizer (MoVQGAN) independently (§3.5).

Training: three stages, one loss#

The loss is ordinary next-token cross-entropy (Eq. 1):

Here \(x_i\) is the \(i\)-th token, either a text token or an image code ID, and \(\theta\) is the model's weights. For understanding and text data the loss is computed only on text tokens. For generation data it is computed only on image tokens. All tasks get the same weight.

The three stages (Figure 3, Table 1):

What each stage does:

• Stage I connects frozen parts. It uses 1.25M ShareGPT4V captions for understanding and about 1.2M ImageNet-1k images, formatted as "", for generation (§4.2).
• Stage II follows a curriculum borrowed from PixArt. ImageNet generation data appears only in the first 120K steps, so the model learns basic pixel dependencies first. Open-domain text-to-image data fills the remaining 60K steps. Text-only data comes from the DeepSeek-LLM corpus. In 25% of samples the caption is cut to its first sentence, so the model also learns to handle short prompts.
• Stage III mixes text dialogue, multimodal instruction data and text-to-image pairs, including 4M in-house pairs, into one model. There are no task-specific fine-tunes.

To support CFG at inference, 10% of text-to-image prompts are replaced with a pad token during training (footnote 2). That teaches the model unconditional generation.

Inference with CFG#

For each image token, Janus computes conditional logits \(l_c\) (prompt present) and unconditional logits \(l_u\) (prompt replaced by padding), then samples from

At \(s = 1\) this reduces to ordinary conditional sampling. At \(s = 5\), any token the prompt makes more likely becomes much more likely still. Each image takes one sequential step per image token, and each step runs both the conditional and unconditional pass.

Key numbers#

How to read the paper#

• Abstract and §1 Introduction: Read closely. The "granularity conflict" argument is the whole thesis. Figure 1 summarises the results.
• §2 Related Work: Skim. It is useful only for the tool-use vs. truly-unified split (§2.2–2.3).
• §3.1 Architecture: Read closely with Figure 2 open. It is short. Note which head handles which task.
• §3.2 Training Procedure: Read closely with Figure 3. Track what is frozen in each stage.
• §3.3–3.4 Objective and Inference: Read. Each is a single equation. Don't miss footnote 2 (the 10% prompt dropout behind CFG).
• §3.5 Possible Extensions: Skim. It is speculative, but it shows why the authors value modularity, which JanusFlow later acts on.
• §4.1 Implementation Details: Read closely with Table 1. It has all the hyperparameters and the compute figure.
• §4.2 Data Setup: Skim. Note the ImageNet-first curriculum and the 25% first-sentence trick.
• §4.3–4.4 Evaluation and Comparison: Read the tables, skim the prose. In Table 2, compare against both 1.3B and 7B models. In Table 3, look at the per-category columns, not just Overall.
• §4.5 Ablation Studies: Read closely. This is the evidence for the thesis. Work through Exp A–F in Table 5 yourself.
• §4.6 Qualitative Results and Appendix B: Skim. The meme examples in Figure 5 are cherry-picked but show the OCR gap against Chameleon and Show-o.
• Appendix A (semantic tokenizer): Skip on first pass. It describes how the stronger baseline Exp-B was built, and you need it only to judge whether that baseline is fair.

Suggested order: §1 → Figure 2 and §3.1 → §3.2 to §3.4 → Table 5 and §4.5 → Tables 2 to 4 → the rest. About 45–60 minutes.

Check your understanding#

Limitations and open questions#

• Low resolution. Images are 384×384, while the baselines in Figure 4 render at 1024 (SDXL) or 512 (LlamaGen). FID and visual fidelity numbers should be read with that in mind.
• Weak tokenizer by design. The authors use the plain LlamaGen VQ tokenizer "for simplicity" (footnote 3). Their own ablation shows that a better tokenizer improves FID, so generation quality is probably limited by the tokenizer.
• Counting is weak. GenEval counting is 0.30, below SDXL (0.39) and Show-o (0.49) (Table 3). Janus's strong overall score comes mostly from position (0.46) and color attribute (0.42).
• Slow sampling. Raster-order token-by-token generation with CFG means 576 sequential steps per image, each with two passes. The paper reports no speed figures. §3.5 mentions mixing in parallel decoding as a possible extension.
• Small scale only. Everything is at 1.3B. Whether decoupling still matters as much, or matters more, at 7B+ is left open. Janus-Pro later tested this at 7B.
• Two encoders, no shared visual space. The model sees images in SigLIP space but writes them in VQ space, so nothing explicitly ties the two together. JanusFlow addresses this with a representation-alignment loss.
• Missing details. Some data mixtures are in-house (2M generation pairs in Stage II, 4M in Stage III) and cannot be reproduced. The MMMU score (30.5) is below LLaVA-v1.5-7B (35.4), a reminder that a 1.3B LLM limits reasoning.

Further reading#

• JanusFlow: the follow-up a month later. It keeps the decoupled encoders, replaces VQ-token generation with rectified flow, and adds representation alignment.
• DeepSeek-VL: the understanding-side predecessor whose data Janus reuses. DeepSeek-VL2 continues the understanding-only track.
• DeepSeek LLM: the 1.3B base language model.
• Janus-Pro (Jan 2025), which scales Janus to 7B with more data and a refined training recipe: 2501.17811.
• Chameleon, the single-tokenizer early-fusion baseline: 2405.09818.
• Show-o, the previous best 1.3B unified model: 2408.12528.
• LlamaGen, the source of the VQ tokenizer and AR image generation recipe: 2406.06525.
• SigLIP: 2303.15343.
• Code and weights: github.com/deepseek-ai/Janus.