JanusFlow

TL;DR#

• JanusFlow is a 1.3B unified model. It understands images autoregressively, predicting the next text token, and generates images with rectified flow: it starts from Gaussian noise in a VAE latent space and repeatedly moves it along a predicted velocity until it becomes an image.
• The LLM itself is the velocity network. Only a light ConvNeXt encoder/decoder pair (about 70M parameters total, §4.1) is added around it. The paper's central claim is that rectified flow can be trained inside an LLM "without complex architectural modifications", using plain causal attention.
• Two design choices carry over or are new. Decoupled encoders are inherited from Janus: SigLIP for understanding, a separate generation path for drawing. Representation alignment (REPA) is new: an intermediate LLM layer is trained to match SigLIP features of the target image.
• Headline results at 1.3B: MJHQ FID-30k 9.51 (Janus 10.10), GenEval 0.63 (Janus 0.61), DPG-Bench 80.09, MMBench 74.9, SEED 70.5 (Tables 2–5).
• Ablations (Table 6) show that REPA improves FID from 19.84 to 17.61 and a shared encoder costs about 10 GQA points. The unified model stays close to understanding-only and generation-only models trained under the same conditions.

Why it matters#

Janus showed that separating the visual encoders for understanding and generation fixes the understanding penalty of unified models. But it still generated images the way LlamaGen does: one discrete VQ token at a time, 576 sequential steps per image. Its quality is limited by the VQ tokenizer. Most of the best text-to-image systems (Stable Diffusion 3, Flux) instead use continuous generative models such as diffusion or rectified flow over a VAE latent space.

Earlier attempts to merge an LLM with diffusion had drawbacks. Some kept an external diffusion model that the LLM only conditions (Emu, DreamLLM). Others trained one network for both, like Transfusion and Show-o, but used a shared visual representation for understanding and generation and often custom attention masks. JanusFlow takes the simplest version of the idea: the same LLM, causal attention, a small encoder/decoder, and a rectified-flow loss on the image positions. It then adds Janus's decoupling, which the authors show still matters in this setting.

The result improves on Janus in both directions at the same LLM size. Part of the understanding gain comes from an improved base LLM (see Limitations).

Key idea Let the LLM act as the velocity network of a rectified flow. Feed it noisy latents plus a timestep, read out a velocity, take an Euler step, and repeat. Separate encoders and an alignment loss to SigLIP keep the two tasks from interfering.

In the series, JanusFlow is the second and last of DeepSeek's 2024 unified-model papers. The understanding-only track continued in DeepSeek-VL2 the next month. The follow-up Janus-Pro (January 2025) went back to Janus's autoregressive generation and scaled it, so the flow-based branch has not been the one DeepSeek scaled so far.

Background you need#

• Janus. The predecessor. It uses SigLIP for image understanding, a VQ tokenizer for generation, and one shared transformer. Read the Janus page first.
• Latent space and VAE. Instead of generating 384×384×3 pixels directly, modern image generators work in the compressed latent space of an autoencoder. JanusFlow uses the pretrained, frozen SDXL-VAE. As general background (not from this paper), that VAE downsamples by 8× and has 4 latent channels, so a 384×384 image becomes a 48×48×4 latent.
• ODE and Euler steps. An ordinary differential equation \(\frac{dz}{dt} = v(z, t)\) says how a point moves over time. Euler's method approximates it by taking small steps: \(z_{t+\Delta t} = z_t + v(z_t, t)\,\Delta t\).
• Classifier-free guidance (CFG). Blend predictions with and without the prompt to push the output toward the prompt. JanusFlow applies it to velocities rather than logits.
• SigLIP. A semantic image encoder trained on image–text pairs, used as the understanding encoder and as the alignment target. 2303.15343.
• REPA (representation alignment). A 2024 finding that diffusion transformers train faster and generate better if one intermediate layer is regularised to match features from a pretrained vision encoder. 2410.06940.

The core ideas#

Rectified flow, for newcomers#

Diffusion models learn to undo a gradual noising process. Rectified flow (Liu et al., 2209.03003), closely related to flow matching (Lipman et al., 2210.02747), poses the same problem more simply. It learns a velocity field that carries points from a noise distribution to the data distribution along roughly straight lines.

Training (Eq. 3 in the paper):

1. Take a real data point \(x\), here a VAE latent of an image. Sample noise \(z_0 \sim \mathcal{N}(0, I)\) and a time \(t \in [0, 1]\).
2. Form the point on the straight line between them: \(z_t = t\,x + (1-t)\,z_0\). At \(t=0\) it is pure noise, and at \(t=1\) it is the data.
3. The velocity along that line is constant: \(x - z_0\). Train the network to predict it:

Here \(v_\theta\) is the network's predicted velocity, \(\theta\) its parameters, and the expectation is over random data, noise and times.

A 1-D example. Let the noise sample be \(z_0 = -1\), the data point \(x = 3\), and \(t = 0.25\). Then \(z_t = 0.25 \cdot 3 + 0.75 \cdot (-1) = 0\), and the regression target is \(x - z_0 = 4\). The network learns that at position 0 and time 0.25, the flow moves at speed +4.

Any given noisy point \(z_t\) could lie on lines toward many different images. The best squared-error prediction is therefore the average direction toward plausible data. That averaged field is what lets sampling produce new images instead of memorised ones.

Sampling. Draw fresh noise \(z_0\) and integrate \(\frac{dz}{dt} = v_\theta(z_t, t)\) from \(t=0\) to \(t=1\) with Euler steps (Eq. 4):

With 30 steps, \(dt = 1/30\). Because the training paths are straight lines, the learned trajectories tend to be fairly straight, so relatively few steps work. The "rectified" in the name refers to this straightening. The original paper also has a "reflow" procedure that straightens paths further, but JanusFlow uses only the basic objective.

Two practical details from JanusFlow: \(t\) is sampled from a logit-normal distribution, following Stable Diffusion 3, which concentrates training on intermediate times. And 10% of text prompts are dropped so the model can predict unconditional velocities for CFG (§3.4).

The LLM as the velocity network#

Figure 2 shows how generation runs inside the LLM:

1. Sample noise \(z_0\) in SDXL-VAE latent space.
2. The generation encoder \(g_{enc}\) embeds it into a sequence of tokens. \(g_{enc}\) is a 2×2 patchify layer, two ConvNeXt blocks, and a linear layer (§4.1).
3. The sequence fed to the LLM is: text prompt, a |BOI| (begin-of-image) token, a time embedding for the current \(t\), and the latent tokens. That is \(H_{gen} W_{gen} + 1\) image-side positions. Attention is plain causal: the authors tried other masks and saw no benefit (§3.2).
4. The LLM's outputs at the latent positions go through the generation decoder \(g_{dec}\) (two ConvNeXt blocks, pixel-shuffle upsampling, linear layer) to give a velocity with the same shape as the latent. A long skip connection links \(g_{enc}\) to \(g_{dec}\).
5. Take an Euler step, overwrite \(z_t\) with \(z_{t+dt}\) in the input, and repeat until \(t = 1\). The VAE decoder then turns \(z_1\) into an image.

CFG on velocities (Eq. 5):

Here \(x^{con}\) is the text prompt, \(\varnothing\) means no prompt, and \(w \ge 1\) sets the guidance strength. This is algebraically the same form as Janus's logit guidance. The main FID results use \(w = 2\) and 30 steps (§4.4).

Contrast with Janus. Janus generates 576 tokens one after another, and each token can depend on those before it. JanusFlow updates all latent positions in parallel at every step. It needs about 30 full forward passes, each doubled by CFG, instead of 576 incremental ones. Note one subtlety: because the latent tokens change at every step, their attention states must be recomputed on each pass.

Understanding works exactly as in a LLaVA-style model. SigLIP-Large-Patch/16 (\(f_{enc}\)) turns the image into a feature grid, a linear layer projects it into the LLM, the grid is wrapped in |BOI| and |EOI| tokens, and the LLM predicts the answer token by token.

Decoupled encoders, again#

Transfusion uses one set of shared blocks in VAE latent space for both reading and writing images, and Show-o uses one discrete tokenizer. JanusFlow uses SigLIP for understanding and VAE + ConvNeXt for generation. Table 6 compares this against two alternatives at 256×256 resolution and 50K iterations:

Generation quality is about the same across the three. Understanding jumps by roughly 15 VQAv2 points and 10 GQA points with SigLIP. So in this setting the benefit of decoupling comes from being free to use a strong pretrained semantic encoder for understanding. Separating otherwise identical encoders (C vs. B) gives mixed understanding results and a small FID gain.

Representation alignment (REPA)#

Decoupling creates an opportunity: SigLIP features of the target image are available during generation training. JanusFlow adds a loss that pulls the LLM's internal features for the noisy latent toward those clean semantic features (Eq. 8):

The symbols:

• \(x^{res}\) is the target image.
• \(f_{enc}\) is SigLIP. The stop-gradient means SigLIP is not updated by this loss.
• \(q_\theta(z_t)\) is the LLM's hidden state after its 6th of 24 blocks when the input is the noisy latent \(z_t\).
• \(h_\varphi\) is a small 3-layer MLP that projects into SigLIP's dimension.
• \(\mathrm{sim}\) is the mean element-wise cosine similarity.

To make the positions line up, the generation path is configured so that its token grid matches SigLIP's (\(H_{gen}=H_{im}\), \(W_{gen}=W_{im}\)).

Intuitively, the early LLM layers must already "see" what the clean image means while looking at noise. That gives the later layers a semantic scaffold for predicting velocity. In Table 6, removing REPA (Exp. A vs. F) worsens FID from 17.61 to 19.84 and lowers CLIP similarity from 26.40 to 24.94. Appendix Figure 2 shows the gap persisting over the first 50K iterations.

Total losses: understanding data uses \(\mathcal{L}_{AR}\), cross-entropy on response tokens (Eq. 6). Generation data uses \(\mathcal{L}_{RF} + \mathcal{L}_{REPA}\). All three losses are active in every stage.

Training stages#

Source: Table 1 and §3.3. Note that the data-ratio order differs from Janus's table. Stage 2 is heavily weighted toward generation, which the authors justify by the slow convergence of diffusion-type models. An EMA of weights with ratio 0.99 is used for stability. Each model takes about 1,600 A100 GPU-days (§4.1).

Key numbers#

How to read the paper#

• Abstract and §1 Introduction: Read. Short. It states the two strategies (decoupling and alignment) and the headline numbers.
• §2 Related Work: Skim. The last paragraph lists the claimed advantages over Transfusion and similar work.
• §3.1 Background: Read closely if rectified flow is new to you. Eqs. 2–3 are the whole generative model. Otherwise skim.
• §3.2 Unified Framework: Read closely with Figure 2. Work out the generation loop: noise → \(g_{enc}\) → LLM → \(g_{dec}\) → Euler step → repeat. Note the causal-attention remark and Eq. 5.
• §3.3 Training Schemes: Read with Figure 3 and Table 1. Track what is frozen in each stage.
• §3.4 Training Objective: Read closely. Eq. 8 (REPA) is the main new component. Check that you understand the stop-gradient and the grid-size matching.
• §4.1 Implementation Details: Read. It gives module sizes, the 6th-block REPA tap, EMA and compute. Don't skip footnote 1 about the enhanced LLM.
• §4.2 Training Data: Skim. It largely follows Janus. Appendix B lists the datasets.
• §4.3–4.4 Evaluation and Results: Read the tables. In Table 2 compare per-category GenEval scores with Janus. In Table 5 compare with Janus row by row.
• §4.5 Ablations: Read closely with Table 6. Note it is at 256×256, with FID-10k at \(w=7.5\), so the numbers are not comparable with the main tables.
• §4.6 and Appendix E (qualitative): Skim.
• Appendix A (256 vs. 384 model): Read. The 256 model beats the 384 model on GenEval (0.70 vs. 0.63) and DPG-Bench (81.23 vs. 80.09), which is worth thinking about.
• Appendix C (CFG and step sweeps): Skim Figure 1. FID has an optimum CFG while CLIP keeps rising, and step count matters little between 25 and 50.

Suggested order: §3.1 → Figure 2 and §3.2 → §3.4 → Table 6 and §4.5 → Tables 2 to 5 → Appendix A and C → the rest. About 60 minutes, or 45 if you already know flow matching.

Check your understanding#

Limitations and open questions#

• Confounded comparison with Janus. The base LLM was upgraded (footnote 1), and the training schedule is much longer: 390K vs. 180K Stage-2 steps, and about 1,600 A100 GPU-days vs. Janus's 7 days on 128 A100s. Generation also gained a better latent space (SDXL-VAE vs. a VQ tokenizer). The paper does not isolate how much each factor contributes to "rectified flow beats autoregression".
• Small scale. Like Janus, everything is at 1.3B and 384px. DALL-E 3 still leads on GenEval (0.67), and VILA-U 7B has a better MJHQ FID (7.69).
• Some regressions against Janus. GenEval two-object drops from 0.68 to 0.59. MMMU and MM-Vet are lower. The 384 model is worse than the 256 model on instruction-following benchmarks, which the authors attribute to easier semantic control at low resolution (Appendix A). That is a hypothesis, not a tested claim.
• Ablations at a different setting. Table 6 uses 256×256, 50K iterations, FID-10k at \(w=7.5\), so it shows direction rather than final magnitudes. The decision to use plain causal attention rests on unreported "preliminary experiments".
• No efficiency analysis. Despite the parallel-update advantage over token-by-token decoding, sampling speed and memory are never measured.
• Two separate pipelines inside one model. Generation and understanding share weights but use different input encoders, different losses and different output heads. Whether the model can edit an image it was shown, or interleave reasoning and drawing, is not tested.

Further reading#

• Janus: the direct predecessor, which introduced decoupled visual encoding.
• DeepSeek-VL: source of the chart and table understanding data. DeepSeek-VL2 continues the understanding-only track.
• Janus-Pro (Jan 2025), a scaled-up successor in the autoregressive Janus style: 2501.17811.
• Rectified flow, "Flow Straight and Fast": 2209.03003. Flow matching: 2210.02747.
• Stable Diffusion 3, which scales rectified-flow transformers and is the source of logit-normal \(t\) sampling: 2403.03206.
• REPA, representation alignment for diffusion transformers: 2410.06940.
• Transfusion, an LLM plus diffusion with shared image encoding and the main baseline design: 2408.11039.
• Show-o: 2408.12528.
• Code and weights: github.com/deepseek-ai/Janus.