Models & Parameters

Diffusion Transformer (DiT): What It Is and Why It Won

Also called dit, dit model, mmdit, transformer diffusion backbone

A diffusion transformer is a diffusion model whose denoising backbone is a transformer rather than a convolutional UNet. The latent is cut into patches, each patch becomes a token, and every token attends to every other one at every layer. That single change is what gave current models legible text, long prompts that hold, and video measured in seconds instead of frames.

From latent grid to tokens

The pipeline up to this point is unchanged: an encoder compresses the image or clip into a latent, and denoising happens there. What changes is the network doing the denoising.

A diffusion transformer takes that latent grid and cuts it into small patches, commonly 2x2 latent cells each. Every patch becomes one token, gets a position embedding so the model knows where it came from, and then flows through a stack of identical transformer blocks. The timestep and any global conditioning are folded in by modulating the normalisation layers rather than by concatenating channels. At the end, tokens are projected back to patches and reassembled into a latent.

There is no downsampling path, no bottleneck, no skip connections. The spatial resolution of the token grid is constant through the whole network, and every layer does full self-attention. That is expensive, and it is the entire point: no layer is ever restricted to a local window.

Why it changed what prompts can do

Three capabilities arrived more or less together with this architecture.

Prompt adherence over many clauses. When text and image tokens share an attention operation across every block, a clause in the middle of a long prompt has as much access to the canvas as the first one. Convolutional backbones injected text at a handful of resolution levels and lost clauses accordingly.

Text in frame. Spelling is a global constraint on a local texture. Letters must agree with each other across a distance that spans many latent cells. Full attention can enforce that; local convolution cannot, which is why a whole generation of models produced convincing signage covered in nonsense.

Variable resolution and duration. A token sequence has no preferred length. The same weights can generate 1:1, 16:9 or 9:16, or a 5 second and a 10 second clip, because changing the output shape only changes how many tokens there are. Convolutional models learned composition at a fixed size and duplicated subjects when pushed off it.

Tokens are the cost model

If you want to predict how expensive or slow something will be, count tokens. For video the count is roughly the spatial latent area divided by the patch size, multiplied by the number of latent frames after temporal compression.

  • Doubling duration doubles the tokens, and attention cost grows with their square.
  • Going from 720p to 1080p multiplies the spatial term by about 2.25.
  • Halving the patch size quadruples the tokens, which is why detail oriented variants are so much slower.

This is why platform pricing tiers cluster around resolution and duration rather than around prompt complexity, and why a model that offers 10 seconds at 1080p is doing far more work than the numbers on the spec sheet suggest. Video families in production use several strategies to keep this tractable: full 3D attention over space and time in the highest quality tiers, and factorised or windowed attention where speed matters more.

What to actually do with this

Treat the architecture as a filter, not a verdict. If your shot needs readable words, a specific object count, a layout you described in words, or an aspect ratio away from the usual, choose a transformer based family and expect it to hold. If your problem is grain, skin, a niche aesthetic, or a style adapter you already own, architecture will not decide the outcome and you should test rather than assume. And when a long clip is timing out or costing more than expected, the fix is almost always fewer tokens: shorter takes chained together, or a lower resolution pass finished with an upscaler.

The prompt for this

A starting point that reliably produces the effect. Adjust the subject and setting; keep the technical clauses.

A neon sign reading OPEN LATE above a diner door, rain on the pavement, reflections in a shallow puddle, 35mm, night

Try Diffusion Transformer (DiT) yourself

Open the generator with a starting point already filled in.

AI Video GeneratorAI Image Generator

Frequently asked questions

What is the difference between a diffusion transformer and a UNet?
A UNet is convolutional, so most layers only see a local neighbourhood and text enters at a few injection points. A transformer backbone has no local bias: every patch sees every other patch in every block, and text tokens sit in the same attention operation. Global relationships like layout, counting and spelling get much easier.
Is this the same kind of transformer as a language model?
Same building block, different job. A language model predicts the next token from the ones before it, one at a time. A diffusion transformer looks at all tokens at once and predicts how to make the whole latent slightly less noisy, then repeats. There is no left to right ordering and no causal mask.
What is MMDiT?
A multimodal variant where text tokens and image tokens run through the attention layers together with separate weights but a shared attention operation, instead of text being injected into the image stream. It is the design behind the sharp jump in prompt adherence and in-frame text quality from 2024 onward.
Why does doubling the clip length cost far more than double?
Because cost tracks token count and attention scales with the square of it. A 10 second clip has roughly twice the tokens of a 5 second one, so the attention work is closer to four times. This is also why 1080p is a much bigger jump from 720p than the pixel count suggests.
Does a transformer backbone automatically mean better output?
No. It sets the ceiling on prompt understanding and sequence length; training data, the encoder pair and the sampler decide what you actually get. A well trained older family can still beat a weak transformer model on skin texture, film look, or a specific style you need.

Related terms

Back to Models & Parameters