Where the VAE sits in the pipeline
Denoising at full resolution would be unaffordable. A 1024x1024 image is a million pixels; five seconds of 24fps video is over a hundred million. So a diffusion model does not operate on pixels at all. A VAE encoder first squeezes the image down to a latent grid, usually 8x smaller on each spatial axis, every denoising step runs on that grid, and a VAE decoder expands the result back out at the end.
Two things follow from that arrangement. The backbone (a UNet or a diffusion transformer) never sees your image at full resolution, so it cannot reason about anything finer than one latent cell. And the decoder is the last thing to touch every pixel you ship, which makes it responsible for a specific, recognisable family of defects that no amount of prompt rewriting will remove.
Channel count is the detail budget
Spatial compression is only half the story. Each latent cell also carries a fixed number of channels, and that number is the real ceiling on fine detail.
The first generation of Stable Diffusion models used 4 channels per cell. Flux, SD3, Qwen-Image and the current video families moved to 16. Same 8x spatial squeeze, four times the information per cell, and the difference is exactly where you would expect it: legible signage, jewellery, teeth, eyelashes, embroidery, thin architectural lines. If you have ever wondered why models from 2024 onward suddenly stopped mangling short text strings, the redesigned VAE deserves as much credit as the bigger backbone.
This is also why upscaling and generating large are not the same move. Generating at 1536px gives small features more latent cells to live in during denoising. Upscaling afterwards can only hallucinate detail into pixels that were already committed.
Video adds a time axis to the compression
Image VAEs compress two dimensions. Video VAEs compress three. A typical 3D causal VAE folds 4 input frames into 1 latent frame on top of the 8x8 spatial squeeze, so a 5 second clip becomes a very small tensor, which is the only reason video generation is affordable at all.
The cost shows up as temporal detail loss. Motion faster than the temporal window gets averaged rather than resolved, which is the smear you see on whip pans, on spinning wheels, and on hands at the fast end of a gesture. Flicker on fine texture (hair, foliage, fabric weave) is the same effect running the other way: the decoder reconstructs high-frequency detail slightly differently on each latent frame, and your eye reads the variance as shimmer.
Reading VAE artifacts in your own output
Worth learning to recognise, because it tells you whether to change the prompt or change the plan:
- Small features soft while large features are crisp: spatial compression. Reframe or generate bigger.
- Text legible at 20 characters, garbled at 60: same cause. Fewer, larger words.
- Faint grid or checkerboard on smooth gradients (skies, studio backdrops): decoder upsampling artifact.
- Saturation shift, usually reds and skin tones running warm: reconstruction bias, correct it in grading.
- Halos or ringing on hard high-contrast edges: also reconstruction, most visible on white text over dark.
- Uniformly waxy skin and fabric: high-frequency content lost in compression.
None of these respond to negative prompts, because the prompt influences the backbone and the artifact is downstream of it. The remedies are structural: pick a model family with a 16-channel VAE for detail work, compose so that what matters is physically large in frame, generate at the top of the model's comfortable resolution range, then add grain and micro-contrast during grading rather than asking the model for them.