The same mechanism, a different symptom
In language models, hallucination is usually explained as a made-up citation. The mechanism is identical in visual models, and the symptoms are just easier to see. A diffusion model has learned what convincing images look like. It has not learned that hands have five fingers, that reflections obey geometry, or that letters spell words. It reproduces those things only to the extent that they were statistically consistent in its training data.
Hands are the famous example because they are wildly variable in photographs: occluded, foreshortened, overlapping. The model learned a distribution of hand-shaped regions, so it produces hand-shaped regions. Nothing in the process counts fingers.
The failure types you will actually meet
- Anatomy. Fingers, teeth, ears, and limbs at joints. Worst in motion, in crowds, and at small sizes.
- Typography. Text on signs, labels, packaging and screens. Improving fast in newer models, still unreliable for anything more than a few words.
- Physics and causality. Water that flows wrong, shadows from absent lights, cloth that ignores gravity, objects that pass through each other in video.
- Fused objects. A strap merging into a shoulder, glasses growing out of a cheekbone, a chair leg becoming a table leg. Small overlapping details are where the model has the least signal.
- Invented detail on enlargement. The upscaler case. Not a mistake in the model so much as its actual job, applied where you wanted preservation instead.
- Style hybridisation. Two conflicting references averaged into a look that belongs to neither.
- Temporal invention in video. Objects that appear midway through a clip, or a background element that changes identity between second one and second three.
Reducing it
Ranked by how much difference they make on real work:
- Supply an image instead of a description. Reference images, a first frame, or image to image at low strength. This removes the whole region of the sample space where the invention was happening.
- Regenerate the region, not the frame. Inpainting a bad hand with a tight mask costs one small run. Rerolling the whole image throws away everything that worked.
- Keep it away from the small stuff. Put hands, text and jewellery out of focus, out of frame, or behind an object. Cheap, and it is a legitimate compositional choice rather than a compromise.
- Do text in post. Compositing real type onto a generated plate is faster and more reliable than any number of retries, and it stays sharp at any size.
- Use negative prompts for recurring artifacts. Effective against a specific repeated defect, not against structural weakness.
- Reduce enlargement per pass. Two conservative upscale passes invent less than one aggressive one.
- Check before you build on it. The expensive version of this problem is discovering an extra finger after you have already animated, graded and cut the shot.
The judgement call
Every attempt to suppress invention costs something: an extra step, a tighter constraint, a narrower result. The useful question is not whether the model is hallucinating, because it always is in the sense that everything it produces is invented. The question is whether the invented part is somewhere it matters.
For a mood board, it never does. For a product shot with a real logo, it always does. Deciding which one you are making before you start is what stops model hallucination from turning into a day of rework.