Joint understanding, not a relay race
The distinction that matters is not how many boxes are in the diagram, it is whether the signals meet.
A relay pipeline captions your reference image, hands the caption to an image model, and throws the pixels away. Everything the caption omitted is gone: the exact face, the specific weave of a jacket, the way that one practical light falls. You can feel this in output that is thematically right and specifically wrong.
A joint model keeps both. Image tokens and text tokens sit in the same attention operation, so an instruction can point at something in the picture. That is the whole basis for instruction editing, for keeping a character recognisable across a series, and for reference sets where each image contributes a different thing.
Where you actually run into it
On the understanding side, quietly, in most of the tooling around generation. Prompt enhancement reads your reference before rewriting your prompt. Auto-captioning builds training and search data. Moderation looks at the generated frame rather than only at your words. Evaluation harnesses use a vision model to score whether the output matches the request, which is how large scale comparisons get done at all.
On the generation side, visibly, in four places:
- Image plus instruction in, image out. Conversational editing, and the reason character consistency no longer requires training anything.
- Multiple references with stated roles: identity from one, wardrobe from another, lighting from a third.
- Image plus text in, video out. The reference locks appearance so the prompt can spend itself entirely on motion.
- Video with native synchronised audio: speech, effects and room tone generated with the picture rather than assembled after it.
Prompting a model that can see
Once a reference image is attached, the prompt stops being a description and becomes an instruction. That change is worth making deliberately, because most people keep writing descriptions and then wonder why nothing moved.
- Name what changes, then name what must not. "Same face, same jacket, same framing" does real work.
- One change per generation. Two simultaneous edits is how you get a third unrequested one.
- Do not re-describe what the reference already shows. Those words compete with the pixels and usually lose, or worse, half win.
- Give each reference a job in words. Unlabelled reference stacks blend, and blending is rarely what you wanted.
- If a specific attribute keeps leaking in from a reference, crop it out. Removing the pixels is more reliable than asking the model to disregard them.
Testing one before you trust it
The single most informative experiment takes two minutes: attach a reference and give an instruction that contradicts it. Ask for a red coat when the reference shows blue. Whatever the model does tells you its weighting, and therefore how to write for it.
Then check the three failure modes that only multimodal AI has. Modality imbalance, where one input dominates and the other is decoration. Reference bleed, where attributes you did not ask for come along, background colour, grain, aspect ratio, a logo. And cross-modal drift in longer video, where audio and picture start together and separate by the end, most visible on lip movement past the first few seconds.
None of these show up in a single-modality test, and all of them decide whether the model is usable for the work you had in mind.