What the attention mechanism does
Older networks were restricted by design. A convolution only looks at nearby pixels. A recurrent network reads one step at a time and forgets. The attention mechanism removes the restriction: every token is allowed to look at every other token and decide how much it cares about each one. Self-attention is the version where a sequence looks at itself, which is how a prompt resolves its own internal references.
That is the whole trick, and it explains a class of failure you have already seen. Ask for a red glass cube on a blue velvet sphere and you sometimes get a blue cube. Nothing went wrong with the renderer. The word red spread its influence across both objects instead of landing on one. Simplifying the prompt fixes it far more often than repeating the word red does.
Why image and video models became transformers
Stable Diffusion 1.5 and SDXL denoised with a convolutional U-Net. Current flagship image and video models replaced it with a diffusion transformer, usually shortened to DiT. Three reasons drove the switch.
The first is scaling. Transformer quality improves predictably as you add parameters, data, and compute, which makes a training run a budgeting exercise rather than a gamble. The second is video. Cut a clip into patches across space and time, call each patch a token, and video becomes the same problem as text with a longer sequence. The third is text conditioning: a transformer model can be fed a much longer text embedding, which is why current models take paragraph-length prompts while older ones effectively truncated at a couple of dozen words.
What it changes about your prompts
- Full sentences beat keyword soup. Older models leaned on a CLIP text encoder with a short token window, so tag lists were the efficient format. DiT models read language, so plain descriptive sentences carry further.
- Length has a real ceiling anyway. Long prompts dilute attention. Somewhere past three or four clauses, extra instructions start getting silently dropped.
- Order still matters, less brutally. Earlier tokens keep an edge, so lead with whatever you are least willing to lose.
- Repetition is not emphasis. Saying cinematic three times mostly wastes tokens. Use weighting or a negative prompt if the model exposes one.
Where the architecture is the wrong thing to blame
Plenty of failures are not architectural. Counting objects, spelling long words, and holding a face steady across a five second clip are limitations of a specific checkpoint and its training data, not of transformers as such. If a model has never seen good hands, no prompt structure will produce good hands.
The useful takeaway is diagnostic. Attribute bleed, dropped clauses, and layout drift are attention problems, so they respond to shorter and cleaner prompts. Missing knowledge is a data problem, so it responds to a different model, a reference image, or fine-tuning. Sorting a bad result into one of those two buckets first will save you a lot of wasted generations.