What these models are actually reading
A diffusion model does not receive your sentence. A text encoder converts it into a vector, and that vector steers the denoiser toward images whose learned description is nearby. Everything practical follows from that.
It means the model is doing matching, not obeying. It has no way to hold "do not" as an operation, no way to keep a running plan, and no way to ask what you meant. So the whole craft is describing a finished frame precisely enough that only frames you would accept are nearby, then removing every word that does not narrow the set.
Chat models are the opposite, which is why prompt engineering advice from LLM work transfers badly. Techniques like few shot prompting and a carefully written system prompt are real and useful, but they belong to the language layer of a pipeline (an assistant that expands your idea into a shot description, or a captioner writing training data), not to the image model itself.
A prompt template that holds up
Order matters, because encoders weight earlier tokens more heavily and truncate the tail. Lead with what you are least willing to lose.
[shot size] of [subject] [action], [environment],
[lighting], [lens/optics], [grade/film stock], [motion for video]
Applied: Medium close-up of a fisherman mending a net, harbour wall, overcast diffused light, 85mm shallow focus, desaturated blue-grey grade.
What earns its place in a prompt:
- Concrete nouns and verbs. "Mending a net" beats "doing something with rope".
- Photographic specifics. Lens length, light direction and quality, time of day. These are heavily represented in training captions, so they land.
- One clear subject. Two subjects with two descriptions bleed attributes into each other.
What almost never earns its place: stacked quality words (masterpiece, 8k, hyper detailed) which mostly shift style toward a generic look, emotional appeals, and instructions about the process rather than the picture.
Techniques ranked by how much they change the frame
- Cutting words. The highest yield edit available. Delete every clause you cannot see in the output and adherence on what remains improves.
- Reordering. Move the thing you care about to the front. Same words, different result.
- Negative prompt. Effective against recurring artifacts (watermark, extra limbs, text) and useless as a way to add control the model lacks.
- Prompt weighting. Useful for rebalancing two competing elements. Keep multipliers modest.
- Style anchors. A named medium, era, or process (Polaroid, cel animation, tungsten-lit interior) is far more reliable than a stack of adjectives, and more repeatable across a sequence.
When words are the wrong tool
The single most useful judgement in prompt engineering is recognising the ceiling. If the same problem survives three rewrites, no fourth wording will fix it, and you are burning credits proving it. Reach past the prompt:
| Problem | The move that works |
|---|---|
| Look is not what you pictured | Generate the frame as an image first, then use it as a first frame |
| Character changes between shots | Reference image, fixed seed, or a trained LoRA |
| Composition keeps drifting | A control input or a mask, not more adjectives |
| Text on signs is garbled | Render it clean in post, or use a model with a strong text encoder |
| Motion is wrong | One camera move per clip, and separate shots that need two |
The workflow habit that separates fast operators from slow ones: keep a small library of prompts that worked, with the model version and seed attached. A prompt template you have already validated on a given model is worth more than any general rule, because it encodes that model's quirks, and every model has its own.