What sits in the shared embedding space
Train a network on hundreds of millions of image and caption pairs, rewarding it when a caption and its picture end up near each other and penalising it when they do not, and you get one useful artefact: a coordinate system where the phrase golden hour lands next to actual photographs taken at golden hour. That coordinate system is what a CLIP model gives you.
Two consequences follow. First, you can measure how well an image matches a caption by comparing positions, which is the CLIP score used in benchmarks and in some automatic reranking. Second, and this is what you use every day, the text encoder can convert a prompt into a vector that a generator knows how to follow.
Why prompts behave the way they do
Most prompting folklore is downstream of one property. Contrastive training rewards getting the overall gist of a caption right, so the encoder ends up treating a sentence closer to a weighted bag of concepts than to a parsed grammatical structure. Three familiar symptoms come out of that.
- Attribute bleed. A yellow raincoat next to a red bicycle can arrive as a red raincoat. The colours are in the embedding, the bindings are loose.
- Negation failures. No text and without a hat both add their nouns rather than removing them.
- Weak spatial logic. On top of, behind, and to the left of are frequently traded for each other, because the phrase carries less signal than the objects.
The fix that actually works is separation. Put one attribute per subject, keep the subject count low, and use the model's own controls (a negative prompt, weighting syntax, an inpaint pass) for anything the sentence cannot pin down.
The 77 token window and what replaced it
The original encoder reads 77 tokens and stops. Interfaces hide this by chunking longer prompts and blending the chunks, which is why a 200 word prompt on an older checkpoint tends to produce a vague average rather than a precise scene.
Current flagship models mostly moved on. They use a T5 encoder or a full language model as the text encoder, sometimes running CLIP in parallel for its visual grounding. The window goes from 77 tokens to several hundred, and the encoder has real syntax, so a written sentence outperforms a comma-separated tag list. If you learned to prompt on SD 1.5, this is the single habit worth unlearning.
Telling the encoder apart from the model
When a result is wrong, work out which half failed. Wrong content that you clearly asked for, swapped attributes, dropped clauses, ignored spatial words: that is text encoding, and it responds to shorter and better separated prompts. Correct content rendered badly, mangled hands, mushy textures, unstable motion: that is the generator and its training data, and no rewrite of the prompt will fix it. Knowing which one you are looking at is the difference between three test generations and thirty.