The step nobody shows you
Between your prompt and the model sits a lookup table. The tokenizer holds a fixed vocabulary, typically 30k to 100k entries, of characters, word fragments and whole common words. Your text is greedily matched against it, and the result is a list of integers.
weathered brass diving helmet might become five or six tokens. Zooop becomes something like zo + op + p, because it is not in the vocabulary. Each id then indexes an embedding table, and those vectors are what the text encoder processes into the conditioning signal that steers generation.
Everything creators notice about prompt behaviour traces back to this step, and none of it is visible in the interface.
The token budget, and why it varies by model
The limit that bites hardest is 77 tokens, inherited from CLIP. It covers roughly 60 to 70 English words, and two of those 77 slots are reserved markers. Text beyond the limit is not summarised or compressed; it is dropped, without a warning.
That single fact explains a very common experience: you write a detailed 150-word prompt, and the model appears to obey the beginning and ignore the end. It did not ignore the end. It never received it.
Newer architectures changed the ceiling. Models built on T5 or an LLM text encoder accept hundreds of tokens and genuinely reward longer, more structured descriptions. So the correct prompt length is a property of the model, not a universal rule, and a prompt tuned for one family can behave very differently on another.
Practical implications, in order of how often they matter:
- Front-load. Put subject, action and framing first. If anything is going to be truncated or under-weighted, let it be the trailing style adjectives.
- Spend tokens on nouns, not filler. Articles, conjunctions and pleasantries all consume budget.
- Non-English prompts cost more. Scripts that are underrepresented in the vocabulary split into many pieces, so the same content eats two to three times the budget. On CLIP-era models, an English prompt often has a real advantage for that reason alone.
- Emoji, unusual unicode and long hyphenated strings are expensive and rarely carry the meaning you intended.
Why rare words and brand names fail
If a word is not in the vocabulary, it becomes fragments, and the model has no concept attached to those fragments. It will still produce something, assembled from whatever associations the pieces carry. This is the mechanism behind results that seem to come from nowhere when you name a specific product, an obscure location, or a made-up character.
Three ways around it:
- Describe rather than name. Shape, material, colour, era. The model has strong concepts for all of those.
- Use a reference image. One image beats any amount of naming for an object that has to look exactly right.
- Train the token. A textual embedding or a LoRA attaches a real concept to a chosen trigger word, which is exactly the problem being solved here.
The other artifact of tokenization
Attribute bleed. Write a red cube and a blue sphere and you will regularly get a blue cube. The encoder produces a single conditioning vector for the whole prompt, and the association between a colour token and an object token is much looser than the grammar of your sentence suggests.
Reliable workarounds: describe one subject per prompt and compose the rest afterward, use regional conditioning or a mask if your tool supports it, or put the more important pairing first, since early tokens carry more weight. Reordering the same words is often enough to flip which object wins the colour.