What it adds to a diffusion model
A plain prompt gives you no control over layout. You can describe a low angle shot of a figure with an arm raised and get a different composition on every run. ControlNet closes that gap: a copy of the model's encoder is trained to accept a second input, a structural map, and to inject it at every denoising step.
The map is extracted from an image you provide by a preprocessor. An openpose preprocessor reduces a photo to a stick figure. A depth preprocessor turns it into a greyscale distance field. A canny preprocessor reduces it to white outlines on black. What survives the preprocessing is what gets enforced, which is the whole design: throw away everything you want the model to reinvent, keep only the part you want held.
Picking the preprocessor
| Preprocessor | Preserves | Use it for |
|---|---|---|
| Openpose | Joint positions | Reposing a figure, keeping a gesture |
| Depth | Spatial layout, volume | Interiors, camera geometry, product staging |
| Canny / lineart | Exact outlines | Logos, packaging, mechanical shapes |
| Scribble / softedge | Loose shapes | Your own sketch as a layout |
| Tile | Local detail | Upscaling without inventing new content |
The most common mistake here is reaching for canny by default. Canny copies the silhouette exactly, so if the source is a photo of a person, the generated person keeps that person's proportions and haircut. When you want the pose but not the body, openpose is the right tool.
The three numbers that matter
Control weight sets how loudly the map speaks, usually 0 to 2. Real working range is 0.5 to 0.9. Higher is not more accurate, just more rigid.
Guidance start is the fraction of the denoise where conditioning begins, normally 0. Leaving it at 0 is right for layout, since composition is decided in the earliest steps.
Guidance end is where it stops, and it is the underused one. Setting it to 0.6 or 0.7 hands the last third of the process back to the model, which is where shading, texture, and grain get added. This single change fixes most complaints about traced looking output.
When reaching for ControlNet is a waste of time
It cannot add knowledge. If the checkpoint cannot render a convincing horse, a perfect depth map of a horse produces a well shaped bad horse. It also cannot rescue a contradictory prompt: ask for a seated figure while feeding a standing openpose skeleton and you get a broken compromise, usually bent at the hips.
On hosted platforms you often do not see a ControlNet toggle at all. The equivalent surface is a reference image slot, sometimes labelled structure reference or style reference, with the preprocessor chosen for you. The tuning instinct still transfers: if the output feels copied, weaken the reference or shorten how long it applies, and if it feels ignored, the map itself is probably too vague.