Models & Parameters

Fine Tuning AI Models: What Actually Changes the Weights

Also called dreambooth, textual inversion, checkpoint model, stable diffusion checkpoint, custom model training

Fine tuning AI models means continuing training a pretrained model on your own images so it learns a subject, style, or product it did not know before. It changes the weights, which is what separates it from prompting and from reference images, and it is the last resort rather than the first.

Try the cheaper rungs first

There is a ladder here, and skipping to the top wastes both money and time.

  1. Prompt harder. If the model already knows the concept, rewriting words costs nothing.
  2. Give it a picture. A reference image, a structural map, or a first frame transfers appearance without touching a single weight.
  3. Train a LoRA. Small, cheap, an hour or less of compute, and reversible because it loads on top of any compatible base.
  4. Fine tune the whole model. Full DreamBooth or a checkpoint model rewrite. Expensive, slow, and worth it only when a LoRA has already been tried and fell short.

Most people who think they need training need step two. The honest test is whether a good reference image gets you 80 percent of the way, because if it does, the remaining 20 percent rarely justifies a training run.

What the words mean

Checkpoint model is the full set of weights, the multi gigabyte file people mean by a stable diffusion checkpoint. Fine tuning one produces another one.

DreamBooth is a method for teaching a specific subject with a rare token as its name, plus class images that stop the new knowledge from swallowing the general category.

Textual inversion learns an embedding rather than weights: a new word aimed at what the model can already do.

LoRA trains low rank adjustment matrices instead of the full weights, which is why it is small and stackable.

The dataset decides the outcome

Training is mostly a data problem, and the failures are predictable.

  • Variety is the requirement. Same lighting in every shot and the model learns the lighting. Same background and it learns the background.
  • Resolution and sharpness matter more than count. Soft or upscaled images teach softness.
  • Caption consistently. Use one rare identifier for the subject and describe what varies, not what stays the same.
  • Cut near duplicates. Consecutive video frames are effectively one image repeated, and repetition is how overfitting starts.

Reading an overfitted result

Overfitting looks like a subject that arrives correct but frozen. Same three quarter angle, same haircut lighting, a chunk of the training background reappearing, and the prompt unable to move any of it. Undertrained is the opposite: the subject only half resembles itself and the identifier behaves like an ordinary adjective.

Check both by generating at several checkpoints during training rather than judging only the last one. Fine tuning AI models is an experimental process, and the best epoch is frequently not the final epoch.

The prompt for this

A starting point that reliably produces the effect. Adjust the subject and setting; keep the technical clauses.

A portrait of sks man in a charcoal wool coat, overcast daylight, 85mm, shallow depth of field

Try Fine-Tuning yourself

Open the generator with a starting point already filled in.

AI Image GeneratorAI Image Editor

Frequently asked questions

What is the difference between a LoRA and a fine tuned checkpoint model?
A full checkpoint model rewrites every weight and ships as a file of several gigabytes. A LoRA trains a small set of adjustment matrices and ships as a few dozen megabytes that load on top of a base model. For a single face, style, or product the LoRA is almost always the right choice.
How many images do I need?
For one face, 15 to 30 is the working range, and quality beats quantity every time. Vary the lighting, distance, angle, and background, keep only sharp images, and remove near duplicates. Ten excellent varied photographs train better than eighty frames pulled from one video.
What is textual inversion and is it still worth using?
Textual inversion learns a new word rather than new weights: it finds an embedding that points at a concept the model can already render. Files are tiny, a few kilobytes, but it cannot teach anything genuinely absent. It survives mainly for style tokens where its weakness does not matter.
How do I know if training overfitted?
The subject comes back in the same pose, the same crop, or with a background from the training set bleeding in, and the prompt stops being able to change clothing or setting. Fix it by training fewer steps, lowering the learning rate, or adding variety to the dataset, in that order.
Do I need to train anything to keep a character consistent?
Usually not. A reference image, a first frame, or an identity adapter handles most character consistency work at zero training cost. Training earns its keep when you need the same subject across dozens of shots, in poses and outfits no single reference can cover.
What does DreamBooth do that plain fine tuning does not?
It adds prior preservation: while learning your subject it keeps generating examples of the general class, so learning one person does not overwrite the model's idea of people. That is what stops every prompt containing the word man from returning your subject.

Related terms

Back to Models & Parameters