What stays and what changes
The reason to start from footage is everything you do not want the model to invent: the performance, the camera move, the timing of a gesture, the cut points. All of that is expensive or impossible to specify in words. Handing it to the model as a video gets you all of it at no cost.
What you change is the rendering: the medium, the palette, the lighting, an object in frame, a background. A useful mental model is that the source supplies the when and the where, and the prompt supplies the what it looks like.
Three families worth telling apart
- Restyling. The whole frame is re-rendered in a new medium or grade. Loosest control, largest visual change, highest risk of losing detail you cared about.
- Structure-guided. The model is conditioned on extracted signals such as depth, pose, or edges rather than on raw pixels. This is how you preserve layout and body position while changing the look substantially.
- Prompt-based editing. A targeted change ("make the coat leather", "remove the parked car") applied consistently through the clip, usually with a mask. Narrowest and most reliable.
Most disappointment with video to video ai comes from reaching for the first family when the third one was the actual request.
Temporal consistency is the entire problem
Any per-frame image edit looks correct in a still and wrong in motion, because the model resolves each ambiguous region independently and its choices differ frame to frame. Fabric texture crawls, hair reshuffles, and background detail boils. Viewers describe this as looking cheap without being able to name it.
The approaches that work all reduce the number of independent decisions:
| Approach | How it stabilizes |
|---|---|
| Temporal attention | Each frame attends to its neighbors during generation |
| Flow-guided warping | Styled detail is carried forward along estimated motion |
| Keyframe propagation | A few frames are styled, then spread across the clip |
| Masked edits | Only a small region is re-rendered at all |
The practical corollary: ask for less change and you get more stability. A clip restyled 40 percent of the way holds together; the same clip pushed to a total reinterpretation boils.
Preparing footage that survives the pass
- Split on every cut. One shot per run, always.
- Prefer stable, well-exposed source. Handheld shake, heavy compression, and rolling shutter all feed the model bad motion cues.
- Trim to what you need. Cost usually scales with frames or seconds, so a three-second run beats a ten-second run you will re-do.
- Consider frame rate. Processing at 12 to 16fps and interpolating afterward is cheaper and often steadier than processing every frame of 30fps footage.
- Mask what must not change. Faces, logos, and typography are exactly the elements a re-render damages first.
- Test on the hardest two seconds. The fastest motion in the shot tells you whether the settings hold. The easy part always looks fine.
Where it still falls down
Fast subject motion, heavy motion blur, small faces, on-screen text, and shots where an object leaves and re-enters frame remain the reliable failure cases. If your shot has all five, no amount of prompt refinement will rescue it. The alternative is to generate the shot rather than convert it, or to isolate the subject with a matte and rebuild the frame in layers, which trades one uncontrollable pass for several controllable ones.