The pipeline in three stages
Understanding where the failure happened means knowing which of three stages produced it.
- Extraction. A pose estimator reads the driving video and produces a per-frame track: 2D keypoints, a skeleton, a dense body map, or facial landmarks. Nothing about the driver's appearance is carried forward, only geometry.
- Retargeting. The track is remapped onto your target's proportions. A tall driver's shoulder positions mean something different on a short character, and this is where most visible damage originates.
- Generation. Frames are produced conditioned on the retargeted pose plus your identity reference, so the character keeps its design while adopting the movement.
Choosing a driving video
The driver matters more than the model. In practice:
- One subject, fully framed, for the whole clip. A limb that leaves frame produces a limb that vanishes or flails.
- Static camera. Camera motion in the driver becomes subject motion in the output, since the pipeline cannot tell the two apart.
- No cuts. Every cut is a discontinuity in the pose track, which reads as a snap.
- Moderate speed with clear silhouettes. Fast motion means blur, and blur means an unreliable pose estimate.
- Minimal self-occlusion. Arms crossing the torso, hands behind the back, and turning fully away are all points where the estimator guesses.
- Framing matched to the target. Full-body driver for a full-body character, upper body for a portrait.
Why proportions dominate everything
Noodle limbs, sliding feet, and joints that bend backward are almost always retargeting artifacts, not generation artifacts. The pose track is a set of coordinates, and if those coordinates were produced by a body shaped differently from your character, no amount of prompting fixes the mismatch.
Two concrete habits help a lot. Match the reference image's pose to the driver's first frame, so the sequence starts from agreement rather than a jump. And pick a driver whose build resembles your character, which matters more than whether the motion itself is impressive.
What transfers well and what does not
Body motion, timing, and weight shift transfer well, and that is the whole point. Hands are the standard casualty, since fingers occupy few pixels and the pose track for them is sparse. Fine facial performance rarely survives, so pair motion transfer with a separate mouth pass when the character speaks. Cloth and hair are generated rather than simulated, so they follow the body loosely and can lag or clip.
It also helps to keep the three related tasks distinct. Motion transfer drives a subject's performance from a driving video. Lip sync drives only the mouth, from audio. Camera motion control specifies how the camera moves, not what the subject does. Reaching for the wrong one of the three is a common way to spend a lot of credits on the wrong problem.