Reference-to-Video: How AI Keeps a Face Consistent in Every Frame

Face drift is the tell of amateur AI video. Frame one is unmistakably your friend; by frame three he is a cousin; by the last frame he is a stranger wearing your friend's shirt. Reference-to-video generation exists to kill exactly that failure — and the way it does it is more interesting than "the model looks at your photo."

This is a deep dive into identity preservation. For the surrounding pipeline, start with how AI video generation works; for the other half of input control — staging the first frame — see image-to-video first-frame conditioning.

TL;DR

• Text prompts are a lossy way to specify a face; references are not
• Reference-plate conditioning injects your image as tokens at every denoising step, so identity is continuously re-anchored instead of decided once
• References and prompt have separate jobs: identity vs. action — mixing them is the #1 user error
• More references ≠ better; one clean image per subject usually wins

Why text cannot pin a face

Say "a man with brown hair and a beard" and you have described a few hundred million people. During denoising, the model re-resolves that description at every step, and each resolution is a fresh sample from the same huge family. Adjacent frames disagree by a percent — the eyebrows sit slightly differently, the jaw is a pixel wider — and the disagreements accumulate visibly. That is drift: not one bad decision, but thousands of tiny independent ones.

The fix is structural: give the model the actual pixels, not a description of them.

Reference-plate conditioning, informally

Think of the reference as a plate — a fixed calibration target bolted next to the canvas:

1. Your photo is encoded by the model's vision encoder into a set of reference tokens.
2. These tokens are concatenated into the attention stream alongside the text tokens and the noisy latent patches.
3. At every denoising step, the layers computing "what should this face look like right now" can attend directly to the reference tokens — the same mechanism that reads your prompt, now reading your face.

The key property is recurrence. Identity is not copied once at frame one and remembered; it is re-injected continuously, so drift has nowhere to accumulate. This is the quiet difference between models that hold a face for ten seconds and models that don't — and why the same model can look inconsistent in text-to-video mode but solid in reference mode.

The contract: identity from references, action from the prompt

The strongest convention in reference-driven generation is role separation:

• References answer "who/what" — the face, the outfit, the instrument.
• The prompt answers "where and how" — the scene, the action, the camera.

Break the contract and you get mush. Describing the face in the prompt while also passing her photo forces the model to reconcile two identity sources, and the average of two faces is a stranger. Describe the face once, in the photo, and spend the prompt entirely on action. If you want to see the difference in practice, the prompt-craft examples on this site are written exactly this way.

Reference clips: motion, not just appearance

Modern reference modes also accept short reference videos — a few seconds of the motion you want. The model extracts a motion signature (gait, rhythm, weight transfer) and applies it to your reference subjects. This is how a still photo of your grandfather inherits an entire stage routine — the sway, the lean into the microphone: appearance from the image, choreography from the clip, stitched by the same attention layers.

The failure mode to know: when reference motion and prompt action contradict, motion usually wins — the clip is a stronger signal than the sentence. Keep them saying the same thing.

Practical staging rules

• One clean image per subject. Front-facing, well-lit, unobstructed. More references of the same subject mostly add contradiction.
• Match reference crop to target framing. A face-only reference will not reliably reconstruct full-body proportions; give the model the body you expect to see.
• References dominate style. A photo shot at night will pull the generated scene toward night. Pick reference lighting that matches the scene you asked for.
• Expect a motion tax. Locked identity spends some model capacity on consistency; wild full-body motion is where drift creeps back. Shorter clips, stronger models.

Try it

The home-page generator's photo mode is reference-to-video underneath: upload one photo (or two, for a duo clip), and the step-by-step tutorial shows the exact settings.

FAQ

Why do faces change or drift in AI-generated video?

Because a text-to-video model has no persistent record of the face — it re-decides appearance at every denoising step, and small disagreements between steps accumulate into visible drift. Reference conditioning fixes this by injecting an actual image of the face as a steering signal at every step.

What is reference-to-video generation?

It is a generation mode where the model receives one or more reference images (and optionally reference video clips) alongside the prompt, and reproduces the subjects in them inside a new scene. Identity comes from the references, action comes from the prompt — the two roles are separated on purpose.

How many reference images can reference-to-video use?

Current-generation models accept up to nine reference images plus a few reference video clips. Practically, one clean photo per subject beats a pile of similar ones — extra references mostly add contradiction, not information.