How deepfakes make one face move like another

Separating a face’s motion from its appearance

A face in a video carries two kinds of information at once: what it looks like—skin texture, features, and identifying details—and what it is doing—turning, blinking, smiling, or speaking. Many face-swap deepfakes work by treating those as partly separable signals. The movements and rough geometry of one person’s face guide an image of another person’s face.

A neural network, a computer system that learns patterns from examples, is trained to recognize how facial features change across images. Depending on the method, it may encode a face as a set of landmarks, a compact internal representation, or other learned measurements. Those measurements describe things such as mouth shape and head angle; they are not a literal 3D scan of the person.

Rebuilding the face in each frame

During generation, the system uses the driving person’s expression and pose to shape the replacement face. It then produces pixels that resemble the person whose identity is being inserted, drawing on patterns learned from images of that person. In effect, a grin or a turn can come from one performance while the visible facial appearance comes from another.

The result must fit the surrounding video. The face’s size and position need to match the head, and its color and lighting must blend with the scene. The system also has to keep the face coherent from frame to frame. If the mouth flickers, the jaw shifts unnaturally, or the boundary around the face does not match, the illusion becomes easier to spot.

Why the swap has limits

A neural network can only reconstruct details it has learned well enough to predict. A sharply turned head, a hand covering the mouth, unusual lighting, or an expression missing from the examples can leave it with little reliable information. The model may smooth over those gaps or invent plausible-looking details, which can create distortions.

That is why a convincing swap is not simply one face pasted over another. It is a sequence of generated images in which motion, identity, and the scene must agree well enough to look like one continuous performance.