From Photo to Talking Video
How SadTalker converts a single still image and an audio clip into a realistic talking head video — the technology, step by step.
The Short Version
One photo. One audio clip. One talking video.
SadTalker takes one still photo and one audio clip and outputs a video of that photo talking — with head movement, blinking, and expression that's driven by the audio, not just lip-flapping over a static face. It comes out of a peer-reviewed research paper, not a black-box product: "SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation," published at CVPR 2023 by researchers from Xi'an Jiaotong University, Tencent AI Lab, and Ant Group.
That distinction matters, because it explains why SadTalker looks more natural than most "photo talks" tools: instead of directly generating pixels for every frame, it generates 3D motion coefficients first, and only turns those into a video at the last step.
Input A
Single still photo
Front or near-front facing face
Input B
Audio clip (.wav)
Drives expression and pose
Output
Talking head video
With natural head motion
Under the Hood
The Pipeline, Step by Step
Six model components work in sequence to turn a static face and an audio file into a natural-looking video.
The Key Difference
Why This Approach Produces More Realistic Results
Most simpler talking-photo tools work as 2D lip-sync: they detect the mouth region and warp it frame-by-frame to match audio phonemes, leaving the rest of the face frozen. SadTalker's 3D-coefficient approach changes this fundamentally.
Head motion looks natural
Pose is modeled explicitly via PoseVAE, not ignored — the head nods, tilts, and shifts the way a real person does while speaking.
Expression extends beyond the mouth
Coefficients can drive eyebrow and cheek movement, not just lip shape, giving more lifelike facial animation.
Identity is fully preserved
The final render step warps the actual source photo rather than synthesizing a new face, so lighting, background, and likeness stay intact.
Tradeoff: This pipeline has more moving parts — six model components instead of one — which is part of why local installation involves downloading several separate checkpoint files rather than a single model.
Inputs
What You Need to Provide
SadTalker's pipeline requires just two inputs, with a few optional generation flags to control output style.
| Input | Requirements |
|---|---|
| Source image | A single photo with a visible, front-or-near-front-facing face. Higher resolution source images produce cleaner results, especially with the enhancer enabled. |
| Driven audio | A .wav audio clip — this drives both the expression (ExpNet) and pose (PoseVAE) predictions. |
Generation Mode Flags
--stillStill mode
Reduces head motion for a calmer, more news-anchor-style result — often preferred for professional or corporate use.
--preprocess fullFull preprocess
Animates more of the frame — useful for full-body or half-body source images, not just tight face crops.
Context
Where This Fits vs. Other Approaches
SadTalker sits in the "audio-driven 3D-aware talking face" category of research, alongside related work like StyleHEAT and VideoReTalking. Compared to a pure lip-sync model (e.g., Wav2Lip, which SadTalker actually references as a component for accurate lip alignment), SadTalker's addition of explicit 3D pose and expression modeling is what produces head movement rather than a static talking mouth.
| Approach | Head motion | Expression depth | Identity preservation |
|---|---|---|---|
| SadTalker (3D-aware) | Natural (PoseVAE) | Full face (3DMM) | Source photo warping |
| 2D lip-sync (e.g., Wav2Lip) | None — rigid head | Mouth region only | Synthetic generation |