Video Agent Generation: Diffusion Based Model

A novel approach for powerful, controllable human-like video generation.

Despite notable progress in the field [75, 76, 81], the generation of natural, lifelike talking faces remains a challenge. While current methods excel at lip synchronization [11], they often fall short in capturing expressive facial dynamics and head movements, both of which are critical to realism. Our framework addresses this gap by integrating a diffusion-based model that conditions on both visual and audio inputs to generate expressive talking face videos from a single portrait image.

Our pipeline comprises several key components: an audio processor, a video processor, a landmark processor, an appearance generator, and a warp model. Together, these components ensure that the generated video is realistic, expressive, and temporally coherent.

Figure 11: Pipeline for Talking Head Generation. Starting from a static portrait image, a pre-trained landmark detector extracts facial landmarks and key points. Input audio or video is then encoded into a latent space, conditioning a diffusion-based appearance generator to produce a sequence of landmarks. Finally, these landmarks are warped to the portrait image to synthesize the output video.

Audio Processor. The audio processor transforms the input audio A into a latent feature representation, capturing essential speech characteristics that guide lip synchronization and facial expressions. The audio input, represented as a waveform or mel-spectrogram, is passed through an encoder network E_{audio}, producing latent audio features F_a:

$$F_a = E_{\text{audio}}(A; \theta_{\text{audio}})$$

where \theta_{audio} are the parameters of the audio encoder. The latent features F_a encode speech patterns such as phonetic content and prosody, enabling the system to generate lip movements and facial dynamics synchronized with the spoken words. These features are crucial for aligning lip articulation and other facial behaviors with the temporal progression of the speech.

To ensure robust lip synchronization, the system minimizes the distance between the generated lip features and the ground-truth lip positions in a latent feature space, leading to an optimal match between audio cues and visual expressions.

Figure 12: Showcases. Examples of generated talking face videos, demonstrating the system's ability to produce realistic facial expressions and movements based on the given source video input.

Video Processor. The video processor is responsible for capturing visual dynamics, such as head movements and facial expressions, from an input video sequence V_{in}. For scenarios where the system is provided with an initial video clip, the video processor extracts temporal features, including changes in head orientation, gaze direction, and facial expressions, through an encoder E_{video}, yielding latent video features F_v:

$$F_v = E_{\text{video}}(V_{\text{in}}; \theta_{\text{video}})$$

where \theta_{video} represents the parameters of the video encoder. These latent video features encapsulate dynamic visual information from the input sequence, enabling the generation of smooth and realistic transitions in head movement and facial dynamics over time. Additionally, the temporal features ensure consistency across frames, allowing the system to generate coherent sequences even for long-duration videos.

The integration of these temporal dynamics with the facial features ensures that the generated video not only synchronizes lip movements with audio but also exhibits natural head movements, gaze shifts, and expression changes, matching the flow of the video input.

Landmark Processor. The landmark processor takes a static portrait image I_s and extracts facial landmarks and keypoints corresponding to essential facial features like the eyes, mouth, and nose. These landmarks provide the foundational structure for generating facial expressions and head movements. Let L_{landmark} represent the set of facial landmarks extracted by the processor, which can be expressed as:

$$L_{\text{landmark}} = E_{\text{landmark}}(I_s; \theta_{\text{landmark}})$$

where \theta_{landmark} denotes the parameters of the landmark extraction model. These keypoints L_{landmark} serve as control points for the subsequent generation of facial dynamics. The landmarks are further modulated by the audio and video latent features to ensure that facial expressions and movements correspond accurately to the input modalities.

By using keypoints to structure the generation of facial movements, the system maintains geometric consistency in facial appearance and ensures that critical features such as eye positions and mouth shape are preserved throughout the video.

Appearance Generator. The appearance generator synthesizes facial expressions and movements by leveraging the latent features F_a from the audio processor and F_v from the video processor. Utilizing a diffusion-based approach, the appearance generator refines these facial dynamics iteratively, producing smooth and continuous facial motions. The appearance generator uses the extracted landmarks L_{landmark} as control points, synthesizing dynamic facial features through the function G_{appearance}:

$$L_{\text{appearance}} = G_{\text{appearance}}(L_{\text{landmark}}, F_a, F_v; \theta_{\text{appearance}})$$

where \theta_{appearance} are the generator parameters, and L_{appearance} denotes the appearance-modulated landmarks. These generated landmarks control key facial regions, such as the mouth for lip movements, the eyes for gaze tracking, and the overall head pose for natural head orientation adjustments.

The diffusion-based mechanism ensures that these facial movements are refined across time steps, gradually enhancing the smoothness and natural appearance of the generated expressions.

Warp Model. Once the appearance-modulated landmarks L_{appearance} are generated, the warp model W applies these dynamic facial movements to the static portrait image I_s. The warp model deforms the facial features of the portrait image to match the generated dynamics, ensuring that the facial expressions and head movements appear natural and realistic. This can be formalized as:

$$V_g = \mathcal{W}(I_s, L_{\text{appearance}})$$

where V_g is the final generated video. The warp model handles critical tasks such as adjusting the head pose, synchronizing lip movements with the input audio, and maintaining continuity in facial expressions across frames. The warp model ensures that the generated facial dynamics align seamlessly with the target subject's facial geometry, preserving the individual's identity while introducing dynamic expressions.

Conclusion. This framework represents a significant advancement in generating expressive talking faces by seamlessly integrating audio, video, and static portrait images. The system focuses on the nuances of facial dynamics, natural head movements, and accurate lip synchronization, enabling the production of high-quality, lifelike video outputs. This approach offers promising applications in human-computer interaction, entertainment, digital media, and healthcare, particularly in improving communication and accessibility.

Audio-Driven Talking Faces Generation

In multimedia and communication, the human face serves as a dynamic canvas, where each subtle movement and expression conveys emotions and unspoken messages, and fosters empathetic connections. The advent of AI-generated talking faces opens a window to a future where technology enhances the depth of human-to-human and human-AI interactions. This innovation holds the potential to enrich digital communication [41, 62], improve accessibility for individuals with communication impairments [26], and revolutionize education through interactive AI tutoring [2, 21, 61].

Motivation To move closer to realizing these capabilities, we present a novel method for generating highly realistic and dynamic talking faces from audio input. Using a single static image of any individual and an accompanying speech audio clip from any speaker, our approach efficiently creates a hyper-realistic video. This video not only achieves precise lip synchronization with the audio but also captures a broad spectrum of natural facial expressions and head movements, resulting in a lifelike portrayal.

Setting Let’s denote the source image as (I_s), the driving audio as (A_d), and the generated video as (V_g). The process can be mathematically described as:

$$V_g = f(I_s, A_d; \theta)$$

where (f) is a generative function parameterized by (\theta), which represents the parameters of our model.

For a more detailed discussion, let’s break this process into a few key steps: encoding, transformation, and decoding.

Encoding The source image (I_s) and the driving audio (A_d) are passed through an encoder network (E), which extracts deep feature representations. This can be formulated as:

$$F_s = E(I_s; \theta_E), \quad F_d = E(A_d; \theta_E)$$

where (F_s) and (F_d) are the deep feature representations of the source image and driving audio, respectively. (\theta_E) denotes the parameters of the encoder.

Transformation Next, a transformation function (T) maps the source image features (F_s) and driving audio features (F_d) to a new set of features (F_t) that retains the identity from (F_s) and the expressions from (F_d). This can be expressed as:

$$F_t = T(F_s, F_d; \theta_T)$$

where $\theta_T$ are the parameters of the transformation function.

Figure 13: Pipeline for audio-driven talking head generation.

Decoding Finally, a decoder function (D) is used to generate the output video frames (V_g) from the transformed features (F_t). This can be described as:

$$V_g = D(F_t; \theta_D)$$

where $\theta_D$ are the parameters of the decoder.

Method We first capture the overall facial dynamics and head movements within a latent space influenced by audio and supplementary signals. These motion latent codes are then used to generate the final video frames, leveraging a face decoder. The decoder also integrates appearance and identity features derived from the input image through a face encoder.

The process begins with the creation of a dedicated latent space for facial features, alongside the training of both the encoder and decoder. Our approach utilizes a disentangled learning framework that is trained on real-world face videos to ensure expressiveness and separation of features. To model the motion distribution, we employ a Diffusion Transformer, which, during inference, generates the motion latent codes based on audio input and other conditions, enabling lifelike facial motion and expression in the final output.

Step 1 Given a corpus of unlabeled talking face videos, the objective is to build a latent space for human face features that enables disentanglement and expressiveness. The disentanglement facilitates effective generative modeling of human head and facial behaviors on various videos while retaining subject identities. It also enables fine-grained control over facial factors, which is desirable for multiple applications.

To achieve this, we base our model on a 3D-aided face reenactment framework [11, 65]. The 3D appearance feature volume provides a better characterization of the appearance details compared to 2D feature maps. We decompose a face image into a 3D canonical appearance volume V^{app}, identity code z^{id}, head pose z^{pose}, and facial dynamics z^{dyn}. These are extracted using independent encoders. For appearance volume V^{app}, it is constructed by rigid and non-rigid 3D warping [11].

We use latent variables from different frames to construct a motion transfer loss. To improve disentanglement between head pose and facial dynamics, several additional losses are introduced:

• A pairwise head pose and facial dynamics transfer loss.

• Identity similarity loss to minimize discrepancies.

Let I_s and I_d be two frames from different subjects. We transfer the motion of I_d to I_s and calculate:

$$\hat{I}_s^{z^{pose}_d, z^{dyn}_d} = D(V^{app}_s, z^{id}_s, z^{pose}_d, z^{dyn}_d)$$

A cosine similarity loss between the deep features of I_s and \hat{I}_s is applied.

Step 2. Our method represents motion sequences as a time series of pose, dynamics, and bone length parameters extracted from video data. We utilize a pre-trained Wav2Vec2 [1] to extract rich audio features from corresponding audio clips.

At the core of our framework lies a diffusion model trained to generate realistic motion sequences conditioned on the input audio. This model operates in two stages: a forward process gradually injects Gaussian noise into the target motion data, while a reverse process, parameterized by a transformer network, learns to recover the original motion data from the noisy input. The network is trained using a denoising score-matching objective, minimizing the discrepancy between predicted and actual noise distributions.

To enhance controllability and realism, we introduce a set of conditioning signals beyond the audio features. These include gaze direction, head-to-camera distance, and emotional valence extracted from training data. Furthermore, we incorporate a mechanism for emotion control, allowing users to infer emotion from the audio or explicitly specify it as an input parameter. We condition the model on the last K frames of the preceding window to ensure smooth transitions between generated segments.

Our proposed method offers several advantages over existing techniques. The use of diffusion models enables the generation of high-fidelity motion sequences exhibiting natural and fluid movements. Incorporating multiple conditioning signals allows for fine-grained control over various aspects of the generated video, including gaze, head pose, and emotional expression. Finally, conditioning on previous frames ensures temporal coherence and smooth transitions within the generated video sequence.

Inference During inference, we begin with an arbitrary face image and an accompanying audio clip. The initial step involves extracting the 3D appearance volume and the identity code utilizing our pre-trained face encoders. Following this, we extract audio features, dividing them into segments of length W. Subsequently, we generate the sequences of head and facial motion iteratively through a sliding-window approach with our trained diffusion transformer H. Finally, the complete video is produced using our trained decoder.

Conclusion The proposed approach aims to improve TTA performance under limited data and computational resources by integrating a pre-trained LDM. This method demonstrates effective cross-modal alignment and high-quality audio generation, even in constrained environments.

Lip Synchronization in Talking Head Videos

In the realm of audio-driven talking head generation, \textit{lip synchronization} is critical to ensuring the realism of the generated video. Accurate lip movements synchronized with the speech input significantly enhance the perceived quality of the video and contribute to a more immersive experience. The challenge in lip synchronization lies in generating lip movements that not only match the phonetic content of the speech but also exhibit natural dynamics, smooth transitions, and temporal coherence across video frames.

Mathematical Formulation of Lip Synchronization

The process of lip synchronization can be formalized as a mapping between the input speech audio and the lip movements of the target subject. Let the speech input be represented as an audio clip A_d, and the target video frames containing the synchronized lip movements as V_g^{lip}. The objective is to generate lip movements that match the timing and phonetic content of A_d, ensuring that the lip shapes evolve naturally over time.

Let the set of audio features extracted from A_d be denoted as:

$$F_a = E_{\text{audio}}(A_d; \theta_{\text{audio}})$$

where E_{audio} is the audio encoder and \theta_{audio} represents its parameters. The lip region of the target face is encoded as:

$$F_l = E_{\text{lip}}(I_s; \theta_{\text{lip}})$$

where E_{lip} is the lip region encoder applied to the source image I_s.

The task of generating synchronized lip movements can then be modeled as the generation of a set of dynamic lip features F_t^{lip}, conditioned on the input audio features F_a. This transformation can be expressed as:

$$F_t^{\text{lip}} = T(F_l, F_a; \theta_{\text{sync}})$$

where T is the transformation function that maps the static lip features F_l and audio features F_a to dynamic lip movements F_t^{lip}, and \theta_{sync} denotes the parameters of the synchronization model.

Lip-Sync Loss Function To enforce lip synchronization, a specialized lip-sync loss is introduced, inspired by discriminators such as those used in Wav2Lip. This loss computes the similarity between the lip movements and the phonetic content of the speech over time. The lip-sync loss L_sync can be formalized as:

$$L_{\text{sync}} = - \log \left( \frac{F_v^{\text{lip}} \cdot F_a}{|F_v^{\text{lip}}|_2 \cdot |F_a|_2} \right),$$

where F_v^{lip} represents the video embedding of the lip region, and F_a is the audio embedding. The cosine similarity between these two embeddings ensures that the generated lip movements are temporally and semantically aligned with the speech.

Lip Dynamics and Temporal Coherence Beyond matching the phonetic content of the speech, the generated lip movements must also exhibit natural dynamics and temporal coherence. To achieve this, we introduce a temporal consistency loss L_{temp}, which minimizes the difference between consecutive frames of the generated lip movements, ensuring smooth transitions over time. This is especially important for maintaining realistic lip motions, as abrupt changes would degrade the visual quality of the video.

The temporal consistency loss is defined as:

$$L_{\text{temp}} = \sum_{t=1}^{T-1} \left| F_{t+1}^{\text{lip}} - F_t^{\text{lip}} \right|^2$$

where T represents the total number of frames, and F_t^{lip} and F_{t+1}^{lip} are the lip features at consecutive time steps t and t+1, respectively.

Generation of Lip-Synchronized Video. Once the lip-synchronized features F_t^{lip} are obtained, they are combined with other facial features (such as eye movements, head pose, and expression dynamics) to generate the final video frames V_g^{lip} using a video decoder D. The decoding process can be described mathematically as:

$$V_g^{\text{lip}} = D(F_t^{\text{lip}}; \theta_D)$$

where \theta_D represents the parameters of the video decoder, and V_g^{lip} is the output video sequence with synchronized lip movements.

Training Objective. The overall training objective for lip synchronization combines the lip-sync loss and the temporal consistency loss, ensuring that the generated lip movements are both temporally coherent and in sync with the input audio. The total loss function is given by:

$$L_{\text{total}} = \lambda_{\text{sync}} L_{\text{sync}} + \lambda_{\text{temp}} L_{\text{temp}}$$

where \lambda_{sync} and \lambda_{temp} are weights that control the relative importance of lip synchronization and temporal consistency.

Conclusion The proposed method for lip synchronization in audio-driven talking head videos leverages deep feature representations of both audio and facial data to generate dynamic and temporally coherent lip movements. By employing a specialized lip-sync loss and a temporal consistency loss, we ensure that the synthesized lip movements closely match the phonetic content of the speech, while also exhibiting natural dynamics over time. This approach is a key component of generating lifelike talking head videos, enhancing realism and expressiveness in human-AI interactions.