Command Palette
Search for a command to run...
Style-Preserving Lip Sync via Audio-Aware Style Reference
Style-Preserving Lip Sync via Audio-Aware Style Reference
Weizhi Zhong Jichang Li Yinqi Cai Ming Li Feng Gao Liang Lin Guanbin Li
One-click Deployment of High-Quality Lip-Sync Model MuseTalk
Abstract
Audio-driven lip sync has recently drawn significant attention due to its widespread application in the multimedia domain. Individuals exhibit distinct lip shapes when speaking the same utterance, attributed to the unique speaking styles of individuals, posing a notable challenge for audio-driven lip sync. Earlier methods for such task often bypassed the modeling of personalized speaking styles, resulting in sub-optimal lip sync conforming to the general styles. Recent lip sync techniques attempt to guide the lip sync for arbitrary audio by aggregating information from a style reference video, yet they can not preserve the speaking styles well due to their inaccuracy in style aggregation. This work proposes an innovative audio-aware style reference scheme that effectively leverages the relationships between input audio and reference audio from style reference video to address the style-preserving audio-driven lip sync. Specifically, we first develop an advanced Transformer-based model adept at predicting lip motion corresponding to the input audio, augmented by the style information aggregated through cross-attention layers from style reference video. Afterwards, to better render the lip motion into realistic talking face video, we devise a conditional latent diffusion model, integrating lip motion through modulated convolutional layers and fusing reference facial images via spatial cross-attention layers. Extensive experiments validate the efficacy of the proposed approach in achieving precise lip sync, preserving speaking styles, and generating high-fidelity, realistic talking face videos.
One-sentence Summary
Addressing the inaccurate style aggregation of prior methods, this work proposes an audio-aware style reference scheme that integrates a Transformer-based lip motion predictor enhanced by cross-attention layers for style aggregation and a conditional latent diffusion renderer fused via modulated convolutions and spatial cross-attention, with extensive experiments validating its ability to achieve precise lip synchronization, preserve individual speaking styles, and generate high-fidelity talking face videos.
Key Contributions
- This work proposes an audio-aware style reference scheme that models the relationship between input audio and reference audio to preserve individual speaking styles. A Transformer-based architecture predicts target lip motions by aggregating personalized style cues through cross-attention layers.
- A conditional latent diffusion model renders the predicted lip motions into realistic talking face videos. This renderer integrates motion signals through modulated convolutional layers and fuses reference facial images via spatial cross-attention mechanisms.
- Extensive experiments validate that the proposed framework achieves precise lip synchronization, effectively preserves individual speaking styles, and generates high-fidelity talking face videos. The results confirm the effectiveness of the integrated style aggregation and rendering pipeline.
Introduction
No source text was provided for analysis. Please share the abstract or body snippet so I can draft a concise research background summary that outlines the technical context, prior limitations, and the authors’ main contribution in a clear, technical yet readable format.