Jianwen Jiang Weihong Zeng Zerong Zheng Jiaqi Yang Chao Liang Wang Liao Han Liang Yuan Zhang Mingyuan Gao

Abstract

Existing video avatar models can produce fluid human animations, yet theystruggle to move beyond mere physical likeness to capture a character'sauthentic essence. Their motions typically synchronize with low-level cues likeaudio rhythm, lacking a deeper semantic understanding of emotion, intent, orcontext. To bridge this gap, we propose a framework designed togenerate character animations that are not only physically plausible but alsosemantically coherent and expressive. Our model, OmniHuman-1.5, isbuilt upon two key technical contributions. First, we leverage Multimodal LargeLanguage Models to synthesize a structured textual representation of conditionsthat provides high-level semantic guidance. This guidance steers our motiongenerator beyond simplistic rhythmic synchronization, enabling the productionof actions that are contextually and emotionally resonant. Second, to ensurethe effective fusion of these multimodal inputs and mitigate inter-modalityconflicts, we introduce a specialized Multimodal DiT architecture with a novelPseudo Last Frame design. The synergy of these components allows our model toaccurately interpret the joint semantics of audio, images, and text, therebygenerating motions that are deeply coherent with the character, scene, andlinguistic content. Extensive experiments demonstrate that our model achievesleading performance across a comprehensive set of metrics, including lip-syncaccuracy, video quality, motion naturalness and semantic consistency withtextual prompts. Furthermore, our approach shows remarkable extensibility tocomplex scenarios, such as those involving multi-person and non-human subjects.Homepage: https://omnihuman-lab.github.io/v1_5/