One-sentence Summary

Researchers from Peking University, LIGHTSPEED, The Chinese University of Hong Kong (Shenzhen), and an independent researcher present Motion-Omni, an end-to-end framework where a spoken dialogue model natively generates facial expressions and full-body motion from speech hidden states, avoiding the two-pass cascade and enabling joint optimization; trained on $422856$422856 pseudo-labeled pairs with a frozen speech pathway, Motion-Omni-Q7 (Qwen2.5-7B-Instruct) matches teacher cascades within $2\%$2% on motion metrics while running $5.4\times$5.4× faster ($\text{RTF} = 0.78$RTF=0.78) and achieving a $2.62\%$2.62% word error rate, plus releasing SwDA-500 and the first public evaluation protocol for stochastic open-ended full-body spoken dialogue.

Key Contributions

• Introduces Motion-Omni, an end-to-end framework where a spoken dialogue model natively outputs explicit facial expressions alongside hand, upper-body, and lower-body motion, generated directly from the hidden states that produce speech rather than from a decoded waveform.
• Demonstrates that joint training is essential: freezing the speech pathway leaves motion misaligned with audio, while co-adapting the LLM, Speech Generator, and Motion Generator under both objectives restores alignment and preserves spoken-dialogue ability, enabled by a model-agnostic pseudo-labeling pipeline yielding 422,856 quality-ranked pairs (1,402 hours).
• Results show Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4× faster (RTF = 0.78), surpasses all non-teacher cascades on beat correlation and diversity, and achieves a 2.62% word error rate, the lowest among compared omni-modal systems; also releases SwDA-500 and the first public evaluation protocol for stochastic open-ended full-body spoken dialogue.

Introduction

Speech and full-body co-speech motion are tightly coupled in human communication, yet most spoken dialogue models (SDMs) generate only audio, while co-speech motion models require pre-supplied speech as input. A naive cascade, where an SDM produces speech and a separate motion model animates it, works but incurs two structural costs: a second full inference pass after audio is complete, and no motion objective can ever update the speech or dialogue parameters. Prior integrated systems either restrict motion to facial animation, freeze the speech pathway to prevent motion gradients from degrading it, or fail to natively combine explicit face, hand, upper-body, and lower-body output with co-adaptation of the response-generation pathway. The authors address these gaps with Motion-Omni, an end-to-end framework that generates explicit facial and full-body motion as a native output of an SDM, conditioned on the same dialogue context that produces the spoken response. To overcome the rate mismatch between 12.5 Hz speech units and 30 Hz motion, they introduce a dual-input conditioning interface where the Motion Generator attends to the Speech Generator’s hidden states and consumes emitted speech token embeddings, avoiding the rendered waveform as an intermediate input. A four-stage training progression co-adapts all modules end-to-end, and ablations confirm this co-adaptation is necessary for speech-motion alignment. For data, they build a scalable, model-agnostic pseudo-labeling pipeline that yields 422,856 teacher-generated speech-motion samples (1,402 hours) in a consistent target voice, using a dual-metric quality score to rank supervision. They also release SwDA-500 and a reproducible evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems to isolate motion quality. The resulting instance, Motion-Omni-Q7, matches the same-audio teacher cascade within 2% on reference-free motion metrics while responding 5.4 times faster than real time, and achieves the lowest word error rate among omni-modal systems compared.

Dataset

The authors build a large-scale supervision dataset for training a spoken motion model by pairing dialogue responses with accompanying full-body motion. Because captured audiovisual corpora, such as the dyadic Seamless Interaction corpus, contain many voices that do not match the single target voice of the Speech Generator, they construct supervision through pseudo-labeling. A generic pipeline takes a speech-instruction corpus already paired with response audio and uses a co-speech motion generation teacher to produce paired (text, speech, full-body motion) supervision in two steps.

• Motion supervision via a teacher: A pre-trained co-speech motion generation teacher (LOM in this run) processes every response waveform, and its four per-part VQ code streams serve as the motion target. The teacher is replaceable, so future implementations can regenerate supervision with a stronger model.
• Dual-metric quality scoring and curriculum: Each sample is scored on two metrics: (i) a weighted VQ-VAE reconstruction error that flags motion outside the codebook's expressive range, and (ii) a beat correlation score that flags teacher motion weakly coupled to the speech. After robust 5th to 95th percentile normalization, each sample receives a combined score with alpha set to 0.5, which drives the four-substage Stage 3 curriculum and selects the Stage 4 S2SM pool.

The dataset is organized into four stages with distinct sources and filtering rules:

• Stages 1 and 2: Draw ASR and TTS pairs from InstructS2S-200K and the English subset of Ex-Instruct.
• Stage 3: Applies the pipeline to InstructS2S-200K with the LOM teacher. Training directly on the complete unranked corpus diverged, so the network is exposed to progressively larger quantiles of the dual-metric score (12.5%, 25%, 50%, 100%), with each substage warm-started from the previous checkpoint. The stage-wise teacher-reference Fréchet Gesture Distance decreases from 0.3974 at Stage 3a to 0.3040.
• Stage 4: Assembles a four-task mixture of ASR, TTS, S2SM, and T2T. The T2T component, roughly 213K samples, is drawn from six text-only datasets (SODA, WildChat, Tulu-3, WizardLM, OpenThoughts, NuminaMath-CoT) to preserve general text-response planning and multi-turn dialogue behavior during joint fine-tuning. Without this component, responses more often repeated the user's input.

All sources are English-filtered using langdetect and capped at 10,000 characters per dialogue. For evaluation, the authors use SwDA-500, an external 500-prompt dialogue-text evaluation set derived from the Switchboard Dialog Act Corpus. This set covers all 66 SwDA topic descriptions with 7 or 8 semantically complete speaker turns per topic, keeps prompts of moderate length, and removes transcription artifacts not intended to be spoken. SwDA-500 provides real conversational wording but is not a paired real-motion benchmark; all motion references are teacher-generated or baseline-generated under matched prompts.

Method

The authors design the Motion-Omni framework to autoregressively generate spoken responses accompanied by synchronized full-body co-speech motion from user speech or text inputs. The architecture comprises four primary components: a Speech Encoder, a Large Language Model (LLM) Backbone, a Speech Generator, and a Motion Generator.

As shown in the framework diagram, the Speech Encoder utilizes a frozen Whisper-large-v3 model to map 16 kHz waveforms into continuous representations. A speech projector then concatenates every five consecutive frames and passes them through a two-layer MLP into the LLM embedding space, effectively downsampling the sequence fivefold. The LLM Backbone, instantiated as a Qwen2.5-7B-Instruct model, processes these projected continuous speech segments alongside surrounding text tokens. For speech inputs, the projected features replace a designated placeholder in the token sequence.

The Speech Generator is a Qwen2-style transformer initialized from Qwen2.5-0.5B-Instruct. It autoregressively emits discrete speech units at 12.5 Hz over a vocabulary of 16,384 units. To integrate contextual information, the authors introduce a Token-as-Query Gated Fusion (TQGF) block. This mechanism allows token embeddings to query the contextualized hidden states of the LLM through learned head-wise sigmoid gates.

The Motion Generator consists of four parallel, independent decoders dedicated to specific body parts: face, hands, upper body, and lower body. Each decoder emits LOM VQ codes at 30 Hz and is conditioned on the last-layer hidden states of the Speech Generator. The conditioning mechanism relies on two input streams. The key and value stream projects the Speech Generator hidden states $\mathbf{H}_s$Hs​ into the motion working dimension:

The query stream interpolates a learned embedding of the discrete speech units $\mathbf{u}$u from the 12.5 Hz speech rate to the 30 Hz motion rate:

where $\mathbf{E}$E is a look-up table initialized from a pre-trained flow embedding. Each decoder stacks TQGF layers followed by a self-attention Transformer with periodic rotary positional encoding. A part-specific MLP head then produces per-frame logits over the VQ codebook entries. The motion generation is optimized using a weighted cross-entropy loss:

where the weights $w_b$wb​ are proportional to the underlying SMPL-X and FLAME feature dimensions to ensure equal per-feature-dimension importance.

During inference, the emitted speech units are converted to mel spectrograms via a flow-matching decoder and subsequently to 22.05 kHz waveforms using a HiFi-GAN vocoder. Concurrently, the motion codes are decoded by a frozen LOM VQ-VAE into SMPL-X body and hand parameters along with FLAME facial-expression coefficients.

To effectively train this complex system, the authors implement a four-stage progressive training curriculum. In Stage 1, only the speech projector is trained with automatic speech recognition supervision while the LLM remains frozen. Stage 2 trains the Speech Generator on text-to-speech pairs. Stage 3 attaches the Motion Generator and jointly trains it with the Speech Generator on a text-to-speech-and-motion corpus. This stage employs a curriculum that progressively exposes the network to larger quality quantiles of the training data. Finally, Stage 4 unfreezes the LLM backbone, the speech projector, and both generators simultaneously, optimizing a mixture of automatic speech recognition, text-to-speech, speech-to-speech-with-motion, and text-to-text tasks.

The training data for the motion generation stages is constructed using a pseudo-labeling pipeline. A pre-trained co-speech motion generation teacher processes response waveforms to produce target VQ code streams. To ensure high-quality supervision, each sample is evaluated using a dual-metric quality score that combines a VQ-VAE reconstruction error and a beat correlation score. This combined score drives the curriculum in Stage 3 and selects the data pool for Stage 4, ensuring the model learns from well-aligned and expressive motion sequences.

Experiment

The experiments evaluate Motion-Omni-Q7, a model that jointly generates speech and co-speech motion, against cascaded baselines on dialogue, speech quality, motion generation, and latency. On VoiceBench, it outperforms other omni-modal LLMs in spoken dialogue quality, and on Seed-TTS-Eval it achieves the lowest WER among omni-modal LLMs while maintaining competitive speech naturalness. In automatic and human evaluations of speech-to-motion on SwDA-500, it ranks first or second on most metrics, matching or exceeding cascades that use a separate audio-to-motion stage, with human raters preferring its motion over EMAGE and finding it comparable to the LOM teacher. Finally, it completes full responses in 4.32 seconds (RTF 0.78), over five times faster than the matched LOM cascade, showing that joint generation avoids the latency penalty of separate motion inference.

The training data pipeline uses pseudo-labeling with a co-speech motion teacher to create paired supervision for spoken motion generation, and a dual-metric scoring system drives a curriculum that progressively exposes the model to larger quantiles of scored samples. The final mixture combines ASR, TTS, spoken motion, and text-to-text tasks, with the largest stage being TTS and the smallest being the spoken motion stage. The TTS stage has the most samples and hours, while the spoken motion stage is the smallest, with about a quarter of the samples of the TTS stage. Training directly on the complete unranked spoken motion corpus caused divergence, so a four-substage curriculum with increasing quantiles of the dual-metric score was used. The stage-wise teacher-reference Fréchet Gesture Distance improved from 0.3974 at the first substage to 0.3040 at the final substage of the spoken motion stage. The final mixture includes a text-to-text component drawn from six datasets to preserve general text-response planning and multi-turn dialogue behavior.

On Seed-TTS-Eval, Motion-Omni-Q7 achieves the lowest word error rate among omni-modal LLMs at 2.62%, though several dedicated TTS systems report lower WER. Its speech naturalness proxy score is 3.77, which is higher than some dedicated TTS systems but lower than others. Motion-Omni-Q7 achieves the lowest WER among omni-modal LLMs, while also generating co-speech motion. Several dedicated TTS systems report lower WER than Motion-Omni-Q7 on the same speech-only metric. Motion-Omni-Q7's naturalness proxy score exceeds that of several dedicated TTS systems but trails the top performers.

Motion-Omni-Q7 ranks first or second on seven of eight metrics, leading all systems that do not invoke the LOM teacher at motion inference on beat correlation, diversity, facial geometry, and lip-sync scores. The only systems that outperform it are LOM teacher-reference cascades, which use the same model that supplied its training targets, and the integrated model avoids a separate audio-to-motion stage while matching strong cascades in motion quality. Among non-LOM-inference systems, Motion-Omni-Q7 achieves the highest beat correlation, diversity, and best facial and lip-sync metrics. Teacher-referenced cascades that run LOM at inference time surpass the integrated model on some metrics, but they require an extra audio-to-motion stage. Motion-Omni-Q7 obtains the lowest teacher-referenced FGD, indicating high fidelity to the LOM-generated motion distribution.

The proposed model achieves faster-than-real-time response on the SwDA-500 evaluation set, completing speech and motion generation in about 4.3 seconds. Compared to cascade baselines that separate speech and motion stages, the integrated approach reduces latency substantially, with the closest cascade being over five times slower. While some cascades approach the model on either speed or quality, none match it on both dimensions. The proposed model responds faster than real time with a real-time factor below 1, while all cascades are slower than real time. The cascade using the same speech model but a separate motion stage is over five times slower than the proposed model. A cascade with a different motion model matches the proposed model's response time but sacrifices quality, while another cascade improves quality but is over five times slower.

The evaluation setup combines a pseudo-labeling pipeline with a dual-metric curriculum for spoken motion training, a mixed-task final stage, and comparisons against dedicated TTS systems and cascaded motion baselines. Motion-Omni-Q7 achieves the lowest word error rate among omni-modal LLMs, competitive speech naturalness, and top-tier co-speech motion quality, matching teacher-referenced cascades on motion fidelity while avoiding a separate audio-to-motion stage. It also delivers faster-than-real-time response, substantially reducing latency compared to cascades that separate speech and motion generation, with no cascade matching it on both speed and quality simultaneously.