Javed Muhammad Gohar ; Guo Chuan ; Cheng Li ; Li Xingyu

Abstract

Generating realistic 3D human-human interactions from textual descriptionsremains a challenging task. Existing approaches, typically based on diffusionmodels, often produce results lacking realism and fidelity. In this work, weintroduce InterMask, a novel framework for generating human interactions usingcollaborative masked modeling in discrete space. InterMask first employs aVQ-VAE to transform each motion sequence into a 2D discrete motion token map.Unlike traditional 1D VQ token maps, it better preserves fine-grainedspatio-temporal details and promotes spatial awareness within each token.Building on this representation, InterMask utilizes a generative maskedmodeling framework to collaboratively model the tokens of two interactingindividuals. This is achieved by employing a transformer architecturespecifically designed to capture complex spatio-temporal inter-dependencies.During training, it randomly masks the motion tokens of both individuals andlearns to predict them. For inference, starting from fully masked sequences, itprogressively fills in the tokens for both individuals. With its enhancedmotion representation, dedicated architecture, and effective learning strategy,InterMask achieves state-of-the-art results, producing high-fidelity anddiverse human interactions. It outperforms previous methods, achieving an FIDof $5.154$ (vs $5.535$ of in2IN) on the InterHuman dataset and $0.399$ (vs$5.207$ of InterGen) on the InterX dataset. Additionally, InterMask seamlesslysupports reaction generation without the need for model redesign orfine-tuning.