Abstract

Expressive human pose and shape estimation (a.k.a. 3D whole-body meshrecovery) involves the human body, hand, and expression estimation. Mostexisting methods have tackled this task in a two-stage manner, first detectingthe human body part with an off-the-shelf detection model and inferring thedifferent human body parts individually. Despite the impressive resultsachieved, these methods suffer from 1) loss of valuable contextual informationvia cropping, 2) introducing distractions, and 3) lacking inter-associationamong different persons and body parts, inevitably causing performancedegradation, especially for crowded scenes. To address these issues, weintroduce a novel all-in-one-stage framework, AiOS, for multiple expressivehuman pose and shape recovery without an additional human detection step.Specifically, our method is built upon DETR, which treats multi-personwhole-body mesh recovery task as a progressive set prediction problem withvarious sequential detection. We devise the decoder tokens and extend them toour task. Specifically, we first employ a human token to probe a human locationin the image and encode global features for each instance, which provides acoarse location for the later transformer block. Then, we introduce ajoint-related token to probe the human joint in the image and encoder afine-grained local feature, which collaborates with the global feature toregress the whole-body mesh. This straightforward but effective modeloutperforms previous state-of-the-art methods by a 9% reduction in NMVE onAGORA, a 30% reduction in PVE on EHF, a 10% reduction in PVE on ARCTIC, and a3% reduction in PVE on EgoBody.