Li Jiefeng ; Bian Siyuan ; Xu Chao ; Chen Zhicun ; Yang Lixin ; Lu Cewu

Abstract

Recovering whole-body mesh by inferring the abstract pose and shapeparameters from visual content can obtain 3D bodies with realistic structures.However, the inferring process is highly non-linear and suffers from image-meshmisalignment, resulting in inaccurate reconstruction. In contrast, 3D keypointestimation methods utilize the volumetric representation to achieve pixel-levelaccuracy but may predict unrealistic body structures. To address these issues,this paper presents a novel hybrid inverse kinematics solution, HybrIK, thatintegrates the merits of 3D keypoint estimation and body mesh recovery in aunified framework. HybrIK directly transforms accurate 3D joints to body-partrotations via twist-and-swing decomposition. The swing rotations areanalytically solved with 3D joints, while the twist rotations are derived fromvisual cues through neural networks. To capture comprehensive whole-bodydetails, we further develop a holistic framework, HybrIK-X, which enhancesHybrIK with articulated hands and an expressive face. HybrIK-X is fast andaccurate by solving the whole-body pose with a one-stage model. Experimentsdemonstrate that HybrIK and HybrIK-X preserve both the accuracy of 3D jointsand the realistic structure of the parametric human model, leading topixel-aligned whole-body mesh recovery. The proposed method significantlysurpasses the state-of-the-art methods on various benchmarks for body-only,hand-only, and whole-body scenarios. Code and results can be found athttps://jeffli.site/HybrIK-X/