Li Jiefeng ; Xu Chao ; Chen Zhicun ; Bian Siyuan ; Yang Lixin ; Lu Cewu

Abstract

Model-based 3D pose and shape estimation methods reconstruct a full 3D meshfor the human body by estimating several parameters. However, learning theabstract parameters is a highly non-linear process and suffers from image-modelmisalignment, leading to mediocre model performance. In contrast, 3D keypointestimation methods combine deep CNN network with the volumetric representationto achieve pixel-level localization accuracy but may predict unrealistic bodystructure. In this paper, we address the above issues by bridging the gapbetween body mesh estimation and 3D keypoint estimation. We propose a novelhybrid inverse kinematics solution (HybrIK). HybrIK directly transformsaccurate 3D joints to relative body-part rotations for 3D body meshreconstruction, via the twist-and-swing decomposition. The swing rotation isanalytically solved with 3D joints, and the twist rotation is derived from thevisual cues through the neural network. We show that HybrIK preserves both theaccuracy of 3D pose and the realistic body structure of the parametric humanmodel, leading to a pixel-aligned 3D body mesh and a more accurate 3D pose thanthe pure 3D keypoint estimation methods. Without bells and whistles, theproposed method surpasses the state-of-the-art methods by a large margin onvarious 3D human pose and shape benchmarks. As an illustrative example, HybrIKoutperforms all the previous methods by 13.2 mm MPJPE and 21.9 mm PVE on 3DPWdataset. Our code is available at https://github.com/Jeff-sjtu/HybrIK.