Lin Kevin ; Wang Lijuan ; Liu Zicheng

Abstract

We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct 3Dhuman pose and mesh vertices from a single image. Our method uses a transformerencoder to jointly model vertex-vertex and vertex-joint interactions, andoutputs 3D joint coordinates and mesh vertices simultaneously. Compared toexisting techniques that regress pose and shape parameters, METRO does not relyon any parametric mesh models like SMPL, thus it can be easily extended toother objects such as hands. We further relax the mesh topology and allow thetransformer self-attention mechanism to freely attend between any two vertices,making it possible to learn non-local relationships among mesh vertices andjoints. With the proposed masked vertex modeling, our method is more robust andeffective in handling challenging situations like partial occlusions. METROgenerates new state-of-the-art results for human mesh reconstruction on thepublic Human3.6M and 3DPW datasets. Moreover, we demonstrate thegeneralizability of METRO to 3D hand reconstruction in the wild, outperformingexisting state-of-the-art methods on FreiHAND dataset. Code and pre-trainedmodels are available at https://github.com/microsoft/MeshTransformer.