Zhang Hongwen ; Tian Yating ; Zhang Yuxiang ; Li Mengcheng ; An Liang ; Sun Zhenan ; Liu Yebin

Abstract

We present PyMAF-X, a regression-based approach to recovering parametricfull-body models from monocular images. This task is very challenging sinceminor parametric deviation may lead to noticeable misalignment between theestimated mesh and the input image. Moreover, when integrating part-specificestimations into the full-body model, existing solutions tend to either degradethe alignment or produce unnatural wrist poses. To address these issues, wepropose a Pyramidal Mesh Alignment Feedback (PyMAF) loop in our regressionnetwork for well-aligned human mesh recovery and extend it as PyMAF-X for therecovery of expressive full-body models. The core idea of PyMAF is to leveragea feature pyramid and rectify the predicted parameters explicitly based on themesh-image alignment status. Specifically, given the currently predictedparameters, mesh-aligned evidence will be extracted from finer-resolutionfeatures accordingly and fed back for parameter rectification. To enhance thealignment perception, an auxiliary dense supervision is employed to providemesh-image correspondence guidance while spatial alignment attention isintroduced to enable the awareness of the global contexts for our network. Whenextending PyMAF for full-body mesh recovery, an adaptive integration strategyis proposed in PyMAF-X to produce natural wrist poses while maintaining thewell-aligned performance of the part-specific estimations. The efficacy of ourapproach is validated on several benchmark datasets for body, hand, face, andfull-body mesh recovery, where PyMAF and PyMAF-X effectively improve themesh-image alignment and achieve new state-of-the-art results. The project pagewith code and video results can be found at https://www.liuyebin.com/pymaf-x.