Xiu Yuliang ; Yang Jinlong ; Cao Xu ; Tzionas Dimitrios ; Black Michael J.

Abstract

The combination of deep learning, artist-curated scans, and ImplicitFunctions (IF), is enabling the creation of detailed, clothed, 3D humans fromimages. However, existing methods are far from perfect. IF-based methodsrecover free-form geometry, but produce disembodied limbs or degenerate shapesfor novel poses or clothes. To increase robustness for these cases, existingwork uses an explicit parametric body model to constrain surfacereconstruction, but this limits the recovery of free-form surfaces such asloose clothing that deviates from the body. What we want is a method thatcombines the best properties of implicit representation and explicit bodyregularization. To this end, we make two key observations: (1) current networksare better at inferring detailed 2D maps than full-3D surfaces, and (2) aparametric model can be seen as a "canvas" for stitching together detailedsurface patches. Based on these, our method, ECON, has three main steps: (1) Itinfers detailed 2D normal maps for the front and back side of a clothed person.(2) From these, it recovers 2.5D front and back surfaces, called d-BiNI, thatare equally detailed, yet incomplete, and registers these w.r.t. each otherwith the help of a SMPL-X body mesh recovered from the image. (3) It "inpaints"the missing geometry between d-BiNI surfaces. If the face and hands are noisy,they can optionally be replaced with the ones of SMPL-X. As a result, ECONinfers high-fidelity 3D humans even in loose clothes and challenging poses.This goes beyond previous methods, according to the quantitative evaluation onthe CAPE and Renderpeople datasets. Perceptual studies also show that ECON'sperceived realism is better by a large margin. Code and models are availablefor research purposes at econ.is.tue.mpg.de