Chen Zerui ; Chen Shizhe ; Schmid Cordelia ; Laptev Ivan

Abstract

Signed distance functions (SDFs) is an attractive framework that has recentlyshown promising results for 3D shape reconstruction from images. SDFsseamlessly generalize to different shape resolutions and topologies but lackexplicit modelling of the underlying 3D geometry. In this work, we exploit thehand structure and use it as guidance for SDF-based shape reconstruction. Inparticular, we address reconstruction of hands and manipulated objects frommonocular RGB images. To this end, we estimate poses of hands and objects anduse them to guide 3D reconstruction. More specifically, we predict kinematicchains of pose transformations and align SDFs with highly-articulated handposes. We improve the visual features of 3D points with geometry alignment andfurther leverage temporal information to enhance the robustness to occlusionand motion blurs. We conduct extensive experiments on the challenging ObMan andDexYCB benchmarks and demonstrate significant improvements of the proposedmethod over the state of the art.