Park JoonKyu ; Oh Yeonguk ; Moon Gyeongsik ; Choi Hongsuk ; Lee Kyoung Mu

Abstract

Hands are often severely occluded by objects, which makes 3D hand meshestimation challenging. Previous works often have disregarded information atoccluded regions. However, we argue that occluded regions have strongcorrelations with hands so that they can provide highly beneficial informationfor complete 3D hand mesh estimation. Thus, in this work, we propose a novel 3Dhand mesh estimation network HandOccNet, that can fully exploits theinformation at occluded regions as a secondary means to enhance image featuresand make it much richer. To this end, we design two successiveTransformer-based modules, called feature injecting transformer (FIT) and self-enhancing transformer (SET). FIT injects hand information into occluded regionby considering their correlation. SET refines the output of FIT by using aself-attention mechanism. By injecting the hand information to the occludedregion, our HandOccNet reaches the state-of-the-art performance on 3D hand meshbenchmarks that contain challenging hand-object occlusions. The codes areavailable in: https://github.com/namepllet/HandOccNet.