Sun Xingyuan ; Wu Jiajun ; Zhang Xiuming ; Zhang Zhoutong ; Zhang Chengkai ; Xue Tianfan ; Tenenbaum Joshua B. ; Freeman William T.

Abstract

We study 3D shape modeling from a single image and make contributions to itin three aspects. First, we present Pix3D, a large-scale benchmark of diverseimage-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applicationsin shape-related tasks including reconstruction, retrieval, viewpointestimation, etc. Building such a large-scale dataset, however, is highlychallenging; existing datasets either contain only synthetic data, or lackprecise alignment between 2D images and 3D shapes, or only have a small numberof images. Second, we calibrate the evaluation criteria for 3D shapereconstruction through behavioral studies, and use them to objectively andsystematically benchmark cutting-edge reconstruction algorithms on Pix3D.Third, we design a novel model that simultaneously performs 3D reconstructionand pose estimation; our multi-task learning approach achieves state-of-the-artperformance on both tasks.