Gkioxari Georgia ; Malik Jitendra ; Johnson Justin

Abstract

Rapid advances in 2D perception have led to systems that accurately detectobjects in real-world images. However, these systems make predictions in 2D,ignoring the 3D structure of the world. Concurrently, advances in 3D shapeprediction have mostly focused on synthetic benchmarks and isolated objects. Weunify advances in these two areas. We propose a system that detects objects inreal-world images and produces a triangle mesh giving the full 3D shape of eachdetected object. Our system, called Mesh R-CNN, augments Mask R-CNN with a meshprediction branch that outputs meshes with varying topological structure byfirst predicting coarse voxel representations which are converted to meshes andrefined with a graph convolution network operating over the mesh's vertices andedges. We validate our mesh prediction branch on ShapeNet, where we outperformprior work on single-image shape prediction. We then deploy our full Mesh R-CNNsystem on Pix3D, where we jointly detect objects and predict their 3D shapes.