Cao Anh-Quan ; de Charette Raoul

Abstract

MonoScene proposes a 3D Semantic Scene Completion (SSC) framework, where thedense geometry and semantics of a scene are inferred from a single monocularRGB image. Different from the SSC literature, relying on 2.5 or 3D input, wesolve the complex problem of 2D to 3D scene reconstruction while jointlyinferring its semantics. Our framework relies on successive 2D and 3D UNetsbridged by a novel 2D-3D features projection inspiring from optics andintroduces a 3D context relation prior to enforce spatio-semantic consistency.Along with architectural contributions, we introduce novel global scene andlocal frustums losses. Experiments show we outperform the literature on allmetrics and datasets while hallucinating plausible scenery even beyond thecamera field of view. Our code and trained models are available athttps://github.com/cv-rits/MonoScene.