Schwarz Katja ; Liao Yiyi ; Niemeyer Michael ; Geiger Andreas

Abstract

While 2D generative adversarial networks have enabled high-resolution imagesynthesis, they largely lack an understanding of the 3D world and the imageformation process. Thus, they do not provide precise control over cameraviewpoint or object pose. To address this problem, several recent approachesleverage intermediate voxel-based representations in combination withdifferentiable rendering. However, existing methods either produce low imageresolution or fall short in disentangling camera and scene properties, e.g.,the object identity may vary with the viewpoint. In this paper, we propose agenerative model for radiance fields which have recently proven successful fornovel view synthesis of a single scene. In contrast to voxel-basedrepresentations, radiance fields are not confined to a coarse discretization ofthe 3D space, yet allow for disentangling camera and scene properties whiledegrading gracefully in the presence of reconstruction ambiguity. Byintroducing a multi-scale patch-based discriminator, we demonstrate synthesisof high-resolution images while training our model from unposed 2D imagesalone. We systematically analyze our approach on several challenging syntheticand real-world datasets. Our experiments reveal that radiance fields are apowerful representation for generative image synthesis, leading to 3Dconsistent models that render with high fidelity.