Zhang Jason Y. ; Pepose Sam ; Joo Hanbyul ; Ramanan Deva ; Malik Jitendra ; Kanazawa Angjoo

Abstract

We present a method that infers spatial arrangements and shapes of humans andobjects in a globally consistent 3D scene, all from a single image in-the-wildcaptured in an uncontrolled environment. Notably, our method runs on datasetswithout any scene- or object-level 3D supervision. Our key insight is thatconsidering humans and objects jointly gives rise to "3D common sense"constraints that can be used to resolve ambiguity. In particular, we introducea scale loss that learns the distribution of object size from data; anocclusion-aware silhouette re-projection loss to optimize object pose; and ahuman-object interaction loss to capture the spatial layout of objects withwhich humans interact. We empirically validate that our constraintsdramatically reduce the space of likely 3D spatial configurations. Wedemonstrate our approach on challenging, in-the-wild images of humansinteracting with large objects (such as bicycles, motorcycles, and surfboards)and handheld objects (such as laptops, tennis rackets, and skateboards). Wequantify the ability of our approach to recover human-object arrangements andoutline remaining challenges in this relatively domain. The project webpage canbe found at https://jasonyzhang.com/phosa.