Pratt Sarah ; Yatskar Mark ; Weihs Luca ; Farhadi Ali ; Kembhavi Aniruddha

Abstract

We introduce Grounded Situation Recognition (GSR), a task that requiresproducing structured semantic summaries of images describing: the primaryactivity, entities engaged in the activity with their roles (e.g. agent, tool),and bounding-box groundings of entities. GSR presents important technicalchallenges: identifying semantic saliency, categorizing and localizing a largeand diverse set of entities, overcoming semantic sparsity, and disambiguatingroles. Moreover, unlike in captioning, GSR is straightforward to evaluate. Tostudy this new task we create the Situations With Groundings (SWiG) datasetwhich adds 278,336 bounding-box groundings to the 11,538 entity classes in theimsitu dataset. We propose a Joint Situation Localizer and find that jointlypredicting situations and groundings with end-to-end training handilyoutperforms independent training on the entire grounding metric suite withrelative gains between 8% and 32%. Finally, we show initial findings on threeexciting future directions enabled by our models: conditional querying, visualchaining, and grounded semantic aware image retrieval. Code and data availableat https://prior.allenai.org/projects/gsr.