Cho Junhyeong ; Yoon Youngseok ; Kwak Suha

Abstract

Grounded situation recognition is the task of predicting the main activity,entities playing certain roles within the activity, and bounding-box groundingsof the entities in the given image. To effectively deal with this challengingtask, we introduce a novel approach where the two processes for activityclassification and entity estimation are interactive and complementary. Toimplement this idea, we propose Collaborative Glance-Gaze TransFormer(CoFormer) that consists of two modules: Glance transformer for activityclassification and Gaze transformer for entity estimation. Glance transformerpredicts the main activity with the help of Gaze transformer that analyzesentities and their relations, while Gaze transformer estimates the groundedentities by focusing only on the entities relevant to the activity predicted byGlance transformer. Our CoFormer achieves the state of the art in allevaluation metrics on the SWiG dataset. Training code and model weights areavailable at https://github.com/jhcho99/CoFormer.