Tamura Masato ; Ohashi Hiroki ; Yoshinaga Tomoaki

Abstract

We propose a simple, intuitive yet powerful method for human-objectinteraction (HOI) detection. HOIs are so diverse in spatial distribution in animage that existing CNN-based methods face the following three major drawbacks;they cannot leverage image-wide features due to CNN's locality, they rely on amanually defined location-of-interest for the feature aggregation, whichsometimes does not cover contextually important regions, and they cannot helpbut mix up the features for multiple HOI instances if they are located closely.To overcome these drawbacks, we propose a transformer-based feature extractor,in which an attention mechanism and query-based detection play key roles. Theattention mechanism is effective in aggregating contextually importantinformation image-wide, while the queries, which we design in such a way thateach query captures at most one human-object pair, can avoid mixing up thefeatures from multiple instances. This transformer-based feature extractorproduces so effective embeddings that the subsequent detection heads may befairly simple and intuitive. The extensive analysis reveals that the proposedmethod successfully extracts contextually important features, and thusoutperforms existing methods by large margins (5.37 mAP on HICO-DET, and 5.7mAP on V-COCO). The source codes are available at$\href{https://github.com/hitachi-rd-cv/qpic}{\text{this https URL}}$.