Hou Zhi ; Yu Baosheng ; Qiao Yu ; Peng Xiaojiang ; Tao Dacheng

Abstract

Reasoning the human-object interactions (HOI) is essential for deeper sceneunderstanding, while object affordances (or functionalities) are of greatimportance for human to discover unseen HOIs with novel objects. Inspired bythis, we introduce an affordance transfer learning approach to jointly detectHOIs with novel objects and recognize affordances. Specifically, HOIrepresentations can be decoupled into a combination of affordance and objectrepresentations, making it possible to compose novel interactions by combiningaffordance representations and novel object representations from additionalimages, i.e. transferring the affordance to novel objects. With the proposedaffordance transfer learning, the model is also capable of inferring theaffordances of novel objects from known affordance representations. Theproposed method can thus be used to 1) improve the performance of HOIdetection, especially for the HOIs with unseen objects; and 2) infer theaffordances of novel objects. Experimental results on two datasets, HICO-DETand HOI-COCO (from V-COCO), demonstrate significant improvements over recentstate-of-the-art methods for HOI detection and object affordance detection.Code is available at https://github.com/zhihou7/HOI-CL