Chen Tianshui ; Pu Tao ; Wu Hefeng ; Xie Yuan ; Lin Liang

Abstract

Multi-label image recognition is a fundamental yet practical task becausereal-world images inherently possess multiple semantic labels. However, it isdifficult to collect large-scale multi-label annotations due to the complexityof both the input images and output label spaces. To reduce the annotationcost, we propose a structured semantic transfer (SST) framework that enablestraining multi-label recognition models with partial labels, i.e., merely somelabels are known while other labels are missing (also called unknown labels)per image. The framework consists of two complementary transfer modules thatexplore within-image and cross-image semantic correlations to transferknowledge of known labels to generate pseudo labels for unknown labels.Specifically, an intra-image semantic transfer module learns image-specificlabel co-occurrence matrix and maps the known labels to complement unknownlabels based on this matrix. Meanwhile, a cross-image transfer module learnscategory-specific feature similarities and helps complement unknown labels withhigh similarities. Finally, both known and generated labels are used to trainthe multi-label recognition models. Extensive experiments on the MicrosoftCOCO, Visual Genome and Pascal VOC datasets show that the proposed SSTframework obtains superior performance over current state-of-the-artalgorithms. Codes are available at https://github.com/HCPLab-SYSU/HCP-MLR-PL.