Recognizing multiple labels of images is a practical and challenging task,and significant progress has been made by searching semantic-aware regions andmodeling label dependency. However, current methods cannot locate the semanticregions accurately due to the lack of part-level supervision or semanticguidance. Moreover, they cannot fully explore the mutual interactions among thesemantic regions and do not explicitly model the label co-occurrence. Toaddress these issues, we propose a Semantic-Specific Graph RepresentationLearning (SSGRL) framework that consists of two crucial modules: 1) a semanticdecoupling module that incorporates category semantics to guide learningsemantic-specific representations and 2) a semantic interaction module thatcorrelates these representations with a graph built on the statistical labelco-occurrence and explores their interactions via a graph propagationmechanism. Extensive experiments on public benchmarks show that our SSGRLframework outperforms current state-of-the-art methods by a sizable margin,e.g. with an mAP improvement of 2.5%, 2.6%, 6.7%, and 3.1% on the PASCAL VOC2007 & 2012, Microsoft-COCO and Visual Genome benchmarks, respectively. Ourcodes and models are available at https://github.com/HCPLab-SYSU/SSGRL.