Liu Xin ; Yuan Kaishen ; Niu Xuesong ; Shi Jingang ; Yu Zitong ; Yue Huanjing ; Yang Jingyu

Abstract

Facial Action Unit (AU) detection is a crucial task in affective computingand social robotics as it helps to identify emotions expressed through facialexpressions. Anatomically, there are innumerable correlations between AUs,which contain rich information and are vital for AU detection. Previous methodsused fixed AU correlations based on expert experience or statistical rules onspecific benchmarks, but it is challenging to comprehensively reflect complexcorrelations between AUs via hand-crafted settings. There are alternativemethods that employ a fully connected graph to learn these dependenciesexhaustively. However, these approaches can result in a computational explosionand high dependency with a large dataset. To address these challenges, thispaper proposes a novel self-adjusting AU-correlation learning (SACL) methodwith less computation for AU detection. This method adaptively learns andupdates AU correlation graphs by efficiently leveraging the characteristics ofdifferent levels of AU motion and emotion representation information extractedin different stages of the network. Moreover, this paper explores the role ofmulti-scale learning in correlation information extraction, and design a simpleyet effective multi-scale feature learning (MSFL) method to promote betterperformance in AU detection. By integrating AU correlation information withmulti-scale features, the proposed method obtains a more robust featurerepresentation for the final AU detection. Extensive experiments show that theproposed method outperforms the state-of-the-art methods on widely used AUdetection benchmark datasets, with only 28.7\% and 12.0\% of the parameters andFLOPs of the best method, respectively. The code for this method is availableat \url{https://github.com/linuxsino/Self-adjusting-AU}.