Li Ming-Zhe ; Jia Zhen ; Zhang Zhang ; Ma Zhanyu ; Wang Liang

Abstract

Generalized zero-shot skeleton-based action recognition (GZSSAR) is a newchallenging problem in computer vision community, which requires models torecognize actions without any training samples. Previous studies only utilizethe action labels of verb phrases as the semantic prototypes for learning themapping from skeleton-based actions to a shared semantic space. However, thelimited semantic information of action labels restricts the generalizationability of skeleton features for recognizing unseen actions. In order to solvethis dilemma, we propose a multi-semantic fusion (MSF) model for improving theperformance of GZSSAR, where two kinds of class-level textual descriptions(i.e., action descriptions and motion descriptions), are collected as auxiliarysemantic information to enhance the learning efficacy of generalizable skeletonfeatures. Specially, a pre-trained language encoder takes the actiondescriptions, motion descriptions and original class labels as inputs to obtainrich semantic features for each action class, while a skeleton encoder isimplemented to extract skeleton features. Then, a variational autoencoder (VAE)based generative module is performed to learn a cross-modal alignment betweenskeleton and semantic features. Finally, a classification module is built torecognize the action categories of input samples, where a seen-unseenclassification gate is adopted to predict whether the sample comes from seenaction classes or not in GZSSAR. The superior performance in comparisons withprevious models validates the effectiveness of the proposed MSF model onGZSSAR.