Patra Suvajit ; Maitra Arkadip ; Tiwari Megha ; Kumaran K. ; Prabhu Swathy ; Punyeshwarananda Swami ; Samanta Soumitra

Abstract

Automatic Sign Language (SL) recognition is an important task in the computervision community. To build a robust SL recognition system, we need aconsiderable amount of data which is lacking particularly in Indian signlanguage (ISL). In this paper, we introduce a large-scale isolated ISL datasetand a novel SL recognition model based on skeleton graph structure. The datasetcovers 2002 daily used common words in the deaf community recorded by 20 (10male and 10 female) deaf adult signers (contains 40033 videos). We propose a SLrecognition model namely Hierarchical Windowed Graph Attention Network (HWGAT)by utilizing the human upper body skeleton graph. The HWGAT tries to capturedistinctive motions by giving attention to different body parts induced by thehuman skeleton graph. The utility of the proposed dataset and the usefulness ofour model are evaluated through extensive experiments. We pre-trained theproposed model on the presented dataset and fine-tuned it across different signlanguage datasets further boosting the performance of 1.10, 0.46, 0.78, and6.84 percentage points on INCLUDE, LSA64, AUTSL and WLASL respectively comparedto the existing state-of-the-art keypoints-based models.