Zeng Ailing ; Ju Xuan ; Yang Lei ; Gao Ruiyuan ; Zhu Xizhou ; Dai Bo ; Xu Qiang

Abstract

This paper proposes a simple baseline framework for video-based 2D/3D humanpose estimation that can achieve 10 times efficiency improvement over existingworks without any performance degradation, named DeciWatch. Unlike currentsolutions that estimate each frame in a video, DeciWatch introduces a simpleyet effective sample-denoise-recover framework that only watches sparselysampled frames, taking advantage of the continuity of human motions and thelightweight pose representation. Specifically, DeciWatch uniformly samples lessthan 10% video frames for detailed estimation, denoises the estimated 2D/3Dposes with an efficient Transformer architecture, and then accurately recoversthe rest of the frames using another Transformer-based network. Comprehensiveexperimental results on three video-based human pose estimation and body meshrecovery tasks with four datasets validate the efficiency and effectiveness ofDeciWatch. Code is available at https://github.com/cure-lab/DeciWatch.