Lin Ji ; Gan Chuang ; Han Song

Abstract

The explosive growth in video streaming gives rise to challenges onperforming video understanding at high accuracy and low computation cost.Conventional 2D CNNs are computationally cheap but cannot capture temporalrelationships; 3D CNN based methods can achieve good performance but arecomputationally intensive, making it expensive to deploy. In this paper, wepropose a generic and effective Temporal Shift Module (TSM) that enjoys bothhigh efficiency and high performance. Specifically, it can achieve theperformance of 3D CNN but maintain 2D CNN's complexity. TSM shifts part of thechannels along the temporal dimension; thus facilitate information exchangedamong neighboring frames. It can be inserted into 2D CNNs to achieve temporalmodeling at zero computation and zero parameters. We also extended TSM toonline setting, which enables real-time low-latency online video recognitionand video object detection. TSM is accurate and efficient: it ranks the firstplace on the Something-Something leaderboard upon publication; on Jetson Nanoand Galaxy Note8, it achieves a low latency of 13ms and 35ms for online videorecognition. The code is available at:https://github.com/mit-han-lab/temporal-shift-module.