Deep motion and appearance cues for visual tracking

Martin Danelljan, Goutam Bhat, Susanna Gladh, Fahad Shahbaz Khan, Michael Felsberg

Abstract: Highlights • We investigate the impact of deep motion features for visual tracking. • We fuse hand-crafted and deep appearance features with deep motion features in a state-of-the-art tracking framework. • Extensive experiments are performed on the OTB-2015, Temple-Color, and VOT-2015 datasets. • We show that deep motion features lead to significant improvement in tracking performance, leading to state-of-the-artresults. Abstract Generic visual tracking is a challenging computer vision problem, with numerous applications. Most existing approaches rely on appearance information by employing either hand-crafted features or deep RGB features extracted from convolutional neural networks. Despite their success, these approaches struggle in case of ambiguous appearance information, leading to tracking failure. In such cases, we argue that motion cue provides discriminative and complementary information that can improve tracking performance. Contrary to visual tracking, deep motion features have been successfully applied for action recognition and video classification tasks. Typically, the motion features are learned by training a CNN on optical flow images extracted from large amounts of labeled videos. In this paper, we investigate the impact of deep motion features in a tracking-by-detection framework. We also evaluate the fusion of hand-crafted, deep RGB, and deep motion features and show that they contain complementary information. To the best of our knowledge, we are the first to propose fusing appearance information with deep motion features for visual tracking. Comprehensive experiments clearly demonstrate that our fusion approach with deep motion features outperforms standard methods relying on appearance information alone.