DuTriNet: Dual-Stream Triplet Siamese Network for Self-Supervised Action Recognition by Modeling Temporal Correlations

Himanshu Buckchash, Balasubramanian Raman

Abstract: Self Supervised Learning (SSL) is the task of training a model independent of human annotations. A recent path-breaking SSL work is OpenAI's GPT-3. Very limited work has happened on SSL for Action Recognition (AR). Present SSL models either leverage only spatial data or use suboptimal frame sampling algorithms. To this end, we present a comprehensive study and propose DuTriNet for SSL in AR. We introduce the idea of temporal protraction by fusing optical-flow information with weber-maps inside parallel data-streams which share weights under a triplet Siamese architecture. We also propose flow-intensity based non-parametric frame sampling algorithm. Extensive experiments and ablations have been performed on two publicly available benchmark datasets for AR. Our findings suggest the suitability of DuTriNet for SSL.