Mimic the raw domain: Accelerating action recognition in the compressed domain

Barak Battash, haim barad, Amit Bleiweiss, Hanlin Tang

Abstract:

Video understanding usually requires expensive computation that prohibits its deployment, yet videos contain significant spatiotemporal redundancy that can be exploited.
In particular, operating directly on the motion vectors and
residuals in the compressed video domain can significantly
accelerate the compute, by not using the raw videos which
demand colossal storage capacity. Existing methods approach this task as a multiple modalities problem. In this
paper we are approaching the task in a completely different
way; we are looking at the data from the compressed stream
as a one unit clip and propose that the residual frames can
replace the original RGB frames from the raw domain. Furthermore, we are using teacher-student method to aid the
network in the compressed domain to mimic the teacher network in the raw domain. We show experiments on three
leading datasets (HMDB51, UCF1, and Kinetics) that approach state-of-the-art accuracy on raw video data by using
compressed data. Our model MFCD-Net outperforms prior
methods in the compressed domain and more importantly,
our model has 11X fewer parameters and 3X fewer Flops,
dramatically improving the efficiency of video recognition
inference. This approach enables applying neural networks
exclusively in the compressed domain without compromising accuracy while accelerating performance