Interpreting video features: a comparison of 3D convolutional networks and convolutional LSTM networks

Joonatan Mänttäri*, Sofia Broomé*, John Folkesson, Hedvig Kjellström

Keywords: interpretability, spatiotemporal, video, features, saliency, temporal

TL;DR: We investigate what spatiotemporal features are focused on in video data by two models that are principally different in the way that they model temporal dependencies.

Abstract:

A number of techniques for interpretability have been presented for deep learning
in computer vision, typically with the goal of understanding what it is that the networks
have actually learned underneath a given classification decision. However,
when it comes to deep video architectures, interpretability is still in its infancy and
we do not yet have a clear concept of how we should decode spatiotemporal features.
In this paper, we present a study comparing how 3D convolutional networks
and convolutional LSTM networks respectively learn features across temporally
dependent frames. This is the first comparison of two video models that both
convolve to learn spatial features but that have principally different methods of
modeling time. Additionally, we extend the concept of meaningful perturbation
introduced by Fong & Vedaldi (2017) to the temporal dimension to search for the
most meaningful part of a sequence for a classification decision.

Code: https://www.dropbox.com/sh/48kva9cw7keobi5/AABdrhuwzSwohA-Jq-TIsG80a?dl=0

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2002.00367/code)