A Koopman Approach to Understanding Sequence Neural Models

Ilan Naiman, Omri Azencot

Keywords: Koopman methods, sequence neural models, understanding deep learning

Abstract: Deep learning models are often treated as "black boxes". Existing approaches for understanding the decision mechanisms of neural networks provide limited explanations or depend on local theories. Recently, a data-driven framework based on Koopman theory was developed for the analysis of nonlinear dynamical systems. In this paper, we introduce a new approach to understanding trained sequence neural models: the Koopman Analysis of Neural Networks (KANN) method. At the core of our method lies the Koopman operator, which is linear, yet it encodes the dominant features of the network latent dynamics. Moreover, its eigenvectors and eigenvalues facilitate understanding: in the sentiment analysis problem, the eigenvectors highlight positive and negative n-grams; and, in the ECG classification challenge, the eigenvectors capture the dominant features of the normal beat signal.

One-sentence Summary: A linear framework for analyzing and understanding the inner mechanisms of sequence neural models

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2102.07824/code)