Understanding intermediate layers using linear classifier probes

Guillaume Alain, Yoshua Bengio

Abstract: Neural network models have a reputation for being black boxes. We propose a new method to better understand the roles and dynamics of the intermediate layers. Our method uses linear classifiers, referred to as "probes", where a probe can only use the hidden units of a given intermediate layer as discriminating features. Moreover, these probes cannot affect the training phase of a model, and they are generally added after training. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems.