Sparling: End-to-End Spatial Concept Learning via Extremely Sparse Activations

Kavi Gupta, Osbert Bastani, Armando Solar-Lezama

Keywords: machine learning, sparsity, interpretability, optimization, identifiability

TL;DR: We prove it is possible to identify an extremely sparse intermediate latent variable with only end-to-end supervision, and introduce Sparling, an extreme activation sparsity layer and optimization algorithm that can learn such a latent variable

Abstract: Real-world processes often contain intermediate state that can be modeled as an extremely sparse activation tensor. In this work, we analyze the identifiability of such sparse and local latent intermediate variables, which we call motifs. We prove our Motif Identifiability Theorem, stating that under certain assumptions it is possible to precisely identify these motifs exclusively by reducing end-to-end error. Notably, we do not assume identifiability of parameters, but rather of a latent intermediate representation output by a local model, thus allowing these representations to be arbitrarily complex functions of the input. Additionally, we provide the Sparling algorithm, which uses a new kind of informational bottleneck that enforces levels of activation sparsity unachievable using other techniques. We confirm empirically that extreme sparsity is necessary to achieve good intermediate state modeling. On synthetic domains, we are able to precisely localize the intermediate states up to feature permutation with $>90\%$ accuracy, even though we only train end-to-end.

Primary Area: interpretability and explainable AI

Submission Number: 12895