Effect of top-down connections in Hierarchical Sparse Coding

Victor Boutin, Angelo Franciosini, Franck Ruffier, Laurent Perrinet

Keywords: Hierarchical Sparse Coding, Convolutional Sparse Coding, Top-down connections

TL;DR: This paper experimentally demonstrates the beneficial effect of top-down connections in Hierarchical Sparse Coding algorithm.

Abstract: Hierarchical Sparse Coding (HSC) is a powerful model to efficiently represent multi-dimensional, structured data such as images. The simplest solution to solve this computationally hard problem is to decompose it into independent layerwise subproblems. However, neuroscientific evidence would suggest inter-connecting these subproblems as in the Predictive Coding (PC) theory, which adds top-down connections between consecutive layers. In this study, a new model called Sparse Deep Predictive Coding (SDPC) is introduced to assess the impact of this inter-layer feedback connection. In particular, the SDPC is compared with a Hierarchical Lasso (Hi-La) network made out of a sequence of Lasso layers. A 2-layered SDPC and a Hi-La networks are trained on 3 different databases and with different sparsity parameters on each layer. First, we show that the overall prediction error generated by SDPC is lower thanks to the feedback mechanism as it transfers prediction error between layers. Second, we demonstrate that the inference stage of the SDPC is faster to converge than for the Hi-La model. Third, we show that the SDPC also accelerates the learning process. Finally, the qualitative analysis of both models dictionaries, supported by their activation probability, show that the SDPC features are more generic and informative.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2002.00892/code)