TreeConnect: A Sparse Alternative to Fully Connected Layers

Oliver Richter, Roger Wattenhofer

Abstract: We present a generally applicable tree-like sparse multilayer architecture that has a balanced connection from all input neurons to all output neurons. If the ratio between input and output neurons is fixed, the parameters required by our architecture scale with O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1.5</sup> ) as compared to O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) in a fully connected layer, where n is the number of input neurons. Our sparse 2-layer architecture performs similar and/or superior when compared to its fully connected 1-layer and 2-layer counter parts on the IMDB review sentiment classification task, the Reuters news categorization task and the CIFAR-10 image classification task.