Coupled Ensembles of Neural Networks

Anuvabh Dutt, Denis Pellerin, Georges Quénot

Abstract: We investigate in this paper the architecture of deep convolutional networks. Building on existing state of the art models, we propose a reconfiguration of the model parameters into several parallel branches at the global network level, with each branch being a standalone CNN. We show that this arrangement is an efficient way to significantly reduce the number of parameters while at the same time improving the performance. The use of branches brings an additional form of regularization. In addition to splitting the parameters into parallel branches, we propose a tighter coupling of these branches by averaging their log-probabilities. The tighter coupling favours the learning of better representations, even at the level of the individual branches, as compared to when each branch is trained independently. We refer to this branched architecture as "coupled ensembles". The approach is very generic and can be applied with almost any neural network architecture. With coupled ensembles of DenseNet-BC and parameter budget of 25M, we obtain error rates of 2.92%, 15.68% and 1.50% respectively on CIFAR-10, CIFAR-100 and SVHN tasks. For the same parameter budget, DenseNet-BC has an error rate of 3.46%, 17.18%, and 1.8% respectively. With ensembles of coupled ensembles, of DenseNet-BC networks, with 50M total parameters, we obtain error rates of 2.72%, 15.13% and 1.42% respectively on these tasks.

TL;DR: We show that splitting a neural network into parallel branches improves performance and that proper coupling of the branches improves performance even further.

Keywords: Ensemble learning, neural networks

Code: [ vabh/coupled_ensembles](https://github.com/vabh/coupled_ensembles) + [ 1 community implementation](https://paperswithcode.com/paper/?openreview=Hk2MHt-3-)

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [CIFAR-100](https://paperswithcode.com/dataset/cifar-100), [SVHN](https://paperswithcode.com/dataset/svhn)

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1709.06053/code)