Evolution of Convolutional Highway Networks

Oliver Kramer

Abstract: Convolutional highways are based on multiple stacked convolutional layers for feature preprocessing. Like many other convolutional networks convolutional highways are parameterized by numerous hyperparameters that have to be tuned carefully. We introduce an evolutionary algorithm (EA) for optimization of the structure and tuning of hyperparameters of convolutional highways and demonstrate the potential of this optimization setting on the well-known MNIST data set. The EA employs Rechenberg’s mutation rate control and a niching mechanism to overcome local optima. An experimental study shows that the EA is capable of evolving convolutional highway networks from scratch with only few evaluations but achieving competitive accuracy. Further, the EA is able to significantly improve standard network configurations.