Generative Models over Neural Controllers for Transfer Learning

James Butterworth, Rahul Savani, Karl Tuyls

Abstract: We introduce a technique that leverages the power of indirect encodings (IE) from the field of evolutionary computation to improve the speed of evolution in transfer learning control tasks. Although generative models have previously been used to construct IEs, their potential in transfer learning, specifically in reinforcement learning domains, has not yet been utilised. We train three types of generative models: an autoencoder (AE), a variational autoencoder (VAE) and a generative adversarial network (GAN) on the neural network weights of well-performing solutions of a set of paramaterised source domains. The decoder of the AE and VAE or the generator of the GAN is then used as the IE in an evolutionary run on unseen, but related, target domains. We compare against two baselines: a direct encoding (DE) and a DE starting evolution from a controller pre-trained to maximise the average fitness over the set of source domains. We show that, by using these IEs, the speed of learning on the target domains is greatly increased with respect to the baselines.