Parameter Density Inheritance Using Kernel Density Estimation for Efficient CNN Learning

Keisuke Horiuchi, Keisuke Kameyama

Abstract: CNNs have shown state-of-the-art performances on a large variety of classification tasks. However, because of its complexity, they may require an enormous amount of time for training and still not converge. One of the key ideas to address these problems is to initialize the network parameters with appropriate values before training. Several known methods proposed the initialization using an existing trained network, but they suppose both the trained network and the new network have the same structure. In this paper, we propose an initialization method that utilizes existing trained network and can be applied to a network with a different structure based on parameter density inheritance strategy. In the experiments, we verified the effectiveness of the proposed initialization method. When the network structures were the same between the trained network and the new network, the result showed that one of the existing methods is better than the proposed method. However, in the case that network structures are not the same, an improvement was observed using the proposed method.