Training Multilayer Neural Networks Analytically Using Kernel Projection

Huiping Zhuang, Zhiping Lin, Kar-Ann Toh

Abstract: This paper proposes a kernel projection (KP) neural network that analytically determines its network parameters. The proposed network is composed of cascaded modules of 2-layer sub-networks. A technique which encodes the label information into each module has been introduced to enable a locally supervised learning. Such a supervised learning in the 2-layer module begins with a kernel projection in the first layer and determines its parameters analytically via solving a least squares problem in the second layer. We show that the analytic nature of the proposed network allows a learning process significantly faster than that of the traditional backpropagation method as it only needs to visit the dataset once. Experiments of classification tasks on various datasets are carried out, showing comparable or better results compared with several competing methods.