A Low-Memory Learning Formulation for a Kernel-and-Range Network

Huiping Zhuang, Zhiping Lin, Kar-Ann Toh

Abstract: Recently, a learning method based on the kernel and the range space projections has been introduced. This method has been applied to learn the multilayer network analytically with interpretable relationships among the weight matrices. However, the learning method carries a high-memory demand during training. In this study, a low-memory formulation is proposed to address this issue of high-memory demand. The developed method is inspired by a recursive implementation of the Moore-Penrose inverse and is shown to be mathematically equivalent to the original batch learning. Next, we further improved our proposed low-memory formulation to annul the potential divergence caused by rounding errors. The regression and classification behaviors of the proposed learning method are demonstrated using both synthetic and benchmark datasets. Our experiments confirm that the proposed formulation consumes significantly lower memory.