Proximal Regularization of Deep Residual Neural Networks with An Application to Genomic Prediction

TMLR Paper5294 Authors

Abstract: Residual neural networks (ResNets) have become widely used as they allow for smooth and efficient training of deep neural network architectures. However, when trained on small, noisy and high-dimensional data, ResNets may suffer from overfitting due to the large amount of parameters. As a solution, a range of regularization methods have been proposed. One promising approach relies on the proximal mapping technique which is computationally efficient since it can be directly incorporated into the optimization algorithm. However, the performance of ResNets with various convex or non-convex proximal regularizers remains under-explored on high-dimensional data. In our study, we develop a stochastic adaptive proximal gradient ResNet method that can handle both convex and non-convex regularizers that range from $L_0$ to $L_{\infty}$. Moreover, we evaluate the prediction performance in a supervised regression setting on three high-dimensional genomic data sets from mice, pig and wheat. Traditional sparse linear proximal gradient methods are also implemented with the same regularizers and evaluated for comparison. Experimental results demonstrate that a ResNet with 18-layers and $L_{\frac{1}{2}}$ regularization outperforms other configurations on both mice and pig datasets, as well as the sparse linear proximal gradient methods across all the datasets. For the wheat data, a 15-layer ResNet configuration achieves the lowest test mean squared error. These findings highlight the effectiveness of our regularized adaptive proximal gradient ResNet method and its potential for prediction tasks on high-dimensional genomic data.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=mYoE23a3bc

Changes Since Last Submission: The Acknowledgements has been removed.

Assigned Action Editor: ~Mathurin_Massias1

Submission Number: 5294