ResNet-Lite: On Improving Image Classification with a Lightweight Network

Shahriar Shakir Sumit, Sreenatha G. Anavatti, Murat Tahtali, Seyedali Mirjalili, Ugur Turhan

Abstract: Deep learning methods, specifically convolutional neural networks (CNNs), have achieved state-of-the-art in various tasks including image classification. However, the computational and memory requirements of advanced CNNs like ResNet-50 pose deployment challenges, particularly in resource-constrained environments. Our study introduces a new lightweight approach, namely ResNet-Lite, for image classification. It combines knowledge distillation and network tuning along with hyperparameter tuning techniques to overcome deployment barriers. ResNet-Lite involves the extracting of knowledge from a pre-trained ResNet-50 network and transferring it to a smaller network, creating a more compact and effective approach. After that, hyperparameter and network tuning have been applied to determine the optimal combination of parameters that maximizes the generalization and performance of the model. Our experimental results demonstrate that ResNet-Lite achieves a significantly reduced model size while maintaining competitive classification performance. Specifically, it outperforms the original ResNet-50 model by 5.40% and by 7.13% in accuracy on the CIFAR-10 and Fashion-MNIST datasets, respectively. In summary, our study provides a practical solution for developing high-performance image classification models, even in resource-constrained environments, contributing to the field of advanced deep learning.