Lightweight Deep Learning: An Overview

Ching-Hao Wang, Kang-Yang Huang, Yi Yao, Jun-Cheng Chen, Hong-Han Shuai, Wen-Huang Cheng

Abstract: With the recent success of the deep neural networks (DNNs) in the field of artificial intelligence, the urge of deploying DNNs has drawn tremendous attention because it can benefit a wide range of applications on edge or embedded devices. Lightweight deep learning indicates the procedures of compressing DNN models into more compact ones, which are suitable to be executed on edge devices due to their limited resources and computational capabilities while maintaining comparable performance to the original. Currently, the approaches of model compression include but are not limited to network pruning, quantization, knowledge distillation, and neural architecture search. In this work, we present a fresh overview to summarize recent development and challenges for model compression.