ACC: Adaptive Compression Framework for Efficient On-device CNN Training

Gilha Lee, Seungil Lee, Hyun Kim

Abstract: Network compression methods have been studied in various forms, such as pruning and quantization, to enable the deployment of large-scale convolutional neural networks (CNNs) in resource-constrained environments. However, several challenges remain in utilizing these compressed CNNs on resource-constrained platforms (e.g., on-device environments). In particular, previous studies on network compression have mostly focused on inference, and pruning/quantization techniques have been performed separately. To address these issues, this paper proposes a new compression technique, called an adaptive compression framework for CNN training (ACC), which combines the advantages of conventional compression techniques. The ACC is an adaptive solution that addresses the memory bottleneck by reducing the resolution of activation/gradient in the beginning layers of the CNN with weights/activations/gradients all quantized to 8 bits and pruning a large number of CNN filters in the subsequent layers. In addition, the large kernel convolution compression (LKCC) included in the ACC helps minimize the amount of information loss and effectively reduces memory and computation by applying a 2×2 average pooling filter to the activation/gradient of the beginning layers. Fine-tuning experiments using the ResNet18 model on the CIFAR-100 dataset showed that the proposed ACC framework can achieve efficient CNN training on mobile/edge devices by reducing memory consumption and FLOPs by 85% and 37%, respectively, with only a negligible performance degradation of 0.17% compared with the baseline.