Hardware-Friendly Mixed-Precision Neural Networks

Saqib Javed

Abstract: For the deployment of Convolutional Neural Networks (CNNs) on battery-powered, energy-constrained edge devices, both weights and activations in a network can be quantized to reduce the energy consumption associated with CNN inference, as low-precision integer arithmetic uses less energy to execute than operations on full-precision floating-point data. However, this quantization of weights and activations incurs a significant loss in accuracy when weights are quantized to 2-bits. Therefore, we did a thorough evaluation of mixed-precision quantization of neural networks in this work and propose Hardware-Friendly Mixed-Precision Neural Networks where this accuracy loss was reduced using mixed-precision networks but in a more hardware-friendly manner. Using INQ method as a base, we explored different weight partitioning schemes. With an unstructured quantization approach, we can achieve $\sim$95\% quantized weights with only 2\% loss in accuracy as compared to a full-precision model with one of the proposed weight partitioning methods. Moreover, we explored the fact that if we leave the first and the last layer unquantized, this drop decreases to only 1\%. The drop in accuracy is majorly contributed by the quantization of the last layer. In order to make it more amenable to hardware support, we impose a filter-wise structure on the intra-layer quantization. Under this constraint, extensive evaluation of the impact of quantizing the first and the last layer, order of quantization, and the impact of 8-bit activation quantization for ternary neural networks was performed. It was observed that quantizing the activations to 8-bits does not incur a significant loss and leaving the first and the last layer unquantized improves the accuracy significantly. Moreover, quantizing the high magnitude weights first provides the best final accuracy. Using these observations, one of our proposed hardware-friendly quantization strategies achieves the Top-1 validation accuracy of 63.6~\% on the ImageNet dataset using Resnet-18 architecture, where 90\% weights were quantized and the remaining 10\% were left in full-precision. The accuracy was increased by 1.5~\% as compared to the 100\% quantized network using the INQ method. Comparing our results to the other state-of-the-art CNN optimization methods, the proposed method provides a reasonable trade-off between a significant reduction of computational demand and energy required, and an acceptable degradation in the accuracy.