Revisiting Ternary Neural Networks towards Asymmetric Thresholds and Uniform Distribution
Primary Area: general machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Ternary neural networks, asymmetric thresholds, uniform distribution
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: Along with the progress of ternary logic circuits, we are motivated to revisit the ternary neural network. In this paper, we propose a novel ternary quantized method that utilize asymmetric thresholds and uniform distribution.
Abstract: Recently, researchers have made significant progress in ternary logic circuits, which has spurred the utilization of Ternary Neural Network (TNN) due to its compatibility with ternary coding instead of the 2-bit coding used in binary system. However, TNN exhibits significant accuracy degradation compared to its full-precision counterpart. Therefore, we are motivated to revisit ternary neural networks and enhance their performance. To fully leverage the limited representation space, we apply a uniform distribution to three quantized values {-1,0,+1} to maximize the information entropy. To balance the representation ability of TNN while considering convenient hardware implementation, we adopt the asymmetric thresholds and symmetric scaling factors quantization scheme and introduce the bi-STE optimization method. Moreover, a two-stage knowledge distillation scheme is employed to further enhance the performance. Experimental results demonstrate the effectiveness of the proposed method for TNNs, achieving a top-1 accuracy of 74.5% for ResNet-50 on ImageNet. This outperforms previous ternary quantization methods by a large margin and even surpasses representative 2-bit quantization methods such as LSQ (73.7%).
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7278
Loading