ReLU for Inference Acceleration
Keywords: Deep learning, knowledge distillation, quantization, object detection, tranformer, nlp
Abstract: Over the past decade, advancements in neural networks have outpaced human-level performance in a wide range of domains, including but not limited to natural language understanding and image generation. This progress has led to significantly larger networks with hundreds of billions of parameters, creating substantial computational demands. We propose the re-introduction of ReLU activation function to replace gradient-smooth alternatives during inference. We show that this can reduce computational costs while achieving minimal accuracy degradation with the help of specialized knowledge distillation training. The effectiveness of the proposed method is demonstrated by a wide variety of network architectures, covering popular applications such as image classification, object detection, and language modeling. We observed FPS improvement of 2-10% for Convolution based neural networks while observing only 1.8-2.6% accuracy degradation. The different Transformer networks demonstrated accuracy difference of < 1% between proposed ReLU and original GeLU networks with comparable QPS. The improvement in performance is significantly noticeable on AI accelerators like ours, with ReLU based convolution networks showcasing theoretical improvement of 41-74% compared to their SiLU based counterpart.
Primary Area: general machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6026
Loading