Invited: Algorithm-Software-Hardware Co-Design for Deep Learning Acceleration

Zhengang Li, Yanyue Xie, Peiyan Dong, Olivia Chen, Yanzhi Wang

Abstract: With the development of AI techniques, it is appealing but challenging to efficiently deploy deep neural networks on resource-constrained devices. This paper presents two novel algorithm-software-hardware co-designs for improving the performance of deep neural networks. The first part introduces a hardware-efficient adaptive token pruning framework for Vision Transformers (ViTs) on FPGA, which achieves significant speedup under similar model accuracy. The second part introduces a design automation flow for crossbar-based Binary Neural Network (BNN) accelerators using the emerging technique Adiabatic Quantum-Flux-Parametron (AQFP). The proposed method significantly improves energy efficiency by combining AQFP with BNN together, which achieves over 100× better energy efficiency compared with the previous representative AQFP-based framework. Both proposed designs demonstrate superior performance compared to existing methods.