ADD: A Fine-grained Dynamic Inference Architecture for Semantic Image Segmentation

Chi-Hsi Kung, Che-Rung Lee

Abstract: Dynamic inference that adaptively skips parts of model execution based on the complexity of input data can effectively reduce the computation cost of deep learning models during the inference. However, current architectures for dynamic inference only consider the exits at the block level, whose results may not be suitable for different applications. In this paper, we present the Auto-Dynamic-DeepLab (ADD), a network architecture that enables the fine-grained dynamic inference for semantic image segmentation. To allow the exit points in the cell level, ADD utilizes Neural Architecture Search (NAS), supported by the framework of Auto-DeepLab, to seek the optimal network structure. In addition, ADD replaces the cells in Auto-DeepLab with the densely connected cells to ease the interference among multiple classifiers and employs the earlier decision maker (EDM) to further improve the performance. Experimental results show that ADD can achieve similar accuracy as Auto-DeepLab in terms of mIoU with a 1.6 times speedup. For the fast mode, ADD can achieve 2.15 times speedup with only a 1.3% accuracy drop compared to those of Auto-DeepLab.