Explore Positive Noise in Deep Learning

Xiaowei Yu, Yao Xue, Li Wang, Lu Zhang, Yanjun Lyu, Tianming Liu, Dajiang Zhu

Keywords: Deep learning, Positive Noise

Abstract: In computer vision, noise is conventionally viewed as a harmful perturbation in various deep learning architectures, such as convolutional neural networks (CNNs) and vision transformers (ViTs), as well as different tasks like image classification and transfer learning. However, this paper aims to rethink whether the conventional proposition always holds. We demonstrate that specific noise can boost the performance of various deep architectures under certain conditions. We theoretically prove the enhancement gained from positive noise by reducing the task complexity defined by information entropy and experimentally show the significant performance gain in large image datasets, such as the ImageNet. Herein, we use the information entropy to define the complexity of the task. We categorize the noise into two types, positive noise (PN) and harmful noise (HN), based on whether the noise can help reduce the complexity of the task. Extensive experiments of CNNs and ViTs have shown performance improvements by proactively injecting positive noise, where we achieve an unprecedented top 1 accuracy over 95$\%$ on ImageNet. Both theoretical analysis and empirical evidence have confirmed the presence of positive noise, which can benefit the learning process, while the traditionally perceived harmful noise indeed impairs deep learning models. The different roles of noise offer new explanations for deep models on specific tasks and provide a new paradigm for improving model performance. Moreover, it reminds us to utilize noise rather than suppress noise.

Submission Number: 10286