Improving Generalization in Deep Neural Networks by Mitigating Memorization

Yong Zhuang, Tianyu Kang, Zhen Lu, Wei Ding, Ping Chen

Abstract: Better model generalization is one key objective in machine learning. Although deep neural networks (DNNs) achieve impressive progress in many real-world applications, there is still a lack of fundamental understanding of why and when a DNN model generalizes well, which may hinder further improvement of DNN models. In this paper, we introduce a novel generalization measure, the Model Activation Ratio (\(\bar{\varphi }\)), which quantifies how broadly patterns and neurons of a DNN represent the training data. By measuring the average number of samples each neuron or pattern processes, \(\bar{\varphi }\) provides valuable insights into the interplay between memorization and generalization. In addition to quantifying generalization, we propose a training procedure called Progressive Neuron Rebirth (ProNeR), which integrates \(\bar{\varphi }\) directly into the training process. PRoNeR selectively reinitializes underutilized neurons to promote more balanced activation coverage, guiding the model toward improved generalization. We validate our approach with extensive experiments on several image benchmark datasets, using a variety of DNN architectures. The results show that incorporating \(\bar{\varphi }\) and PRoNeR leads to consistent and significant improvements in generalization performance.

External IDs:dblp:conf/pakdd/ZhuangKLDC25