A Tight Error Bound for Deep Learning via Distribution and Loss Complexity

Khoat Than, Vinh Doan, Phuc Le, Phu Phung

Keywords: Generalization, Test error, Deep learning, distribution complexity, Loss landscape

TL;DR: A tighter estimate of test error for deep neural networks, exploiting both the complexity of the data distribution and the loss function.

Abstract: Generalization is a central requirement for machine learning models in real-world applications, yet theoretically verifying it for trained models - especially deep neural networks (NNs) - remains highly challenging. Existing generalization bounds are often vacuous for modern NN architectures. In this paper, we propose model-dependent bounds that connect a model’s behavior in data space with its generalization ability. Our bounds explicitly capture both the complexity of the data distribution and the loss function, and they highlight the role of alignment between data geometry and the loss landscape. These properties enable our bounds to obtain significantly tighter estimates of test error than prior approaches. Extensive experiments on ImageNet classification and segmentation models show that our tractable bound consistently provides the tightest (nonvacuous) test-error estimates across a wide range of large-scale NNs. Moreover, we find that some parts of our bounds can effectly track the dynamic of test error, offering new insights into how to understand and improve performance in deep learning.

Primary Area: learning theory

Submission Number: 5619