Exploring Hessian Regularization in Mixup

Kosuke Sugiyama, Masato Uchida

Abstract: Mixup is a data augmentation technique that gener-ates new samples using a convex combination of samples. Despite its remarkable simplicity, this method significantly improves the generalization performance of Deep Neural Networks (DNNs). Various studies have been conducted to understand Mixup, and it has been theoretically shown that Mixup involves Jacobian regularization. Furthermore, a relationship between Mixup and label smoothing has been suggested. However, these studies are limited to cases where the sample generated by Mixup is close to the original sample, and other cases have not yet been analyzed. In this study, we theoretically proved that applying Mixup to logistic regression results in Hessian regularization when the sample generated by Mixup is far from the original sample, a previously unexplored scenario. Additionally, through numerical experiments, we confirmed that a similar tendency holds true for DNNs as well. Our results provide a novel interpretation that Mixup is a method for collectively approximating several regularization methods, including Jacobian regularization, label smoothing, and Hessian regularization.