P-Mixup: Improving Generalization Performance of Evolutionary Feature Construction with Pessimistic Vicinal Risk Minimization

Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, Mengjie Zhang

Abstract: Genetic programming (GP)-based feature construction has achieved great success as an automated machine learning technique to improve learning performance. The key challenge in GP-based feature construction is that it is easy to overfit the training data. In supervised learning, unseen data usually lie in the vicinity of the training data and behave similar to the training data. However, a rugged model may make significantly different predictions, thus resulting in poor generalization performance. Here, we propose pessimistic vicinal risk minimization method to control overfitting in GP-based feature construction. The idea is to minimize the worst-case loss on vicinal examples of training instances, where vicinal examples are synthesized using an instance-wise mixing method. The experimental results on 58 datasets demonstrate that GP with the proposed overfitting control method clearly outperforms standard GP and seven other overfitting control methods for GP, validating the superiority of using pessimistic vicinal risk minimization to control overfitting in GP for feature construction.