Adaptive complexity knee point selection in multi-objective genetic programming for improving generalization of evolutionary feature construction in regression

Hengzhe Zhang, Qi Chen, Bing Xue, Wolfgang Banzhaf, Mengjie Zhang

Abstract: Multi-objective genetic programming-based feature construction has emerged as a powerful approach to enhancing regression performance. While this approach has demonstrated considerable success across various domains, overfitting remains a significant challenge, particularly when dealing with limited and/or noisy training data. To address this, researchers have developed multi-objective methods that balance model complexity and accuracy. However, these methods invariably produce a Pareto front containing multiple solutions with different trade-offs, raising the critical question of how to select the most appropriate model for deployment. In this paper, we propose a novel Adaptive Complexity Knee Point (ACKP) selection strategy for evolutionary multi-objective feature construction in regression tasks. Our approach adaptively selects between minimal complexity knee points, which favor the simplest knee solution, and traditional knee points, based on estimated dataset difficulty. This adaptive mechanism reduces overfitting in noisy scenarios while mitigating underfitting in low-noise datasets by allowing more complex models when appropriate. Comprehensive experiments on 58 real-world datasets demonstrate that ACKP significantly outperforms nine established model selection strategies, and that genetic programming with ACKP outperforms several mainstream machine learning algorithms, particularly when dealing with sample-limited and noisy datasets, highlighting its practical value in real-world problems.

External IDs:dblp:journals/gpem/ZhangCXBZ25