A Feature Transformation Technique for Improving Ensemble Learning Systems

Truong Thanh Nguyen, Hieu Vu, Truong Dang, Eyad Elyan, Thanh Son Vu, Tien Thanh Nguyen

Abstract: In this study, we propose a feature transformation approach to improve the performance of Ensemble Learning Systems. Our method operates on the predictions of base classifiers within an ensemble system, known as meta-data, which are produced by applying a Cross-Validation procedure on the training data using various learning algorithms. The goal is to transform the meta-data to obtain a better combined output. Specifically, we compute the center vectors associated with each class label on the meta-data. We then increase the dimensionality of the meta-data by concatenating it with residuals, which are obtained by subtracting each vector in the meta-data from the associated center vector. A classifier is then trained on this transformed meta-data to create the combiner which will be used to combine outputs of base classifiers for ensemble prediction. Our method is compared with three well-known ensemble methods: deep forest (gcForest), Decision Template (DT), Sum rule, and the same ensemble without transformation. We evaluated the experimental methods on 22 UCI datasets and used the Friedman and Nemenyi tests to statistically compare their performances. The results show that our method outperforms all base classifiers and the benchmark algorithms on experimental datasets.