Diversity-Based Ensemble with Sample Weight Learning

Chun Yang, Xu-Cheng Yin, Hong-Wei Hao

Abstract: Given multiple classifiers, one prevalent approach in classifier ensemble is to diversely combine classifier components (diversity-based ensemble), and a lot of previous works show that this approach can improve accuracy in classification. However, how to measure diversity and perform diversity-based learning are still challenges in the literature. Moreover, the learning procedure highly depends upon the distribution of the training data. In this paper, we propose a novel classifier ensemble method which combines classifiers with both diversity and sample weighting. First, by designing a matrix for the (sample) data distribution creatively, we formulate a unified optimization model for diversity-based ensemble with sample weighting, where classifier weights are learned through a convex quadratic programming problem with given sample weights. Second, we propose a new self-training algorithm to iteratively run the convex optimization and automatically learn the sample weights. Moreover, these sample weights are updated with a dynamically damped learning trick, which has a good performance for convergence. This paper also discusses the relationship between our optimization model and the margin theory. Extensive experiments on a variety of 50 UCI classification benchmark data sets show that the proposed approach consistently outperforms conventional ensembles such as Bagging, GASEN, and SDP.