On Subagging Boosted Probit Model Trees

Tian Qin, Wei-Min Huang

Abstract:

With the insight of variance-bias decomposition, we design a new hybrid bagging-boosting algorithm
named SBPMT for classification problems. For the boosting part of SBPMT, we propose a new tree
model called Probit Model Tree (PMT) as base classifiers in AdaBoost procedure. For the bagging
part, instead of subsampling from the dataset at each step of boosting, we perform boosted PMTs
on each subagged dataset and combine them into a powerful "committee", which can be viewed an
incomplete U-statistic. Our theoretical analysis shows that (1) SBPMT is consistent under certain
assumptions, (2) Increase the subagging times can reduce the generalization error of SBPMT to
some extent and (3) Large number of ProbitBoost iterations in PMT can benefit the performance
of SBPMT with fewer steps in the AdaBoost part. Those three properties are verified by a famous
simulation designed by Mease and Wyner [2008]. The last two points also provide a useful guidance
in model tuning. A comparison of performance with other state-of-the-art classification methods
illustrates that the proposed SBPMT algorithm has competitive prediction power in general and
performs significantly better in some cases.