An SMC Sampler for Decision Trees with Enhanced Initial Proposal for Stochastic Metaheuristic Optimization

Efthyvoulos Drousiotis, Alessandro Varsi, Paul G. Spirakis, Simon Maskell

Abstract: Bayesian Decision Trees (DTs) have emerged as a significant tool for Machine Learning (ML) tasks, offering better performance compared to traditional approaches. However, the convergence of Bayesian Trees can be affected by slow initialization, particularly when employing the random method. This paper addresses this limitation by proposing an enhanced initialization strategy for the Sequential Monte Carlo (SMC) sampler applied to DTs. We demonstrate both experimentally and theoretically that our approach accelerates convergence to the posterior distribution. Our contribution lies in introducing an SMC sampler for DTs with a refined initialization strategy. By leveraging a more sophisticated approach, we achieve faster convergence to the posterior distribution, as substantiated by experimental results and theoretical analysis. Unlike conventional random initialization methods, our proposed approach unlocks the full potential of the underlying model.