Oblique Bayesian Additive Regression Trees

Paul-Hieu V. Nguyen, Ryan Yee, Sameer Deshpande

Abstract: Current implementations of Bayesian Additive Regression Trees (BART) are based on axis-aligned decision rules that recursively partition the feature space using a single feature at a time. Several authors have demonstrated that oblique trees, whose decision rules are based on linear combinations of features, can sometimes yield better predictions than axis-aligned trees and exhibit excellent theoretical properties. We develop an oblique version of BART that leverages a data-adaptive decision rule prior that recursively partitions the feature space along random hyperplanes. Using several synthetic and real-world benchmark datasets, we systematically compared our oblique BART implementation to axis-aligned BART and other tree ensemble methods, finding that oblique BART was competitive with --- and sometimes much better than --- those methods.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We have added a new section (Sec 4.2.2) that includes additional empirical comparisons requested by the reviewers. We have also added some clarifications and fixed several grammatical and typographical errors.

Second revision: we re-ran our experiments, this time tuning the hyperparameters of each non-BART-based method. We have updated the relevant figures in the Section 4.2 and tables in the Appendix. We also added a table showing the grid of hyperparameter values considered for each method

Third revision: corrected an issue with pre-processing categorical predictors for competing oblique ensemble methods. Re-ran experiments and updated the empirical results and exposition.

Final revision/camera-ready upload: we have fixed the identified typos, updated the caption of Figure 4, addressed the capitalization in Table A2, and added a de-anonymized link to our code. Added the OpenReview link.

Assigned Action Editor: Satoshi Hara

Submission Number: 4094