Revisiting differentially private XGBoost: are random decision trees really better than greedy ones?
Keywords: Differential Privacy, Gradient Boosting, Decision Trees
Abstract: Boosted Decision Trees (e.g., XGBoost) are one of the strongest and most widely used machine learning models. Motivated by applications in sensitive domains, various versions of Boosted Decision Tree learners with provably differential privacy (DP) guarantees were designed. Contrary to their non-private counterparts, Maddock et al. (2022) reported a surprising finding that private boosting with random decision trees outperforms a more faithful privatization of XGBoost that uses greedy decision trees. In this paper, we challenge this conclusion with an improved DP-XGBoost algorithm and a thorough empirical study. Our results reveal that while random selection is still slightly better in most datasets, greedy selection is not far behind after our improved DP analysis. Moreover, if we restrict the number of trees to be small (e.g., for interpretability) or if interaction terms are important for prediction, then random selection often fails catastrophically while greedy selection (our method) prevails.
Primary Area: societal considerations including fairness, safety, privacy
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6548
Loading