GradTree: Learning Axis-Aligned Decision Trees with Gradient Descent

Published: 28 Oct 2023, Last Modified: 03 Nov 2023TRL @ NeurIPS 2023 PosterEveryoneRevisionsBibTeX
Keywords: Decision Trees, Gradient Descent
TL;DR: A novel approach to learn hard, axis-aligned decision trees with gradient descent using the straight-through operator on a dense decision tree representation
Abstract: Decision Trees (DTs) are commonly used for many machine learning tasks due to their high degree of interpretability. However, learning a DT from data is a difficult optimization problem, as it is non-convex and non-differentiable. Therefore, common approaches learn DTs using a greedy growth algorithm that minimizes the impurity locally at each internal node. Unfortunately, this greedy procedure can lead to inaccurate trees. In this paper, we present a novel approach for learning hard, axis-aligned DTs with gradient descent. The proposed method uses backpropagation with a straight-through operator on a dense DT representation, to jointly optimize all tree parameters. Our approach outperforms existing methods on a wide range of binary classification benchmarks and is available under:
Submission Number: 1