Zero-Shot Decision Tree Construction via Large Language Models

ACL ARR 2025 May Submission4559 Authors

Abstract: We present a zero-shot algorithm for building decision trees with large language models (LLMs) based on CART principles. Unlike traditional methods, which require labeled data, our approach uses the pretrained knowledge of LLMs to perform key operations such as feature discretization, probability estimation, and Gini-based split selection without training data. We also introduce a few-shot calibration step that refines the zero-shot tree with a small set of labeled examples. The resulting trees are interpretable, achieve competitive performance on tabular datasets, and outperform existing zero-shot baselines while approaching supervised models in low-data regimes. Our method provides a transparent, knowledge-driven alternative for decision tree induction in settings with limited data.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: NLP Applications, Efficient/Low-Resource Methods for NLP, Machine Learning for NLP

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: english, spanish

Submission Number: 4559