INTERPRETABLE COMPACT CATEGORICAL FEATURES ENCODING FOR SUPERVISED LEARNING

Stephen Zhou

Keywords: Supervised Learning, Categorical Feature Encoding, Compression, Embedding

TL;DR: Interpretable categorical feature encoding

Abstract: In supervised learning, encoding techniques for continuous features are well studied. However, few are specific for categorical features. The categorical encoding approaches that are widely used are one-hot encoding, target encoding and its variant ordered target encoding. In many cases categorical features carry significant, if not a dominant portion of, feature information in a supervised learning problem. Therefore they are key to improve model fit. One hot encoding is known for its curse of dimensionality issue, especially when categorical features are of high cardinality and/or sparse. Such problem not only increases the problem size but may also introduce instability for numerical solvers. Target encoding and its variant ordered target encoding are often used to address such data issues. It is fast and compact. The downside is that target encoding tend to overfit due to the way it is implemented. To our knowledge, the other categorical encoding methods used in machine learning and deep learning algorithms do not preserve interpretability. Our goal is to bridge the gap between dimension reduction, accuracy, feature interpretability, and scalability. In this paper, we introduce a polynomial algorithm called Interpretable Compact Categorical Feature Encoding for Supervised Learning (ICFESL). Under reasonable assumption, our encoding technique ensures no information loss for regression and minimum information loss for classification. At the core, it leverages L2 regularized linear models to efficiently calculate coefficients for one-hot-encoded categorical features and group them together without transforming them. We prove that applying K-means clustering for the grouping problem yields optimal solutions. We test our algorithms on simulations and real-world datasets both in regression and classification to validate the assumption and demonstrate the encoding method’s performance. The results show that for regressions, ICFESL enabled linear models and xgBoost models often significantly outperform state-of-the-art algorithms such as CatBoost and TabNet in terms of RMSE. The results also show that for classifications, ICFESL has comparable performance and outperforms CatBoost measured by AUC when ordered target encoding shows significant overfitting. We demonstrate how interpretability is preserved with example clusters from one of the experiments.

Primary Area: interpretability and explainable AI

Submission Number: 4587