LBC: Language-Based-Classifier for Out-Of-Variable Generalization

Kangjun Noh, Baekryun Seong, Hoyoon Byun, Youngjun Choi, Sungjin Song, Kyungwoo Song

Keywords: Large Language Model, Pre Trained Knowledge, Out Of Variable Generalization

TL;DR: The Language-Based-Classifier (LBC) uses Large Language Models (LLMs) for tabular data classification, excelling in out-of-variable (OOV) tasks.

Abstract: Large Language Models (LLMs) have excelled in natural language processing tasks, but their application in tabular data classification has been limited compared to traditional machine learning models (TMLs) like XGBoost. However, LLMs hold potential in this area due to their ability to interpret context between variables using pre-trained knowledge, which is particularly useful in out-of-variable (OOV) tasks—situations with numerous missing values or new variables. We propose the Language-Based-Classifier (LBC) methodology, which excels in handling OOV tasks by converting tabular data into natural language prompts and leveraging pre-trained knowledge for better inference. LBC uses three strategies: 1) Categorical adjustments for model compatibility, 2) Enhanced data representation through advanced order and indicators, and 3) Logit score mapping to classes via a verbalizer. These strategies highlight LBC’s effectiveness in OOV tasks, making it the first study to apply an LLM-based model in this context, with empirical and theoretical validation of its superiority.

Submission Number: 7