FELIX: Automatic and Interpretable Feature Engineering Using LLMs

Simon Malberg, Edoardo Mosca, Georg Groh

Abstract: Pre-processing and feature engineering are essential yet labor-intensive components of NLP. Engineers must often balance the demand for high model accuracy against interpretability, all while having to deal with unstructured data. We address this issue by introducing F eature E ngineering with L LMs for I nterpretability and E x plainability (FELIX), a novel approach harnessing the vast world knowledge embedded in pre-trained Large Language Models (LLMs) to automatically generate a set of features describing the data. These features are human-interpretable, bring structure to text samples, and can be easily leveraged to train downstream classifiers. We test FELIX across five different text classification tasks, showing that it performs better than feature extraction baselines such as TF-IDF and LLM’s embeddings as well as s.o.t.a. LLM’s zero-shot performance and a fine-tuned text classifier. Further experiments also showcase FELIX’s strengths in terms of sample efficiency and generalization capabilities, making it a low-effort and reliable method for automatic and interpretable feature extraction. We release our code and supplementary material: https://github.com/simonmalberg/felix.