Intent Classification on the ATIS dataset: comparing classical and advanced NLP methods

Seong Woo Ahn, Ladislas Letourneur

Abstract: This report presents a benchmarking study on intent classification using the ATIS dataset. The study aims to compare the performance of traditional machine learning models, such as Naive Bayes and Support Vector Machines (SVM), against advanced NLP architectures, including DistilBERT, which is a state-ofthe-art transformer-based model. Our results demonstrate that advanced NLP architectures, particularly DistilBERT, significantly outperform traditional machine learning models in terms of accuracy, precision, and recall, even when the data is highly imbalanced. These findings indicate that transformer-based models are highly effective in solving the intent classification problem and have significant implications for the design and development of natural language processing systems. These results are particularly relevant for intent classification tasks in specific domains, such as airline reservations.