AutoDW: Automatic Data Wrangling Leveraging Large Language Models

Lei Liu, So Hasegawa, Shailaja Keyur Sampat, Maria Xenochristou, Wei-Peng Chen, Takashi Kato, Taisei Kakibuchi, Tatsuya Asai

Abstract: Data wrangling is a critical yet often labor-intensive process, essential for transforming raw data into formats suitable for downstream tasks such as machine learning or data analysis. Traditional data wrangling methods can be time-consuming, resource-intensive, and prone to errors, limiting the efficiency and effectiveness of subsequent downstream tasks. In this paper, we introduce AutoDW: an end-to-end solution for automatic data wrangling that leverages the power of Large Language Models (LLMs) to enhance automation and intelligence in data preparation. AutoDW distinguishes itself through several innovative features, including comprehensive automation that minimizes human intervention, the integration of LLMs to enable advanced data processing capabilities, and the generation of source code for the entire wrangling process, ensuring transparency and reproducibility. These advancements position AuoDW as a superior alternative to existing data wrangling tools, offering significant improvements in efficiency, accuracy, and flexibility. Through detailed performance evaluations, we demonstrate the effectiveness of AutoDW for data wrangling. We also discuss our experience and lessons learned from the industrial deployment of AutoDW, showcasing its potential to transform the landscape of automated data preparation.