AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models

Daqin Luo, Chengjian Feng, Yuxuan Nong, Yiqing Shen

Abstract:

Automated Machine Learning (AutoML) offers a promising approach to streamline the training of machine learning models. However, existing AutoML frameworks are often limited to unimodal scenarios and require extensive manual configuration. Recent advancements in Large Language Models (LLMs) have showcased their exceptional abilities in reasoning, interaction, and code generation, presenting an opportunity to develop a more automated and user-friendly framework. To this end, we introduce AutoM3L, an innovative Automated Multimodal Machine Learning framework that leverages LLMs as controllers to automatically construct multimodal training pipelines. AutoM3L comprehends data modalities and selects appropriate models based on user requirements, providing automation and interactivity. By eliminating the need for manual feature engineering and hyperparameter optimization, our framework simplifies user engagement and enables customization through directives, addressing the limitations of previous rule-based AutoML approaches. We evaluate the performance of AutoM3L on six diverse multimodal datasets spanning classification, regression, and retrieval tasks, as well as a comprehensive set of unimodal datasets. The results demonstrate that AutoM3L achieves competitive or superior performance compared to traditional rule-based AutoML methods. Furthermore, a user study highlights the user-friendliness and usability of our framework,
compared to the rule-based AutoML methods. Code is available at: https://anonymous.4open.science/r/anonymization_code.

Primary Subject Area: [Content] Multimodal Fusion

Relevance To Conference: AutoM3L explores automated pipeline construction, automated feature engineering, and automated hyperparameter optimization. This enables the realization of an end-to-end multimodal AutoML framework. Leveraging the exceptional capabilities of LLMs, AutoM3L provides adaptable and accessible solutions for multimodal data tasks that are equally applicable to single-modal scenarios. Besides, it offers automation, interactivity, and user customization.

Submission Number: 1432