Panel: Multimodal Large Foundation Models

Mohan S. Kankanhalli, Marcel Worring

Abstract: The surprisingly fluent predictive performance of LLM (Large Language Models) as well as the high-quality photo-realistic rendering of Diffusion Models has heralded a new beginning in the area of Generative AI. Such kinds of deep learning based models with billions of parameters and pre-trained on massive-scale data-sets are also called Large Foundation Models (LFM). These models not only have caught the public imagination but also have led to an unprecedented surge in interest towards the applications of these models. Instead of the previous approach of developing AI models for specific tasks, more and more researchers are developing large task-agnostic models pre-trained on massive data, which can then be adapted to a variety of downstream tasks via fine-tuning, fewshot learning, or zero-shot learning. Some examples are ChatGPT, LLaMA, GPT-4, Flamingo, MidJourney, Stable-Diffusion and DALLE. Some of them can handle text (e.g., ChatGPT, LLaMA) while some others (e.g., GPT-4 and Flamingo) can utilize multimodal data and can hence be considered Multimodal Large Foundation Models (MLFM). Several recent studies have shown that when adapted to specific tasks (e.g., visual question answering), the foundation models can often surpass the performance of state-of-the-art, fully supervised AI models. However, applying foundation models to specialized domain tasks (e.g., medical diagnosis, financial recommendation etc.) raises many ethical issues (e.g., privacy, model bias or hallucinations). The panel members will discuss the emerging trends in the development and use of large multimodal foundation models. Some of the issues to be discussed are: Research issues in going from LLM to MLFM Behaviour of MLFM Application Potential of MLFM Trust issues in MLFM Limitations of MLFM Societal, Legal and Regulatory issues of MLFM Promising future research in MLFM This panel will bring together several leading experts from universities, research institutions, and industry who will discuss and debate together with the audience. We invite everybody to participate and contribute towards this important and promising research direction.