Multi-modal Auto-regressive Modeling via Visual Tokens

Tianshuo Peng, Zuchao Li, Lefei Zhang, hai zhao, Ping Wang, Bo Du

Abstract: Large Language Models (LLMs), benefiting from the auto-regressive modelling approach performed on massive unannotated texts corpora, demonstrates powerful perceptual and reasoning capabilities. However, as for extending auto-regressive modelling to multi-modal scenarios to build Large Multi-modal Models (LMMs), there lies a great difficulty that the image information is processed in the LMM as continuous visual embeddings, which cannot obtain discrete supervised labels for classification. In this paper, we successfully perform multi-modal auto-regressive modeling with a unified objective for the first time. Specifically, we propose the concept of visual tokens, which maps the visual features to probability distributions over LLM's vocabulary, providing supervision information for visual modelling. We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information. Experimental results and ablation studies on 5 VQA tasks and 4 benchmark toolkits validate the powerful performance of our proposed approach.

Primary Subject Area: [Generation] Multimedia Foundation Models

Secondary Subject Area: [Content] Vision and Language, [Content] Multimodal Fusion, [Content] Media Interpretation

Relevance To Conference:

The extension of the successful paradigm of Large Language Model (LLM) to multimodal scenarios has always been challenging due to the inability to obtain discrete supervised labels from continuous visual features. In this work, we successfully performing multi-modal auto-regressive modeling with a unified objective. Our approach introduces the concept of visual tokens, which map visual features to probability distributions over LLM's vocabulary, providing supervision information for visual modeling. Furthermore, we explore the distribution of visual features in the semantic space within LLM and investigate the possibility of utilizing text embeddings to represent visual information.
These contributions offer a novel approach to integrating and aligning LLM and visual modality, while providing insights into the multimodal foundation model's processing and comprehension of multimodal content from an interpretable standpoint.

Submission Number: 5598