Understanding the Emergence of Multimodal Representation Alignment

Megan Tjandrasuwita, Chanakya Ekbote, Liu Ziyin, Paul Pu Liang

TL;DR: We study the emergence of implicit alignment between modalities and find that its impact on performance depend on factors such as modality similarity and information balance, suggesting that alignment is not always beneficial for optimal performance.

Abstract: Multimodal representation learning is fundamentally about transforming incomparable modalities into comparable representations. While prior research has primarily focused on *explicitly* aligning these representations through targeted learning objectives and model architectures, a recent line of work has found that independently trained unimodal models of increasing scale and performance can become *implicitly* aligned with each other. These findings raise fundamental questions regarding the emergence of aligned representations in multimodal learning. Specifically: (1) when and why does alignment emerge implicitly? and (2) is alignment a reliable indicator of performance? Through a comprehensive empirical investigation, we demonstrate that both the emergence of alignment and its relationship with task performance depend on several critical data characteristics. These include, but are not necessarily limited to, the degree of similarity between the modalities and the balance between redundant and unique information they provide for the task. Our findings suggest that alignment may not be universally beneficial; rather, its impact on performance varies depending on the dataset and task. These insights can help practitioners determine whether increasing alignment between modalities is advantageous or, in some cases, detrimental to achieving optimal performance.

Lay Summary:

When does aligning different types of data, like images and text, help improve machine learning models? In our research, "aligning" means making different types of data, like images and text, compatible so the model can understand and process them in a similar way. Recent work has shown that models trained separately on each data type can naturally align as they become more advanced. This raises important questions: when and why does alignment emerge on its own, and does it always lead to better performance?
To answer these questions, we conducted a comprehensive empirical investigation, examining factors in the data that influence the emergence of alignment and its relationship with performance on a task, such as detecting sarcasm in a video.
Our findings show that alignment doesn’t always benefit performance; in fact, its impact depends on the dataset and the task at hand. This insight is important for practitioners, as it helps them determine when trying to align different data types might improve their model’s performance or when it could potentially be detrimental.

Link To Code: https://github.com/MeganTj/multimodal_alignment

Primary Area: Deep Learning->Other Representation Learning

Keywords: representation alignment, multimodal learning, vision-language models

Submission Number: 2947