Multimodal Sentiment Analysis: Recognizing Sentiment in Memes

Georgi Vankov, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov

Abstract: The usage of memes and other visual material coupled with text on social media has been on the rise recently. Recognizing that visual signals are consumed quickly and can trigger emotional responses. It has become essential to discern the sentiment of such content, as it could significantly influence social media users. The paper focuses on the sentiment of memes on popular social networking platforms such as Instagram, Reddit, Facebook, and Tumblr. Our goal is to understand how these memes affect people in a positive, negative, or neutral way. We create a balanced dataset of 5,592 memes using distant supervision, i.e., automatically assigning sentiment labels based on different social media attributes, e.g., hashtags. We verify the accuracy of these labels by manually checking a random subset of the data. We conduct unimodal and multimodal experiments to explore how different cues contribute to identifying sentiment. Our results show that multimodal approaches, combining images and text, effectively identify the emotions in memes. We further experiment with novel closed and open-source LLMs, and we show that they outperform traditional multimodal approaches. The dataset is released publicly.