Pictures are Worth a Thousand Frames: The Impact of Images on the Discovery of Frames of Communication from Multimodal Social Media
Abstract: Frames of Communication (FoCs) are evoked in multiple Social Media Postings (SMPs) that contain not only text, but also images. In this paper we introduce DA-FoC$^{MM}$, a new method capable of uncovering and articulating the FoCs evoked in SMPs while also pinpointing whether the FoC is evoked in the SMP text, image, or both. The DA-FoC$^{MM}$ method successfully discovers FoCs from multimodal SMPs discussing two different controversial topics, namely COVID-19 vaccines and immigration, by using several constrained prompting approaches that determine the combination of counterfactual reasoning with Chain-of-Thought (CoT) reasoning performed by a Language Multimodal Model (LMM). In addition, we show that DA-FoC$^{MM}$ enables the discovery of FoCs from multimodal SMPs across two platforms: Twitter / X and Instagram. Evaluations produced promising results, showing that 90\%-91\% of the FoCs identified by communication experts on the same collections of SMPs were also discovered by the method presented in this paper. We also found that 39\% of FoCs would not have been discovered if the images from SMPs had been ignored.
Paper Type: Long
Research Area: Computational Social Science and Cultural Analytics
Research Area Keywords: frame detection and analysis, quantitative analyses of news and/or social media, NLP tools for social analysis
Contribution Types: Data resources, Data analysis, Position papers
Languages Studied: English
Submission Number: 1209
Loading