Towards Multimodal Campaign Detection: Including Image Information in Stream Clustering to Detect Social Media Campaigns

Lucas Stampe, Janina Pohl, Christian Grimme

Abstract: This work explores the potential to include visual information from images in social media campaign recognition. The diverse content shared on social media platforms, including text, photos, videos, and links, necessitates a multimodal analysis approach. With the emergence of Large Language Models (LLMs), there is now an opportunity to convert image content into textual descriptions, enabling the incorporation of previously text-based methods into a multimodal analysis. We evaluate this approach by conducting a parameter study to assess the resulting differences in image captions and a case study to examine the contribution of textualized image information to campaign recognition. The results indicate that, using image captions separate from or alongside tweet texts, connections between campaigns can be identified, and new campaigns detected.

External IDs:doi:10.1007/978-3-031-47896-3_11