Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation

Anonymous

Paper Link: https://openreview.net/forum?id=4FHSEhaiCy

Paper Type: Short paper (up to four pages of content + unlimited references and appendices)

Abstract: Detecting out-of-context media, such as "miscaptioned" images on Twitter, is a relevant problem, especially in domains of high public significance. In this work we aim to develop defenses against such misinformation for the topics of Climate Change, COVID-19, and Military Vehicles. We first present a large-scale multimodal dataset with over 884k tweets relevant to these topics. Next, we propose a detection method, based on the state-of-the-art CLIP model, that leverages automatically generated hard image-text mismatches. While this approach works well on our automatically constructed out-of-context tweets, we aim to validate its usefulness on data representative of the real world. Thus, we test it on a set of human-generated fakes, created by mimicking in-the-wild misinformation. We achieve an 11% detection improvement in a high precision regime over a strong baseline. Finally, we share insights about our best model design and analyze the challenges of this emerging threat.

Presentation Mode: This paper will be presented virtually

Virtual Presentation Timezone: UTC-8

Copyright Consent Signature (type Name Or NA If Not Transferrable): Grace Luo

Copyright Consent Name And Address: University of California, Berkeley, 2121 Berkeley Way, Berkeley, CA 94704