MuSeD: A Multimodal Spanish Dataset for Sexism Detection in Social Media Videos

Laura De Grazia, Pol Pastells, Mauro Vázquez Chas, Desmond Elliott, Danae Sanchez Villegas, Mireia Farrús, Mariona Taulé Delor

Keywords: sexism, multimodal, classification, social media, LLMs

TL;DR: We present MuSeD, a multimodal Spanish dataset for sexism detection in videos, with an innovative annotation framework for analyzing the contribution of textual and multimodal labels. We evaluate the performance of LLMs and multimodal LLMs on MuSeD.

Abstract:

Sexism is generally defined as prejudice and discrimination based on sex or gender, affecting every sector of society, from social institutions to relationships and individual behavior. Social media platforms amplify the impact of sexism by conveying discriminatory content not only through text but also across multiple modalities, highlighting the critical need for a multimodal approach to the analysis of sexism online. With the rise of social media platforms where users share short videos, sexism is increasingly spreading through video content. Automatically detecting sexism in videos is a challenging task, as it requires analyzing the combination of verbal, audio, and visual elements to identify sexist content. In this study, (1) we introduce MuSeD, a new Multimodal Spanish dataset for Sexism Detection consisting of ≈ 11 hours of videos extracted from TikTok and BitChute; (2) we propose an innovative annotation framework for
analyzing the contributions of textual, vocal, and visual modalities to the classification of content as either sexist or non-sexist; and (3) we evaluate a range of large language models (LLMs) and multimodal LLMs on the task of sexism detection. We find that visual information plays a key role in labeling sexist content for both humans and models. Models effectively detect explicit sexism; however, they struggle with implicit cases, such as stereotypes—instances where annotators also show low agreement. This highlights the inherent difficulty of the task, as identifying implicit sexism depends on the social and cultural context.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Flagged For Ethics Review: true

Ethics Comments:

Dataset contains samples with discrimination/stereotype/inequality/objectification in videos, especially sensitive samples from BitChute platform where annotators returned ~94% samples as sexist.

The proposed Spanish dataset for sexism detection may contribute to the emergence of sexual harassment-related public opinion and provide them with a channel.

Submission Number: 1020