A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings

Fitsum Gaim, Hoyun Song, Huije Lee, Changgeon Ko, Eui Jun Hwang, Jong C. Park

Keywords: Content Moderation, Low-resource Settings, Low-resource Languages, African Languages, Benchmark, Dataset

TL;DR: We present the first benchmark dataset for abusive language detection in Tigrinya, demonstrating how multi-task learning can address critical content moderation gaps in underserved languages.

Abstract: Content moderation research has recently made significant advances, but remains limited in serving the majority of the world's languages due to the lack of resources, leaving millions of vulnerable users to online hostility. This work presents a large-scale human-annotated multi-task benchmark dataset for abusive language detection in Tigrinya social media with joint annotations for three tasks: abusiveness, sentiment, and topic classification. The dataset comprises 13,717 YouTube comments annotated by nine native speakers, collected from 7,373 videos with a total of over 1.2 billion views across 51 channels. We developed an iterative term clustering approach for effective data selection. Recognizing that around 64% of Tigrinya social media content uses Romanized transliterations rather than native Ge'ez script, our dataset accommodates both writing systems to reflect actual language use. We establish strong baselines across the tasks in the benchmark, while leaving significant challenges for future contributions. Our experiments demonstrate that small fine-tuned models outperform prompted frontier large language models (LLMs) in the low-resource setting, achieving 86.67% F1 in abusiveness detection (7+ points over best LLM), and maintain stronger performance in all other tasks. The benchmark is made public to promote research on online safety.

Dataset URL: https://huggingface.co/datasets/fgaim/tigrinya-abusive-language-detection

Code URL: https://github.com/fgaim/TiALD

Primary Area: AL/ML Datasets & Benchmarks for social sciences (e.g. climate, health, life sciences, physics, social sciences)

Submission Number: 2347