Enhancing Hate Speech Detection with Large Language Model-Based Dataset Re-Labeling

Anonymous

TL;DR: The new labels generated by an LLM and Chain-of-Thought prompts significantly enhance hate speech model performance on existing datasets.

Abstract: While large language models have recently gained a surge of interest for their remarkable results, they frequently generate toxic expressions including profanity, offensive language, hate speech, etc. Among them, hate speech is one of the challenging categories because its subcategories are not clearly defined and an unbiased large dataset generation is yet challenging. Upon a rigorous definition of hate speech, we present a new way of labeling hate speech data using LLM with a prompt of Chain-of-Thought. We have applied this approach to re-label 5 widely used training datasets and evaluated them with 4 test sets. In 17 out of 20 cases, we observe an improvement in performance, resulting in an overall 18% improvement. Additionally, for the test sets, we utilize LLM for relabeling, followed by human validation. Upon performance evaluation, we find improvement in 19 out of 20 cases, resulting in an overall 25% performance enhancement.

Paper Type: short

Research Area: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Contribution Types: NLP engineering experiment, Data resources

Languages Studied: English