Reproducibility report: Hate Speech Detection based on Sentiment Knowledge Sharing

Matteo Brivio, Cagri Coltekin

Keywords: hate speech detection

TL;DR: A reproducibility report for a paper proposing a novel framework for hate speech detection relying on sentiment features

Abstract:

This report summarises our efforts to reproduce the results presented in the ACL2021 paper Hate Speech Detection
based on Sentiment Knowledge Sharing by Zhou et al. (2021), as part of the ML Reproducibility Challenge 2021.

Scope of Reproducibility
The main goal of this reproducibility attempt is to confirm the effectiveness of the hate speech detection framework
proposed by Zhou et al. (2021). In particular, our efforts are directed at validating their main claim that sentiment
knowledge sharing in a multi-task learning setup improves the performance of the model in predicting hate speech.
Besides reproducing their main results, we perform repeated experiments to assess the variability of the scores and
perform a hyperparameter search.

Methodology
The authors provide a code-base which is available at https://github.com/1783696285/SKS. We reuse the
available code, modifying it where necessary and integrating it with a few additional scripts for statistics computation and
data preparation. Our code, data and results are available at https://anonymous.4open.science/r/repro-SKS-A.

Results
Our findings diverge substantially from the results reported in the original paper. In particular, in our reproduction
experiments, including sentiment features hurts the performance of the model in the hate speech detection task
(approximately 0.5 to 2.0 F1-score).

What was easy
The paper provides some broad indications with respect to the training details and the code-base is publicly available.
Similarly, the data-sets are also freely available and the authors provide links to them in their repository.

What was difficult
The code-base is rather convoluted. Following the instructions included in the authors’ repository resulted in a number
of exceptions caused by formatting issues, missing code snippets and hard-coded values. Additionally, the lack of a
clear and comprehensive documentation contributed to an arduous code review and reproducibility effort.

Communication with original authors
We managed to reach one of the authors and exchange a few messages over GitHub. However, despite multiple attempts,
we did not manage to reach the authors per email and get an answer to our questions concerning some aspects of the
implementation.

Paper Url: https://aclanthology.org/2021.acl-long.556.pdf

Paper Venue: ACL 2021