Keywords: hate speech detection
TL;DR: A reproducibility report for a paper proposing a novel framework for hate speech detection relying on sentiment features
Abstract: This report summarises our efforts to reproduce the results presented in the ACL2021 paper Hate Speech Detection based on Sentiment Knowledge Sharing by Zhou et al. (2021), as part of the ML Reproducibility Challenge 2021. Scope of Reproducibility The main goal of this reproducibility attempt is to confirm the effectiveness of the hate speech detection framework proposed by Zhou et al. (2021). In particular, our efforts are directed at validating their main claim that sentiment knowledge sharing in a multi-task learning setup improves the performance of the model in predicting hate speech. Besides reproducing their main results, we perform repeated experiments to assess the variability of the scores and perform a hyperparameter search. Methodology The authors provide a code-base which is available at https://github.com/1783696285/SKS. We reuse the available code, modifying it where necessary and integrating it with a few additional scripts for statistics computation and data preparation. Our code, data and results are available at https://anonymous.4open.science/r/repro-SKS-A. Results Our findings diverge substantially from the results reported in the original paper. In particular, in our reproduction experiments, including sentiment features hurts the performance of the model in the hate speech detection task (approximately 0.5 to 2.0 F1-score). What was easy The paper provides some broad indications with respect to the training details and the code-base is publicly available. Similarly, the data-sets are also freely available and the authors provide links to them in their repository. What was difficult The code-base is rather convoluted. Following the instructions included in the authors’ repository resulted in a number of exceptions caused by formatting issues, missing code snippets and hard-coded values. Additionally, the lack of a clear and comprehensive documentation contributed to an arduous code review and reproducibility effort. Communication with original authors We managed to reach one of the authors and exchange a few messages over GitHub. However, despite multiple attempts, we did not manage to reach the authors per email and get an answer to our questions concerning some aspects of the implementation.
Paper Url: https://aclanthology.org/2021.acl-long.556.pdf
Paper Venue: ACL 2021
Supplementary Material: zip