Accuracy and Fairness for Web-Based Content Analysis under Temporal Shifts and Delayed Labeling

Abdulaziz A. Almuzaini, David M. Pennock, Vivek K. Singh

Abstract: Web-based content analysis tasks, such as labeling toxicity, misinformation, or spam often rely on machine learning models to achieve cost and scale efficiencies. As these models impact real human lives, ensuring accuracy and fairness of such models is critical. However, maintaining the performance of these models over time can be challenging due to the temporal shifts in the application context and the sub-populations represented. Furthermore, there is often a delay in obtaining human expert labels for the raw data, which hinders the timely adaptation and safe deployment of the models. To overcome these challenges, we propose a novel approach that anticipates future distributions of data, especially in settings where unlabeled data becomes available earlier than the labels to estimate the future distribution of labels per sub-population and adapt the model preemptively. We evaluate our approach using multiple temporally-shifting datasets and consider bias based on racial, political, and demographic identities. We find that the proposed approach yields promising performance with respect to both accuracy and fairness. Our paper contributes to the web science literature by proposing a novel method for enhancing the quality and equity of web-based content analysis using machine learning. Experimental code and datasets are publicly available at https://github.com/Behavioral-Informatics-Lab/FAIRCAST.