An Approach for Targeted Analysis of Topic Portrayals in Public Service Media Corpora
Track Selection: Track 1: Developing LLM-powered tools for positive outcomes
Keywords: Publice Service Media, Large Language Models, Metrics
TL;DR: We preposed a flexible approach for monitoring portrayals of topics in news stories using NLP tools and contextualized word embeddings.
Abstract: BBC News is beholden to public service values in providing it’s output. It is also exploring the use of recommendation and personalization technologies for news services. There is also an interest in using computational technologies for the monitoring of adherence to
public services. Despite these potential uses, there is a dearth of computational metrics for monitoring the vast text-based output of the BBC News service. Here we propose a method that could be utilized by the editorial team as a high level signal for issues in their output. The
method is intended for monitoring the portrayal of specific topics in a corpus of news stories. It utilizes NLP tools of coreference resolution and dependency parsing to identify words related to the topic and BERT Language Model contextualized word embeddings to assess
those words against a topic. We find that the method provides results consistent with human labeling when used on a benchmark dataset. We then perform a case study using a corpus of recent English language BBC News politics stories, where we test for associations in
the text that relate to known gender-based stereotypes and do not find evidence of those stereotypes in the corpus. The approach presented may be an effective tool to monitor reporting to identify instances of bias or stereotyping. We conclude with a discussion of potential limitations of the approach and planned future work to validate and improve the proposed method.
Submission Number: 8
Loading