Mitigating Covertly Unsafe Text within Natural Language Systems

Alex Mei, Anisha Kabir, Sharon Levy, Melanie Subbiah, Emily Allaway, John Norman Judge, Desmond U. Patton, Bruce Bimber, Kathleen McKeown, William Yang Wang

Abstract:

An increasingly prevalent problem for intelligent technologies is text safety, as uncontrolled systems may generate recommendations to their users that lead to injury or lifethreatening consequences. However, the degree of explicitness of a generated statement
that can cause physical harm varies. In this
paper, we distinguish types of text that can
lead to physical harm and establish one particularly underexplored category: covertly unsafe text. Then, we further break down this
category with respect to the system’s information and discuss solutions to mitigate the generation of text in each of these subcategories.
Ultimately, our work defines the problem of
covertly unsafe language that causes physical
harm and argues that this subtle yet dangerous
issue needs to be prioritized by stakeholders
and regulators. We highlight mitigation strategies to inspire future researchers to tackle this
challenging problem and help improve safety
within smart systems.