Language Model Accidents Waiting to Happen?

Anonymous

Abstract: Artificial Intelligence (AI) is at a crucial point in its development: stable enough to be used in production systems, but not yet as pervasive in our lives as it will soon become. However, even today's AI systems are already highly complex and unpredictable. As they become more ubiquitous, different algorithms will interact with each other directly leading to tightly coupled systems whose capacity to cause harm we will be unable to predict. In his book Normal Accidents, the sociologist Charles Perrow proposed a framework to analyze technologies and the risks they entail in an intelligible way. He suggested classifying them according to their complexity and the interdependence of their components (known as ‘coupling’ in computer science). In this essay, we use Perrow’s framework on AI to assess its potential risks. We start by looking at conversational AI and later extend it to more general language technologies. Drawing on our background in natural language processing and going beyond what other experts have proposed, we argue for a more nuanced approach to understanding risks and safety in AI.