Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold

Anonymous

Abstract: The first NLP experiment many researchers performed in their careers likely involved training a standard architecture on labeled English data and optimizing for accuracy, without accounting for other dimensions such as fairness, interpretability, or computational efficiency. We show through surveys that this is indeed the case and refer to it as the square one experimental setup. NLP research often goes beyond the square one setup, e.g, focusing not only on accuracy, but also on fairness or interpretability, but typically along a single dimension. Most work focused on multilinguality, for example, considers only accuracy; most work on fairness or interpretability considers only English; and so on. We show this through manual classification of recent NLP research papers and ACL Test-of-Time award recipients. Such one-dimensionality of most research means we are only exploring a fraction of the NLP research search space. We provide historical and recent examples of how the square one bias has led researchers to draw false conclusions or make unwise choices, point to promising yet unexplored directions on the research manifold, and make practical recommendations to enable more multi-dimensional research.