A Two-Sided Discussion of Preregistration of NLP Research

Anders Søgaard, Daniel Hershcovich, Miryam de Lhoneux

Abstract: Van Miltenburg et al. (2021) suggest NLP re- search should adopt preregistration to prevent fishing expeditions and to promote publication of negative results. At face value, this is a very reasonable suggestion, seemingly solving many methodological problems with NLP re- search. We discuss pros and cons—some old, some new: a) Preregistration is challenged by the practice of retrieving hypotheses after the results are known; b) preregistration may bias NLP toward confirmatory research; c) prereg- istration must allow for reclassification of re- search as exploratory; d) preregistration may in- crease publication bias; e) preregistration may increase flag-planting; f) preregistration may increase p-hacking; and finally, g) preregistra- tion may make us less risk tolerant. We cast our discussion as a dialogue, presenting both sides of the debate.