Why, when, and how to (or not to) preregister a simulation study

Björn S. Siepe, František Bartoš, Anne-Laure Boulesteix, Daniel W. Heck, Aaron Peikert, Alexandra Sarafoglou, Samuel Pawel

Abstract: Simulation studies play a central role in methodological research by providing evidence aboutthe empirical performance of statistical methods. At the same time, they involve substan-tial researcher degrees of freedom in the choice of data-generating mechanisms, methods, performance measures, and reporting decisions, which can undermine the credibility of theirresults. Preregistration has been proposed as a tool to increase transparency and reducesuch flexibility, yet its role in simulation studies remains contested. In this paper, we dis-cuss why researchers should or should not preregister their simulation studies, elaborating onthe theoretical and pragmatic benefits as well as disadvantages of preregistration. Drawing on philosophical accounts, we argue that preregistration primarily reduces epistemic uncer-tainty about which simulation conditions were considered and why, which enables honestevaluation of whether results support the conclusions drawn. We argue that the resultingbenefits depend critically on the aims of a study and its position within the methodologicalresearch cycle. Late-phase studies that investigate mature methods tend to benefit more,while early-phase studies investigating newly proposed methods generally benefit less frompreregistration. We further discuss practical challenges specific to simulation studies, includ-ing the timing of preregistration, the role of preliminary simulations, and safeguards againstpost hoc registration. Finally, we outline concrete guidance on how to preregister simulationstudies and discuss Registered Reports as a particularly promising format. Overall, we ar-gue for a nuanced, purpose-driven use of preregistration that supports rather than hindersmethodological progress.