We need to talk about random seeds

Anonymous

Abstract: Modern neural network libraries all take as a hyperparameter a random seed, typically used to determine the initial state of the model parameters. In this position piece, I argue that there are some appropriate uses for random seeds: as part of the hyperparameter search to select a good model, creating an ensemble of several variants of a model, or measuring the sensitivity of the training algorithm to the random seed hyperparameter. I argue against some inappropriate uses for random seeds: using a fixed random seed for "replicability" and creating score distributions for performance comparison. I review 85 recent publications from the ACL Anthology and find that more than 50% are using random seeds inappropriately.