Eliminating accidental deviations to minimize generalization error and maximize replicability: Applications in connectomics and genomics

Eric W. Bridgeford, Shangsi Wang, Zeyi Wang, Ting Xu, R. Cameron Craddock, Jayanta Dey, Gregory Kiar, William R. Gray Roncal, Carlo Colantuoni, Christopher Douville, Stephanie Noble, Carey E. Priebe, Brian Caffo, Michael P. Milham, Xi-Nian Zuo, Joshua T. Vogelstein

Abstract: Author summary In recent decades, the size and complexity of data has grown exponentially. Unfortunately, the increased scale of modern datasets brings many new challenges. At present, we are in the midst of a replicability crisis, in which scientific discoveries fail to replicate to new datasets. Difficulties in the measurement procedure and measurement processing pipelines coupled with the influx of complex high-resolution measurements, we believe, are at the core of the replicability crisis. If measurements themselves are not replicable, what hope can we have that we will be able to use the measurements for replicable scientific findings? We introduce the “discriminability” statistic, which quantifies how discriminable measurements are from one another, without limitations on the structure of the underlying measurements. We prove that discriminable strategies tend to be strategies which provide better accuracy on downstream scientific questions. We demonstrate the utility of discriminability over competing approaches in this context on two disparate datasets from both neuroimaging and genomics. Together, we believe these results suggest the value of designing experimental protocols and analysis procedures which optimize the discriminability.