Error estimation based on variance analysis of k-fold cross-validation

Gaoxia Jiang, Wenjian Wang

Abstract: Highlights•When the numbers of samples and folds are both large enough, we proved that CV variance and its accuracy have the quadratic relationship.•The relationships between CV variance and its factors have been derived, allowing to predict which variance is less before applying k-fold CV.•Theoretical explanations have been given for some empirical evidences of Rodriguez and Kohavi from the respect of variance analysis.•The proposed normalized variance has significant correlation with the error and is unrelated to k so that it can serve as a stable error measurement.