Transition of Model Performance in Dependence of the Amount of Data Corruption with Respect to Network Sizes

Thomas Seidler, Markus Abel

Abstract: An important question for machine learning model concerns the achievable quality or performance of a model with respect to given data. In other words, we want to answer the question how robust a model is with respect to perturbation of the data. From statistical mechanics, a standard way to ”corrupt” input data is a study that uses additive noise to perturb data. This, in turn, corresponds to typical situations in processing data from any sensor as measurement noise. Larger models will often perform better, because they are able to capture more variance of the data. However, if the information content cannot be retrieved due to too large data corruptions a large network cannot compensate noise effects and no performance is gained by scaling the network. Here we study systematically the said effect, we add diffusive noise of increasing strength on a logarithmic scale to some well-known datasets for classification. As a result, we observe a sharp transition in training and test accurac