On the interplay between data structure and loss function in classification problems

Stéphane d'Ascoli, Marylou Gabrié, Levent Sagun, Giulio Biroli

Keywords: loss function, generalization, overparametrization, double descent, random features

TL;DR: We analyze the interplay between data structure and loss function in random feature classification problems

Abstract:

One of the central features of modern machine learning models, including deep neural networks, is their generalization ability on structured data in the over-parametrized regime.
In this work, we consider an analytically solvable setup to investigate how properties of data impact learning in classification problems, and compare the results obtained for quadratic loss and logistic loss.
Using methods from statistical physics, we obtain a precise asymptotic expression for the train and test errors of random feature models trained on a simple model of structured data. The input covariance is built from independent blocks allowing us to tune the saliency of low-dimensional structures and their alignment with respect to the target function.
Our results show in particular that in the over-parametrized regime, the impact of data structure on both train and test error curves is greater for logistic loss than for mean-squared loss: the easier the task, the wider the gap in performance between the two losses at the advantage of the logistic.
Numerical experiments on MNIST and CIFAR10 confirm our insights.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/sdascoli/data-structure