Robust Training in High Dimensions via Block Coordinate Geometric Median Descent

Anish Acharya, Abolfazl Hashemi, Prateek Jain, sujay sanghavi, Inderjit S Dhillon, ufuk topcu

Keywords: robust, optimization, efficient, geometric median, median, breakdown point

Abstract:

Geometric median (GM) is a classical method in statistics for achieving a robust estimation of the uncorrupted data; under gross corruption, it achieves the optimal breakdown point of 0.5. However, its computational complexity makes it infeasible for robustifying stochastic gradient descent (SGD) for high-dimensional optimization problems. In this paper, we show that by applying GM to only a judiciously chosen block of coordinates at a time and using a memory mechanism, one can retain the breakdown point of 0.5 for smooth non-convex problems, with non-asymptotic convergence rates comparable to the SGD with GM.

We validate both the runtime and the robustness of our approach empirically on three neural network settings including ResNet-18 on CIFAR-10 and MLP / LeNet on Fashion-MNIST.

One-sentence Summary: We propose BGMD an efficient robust optimization method that achieves optimal breakdown point of 1/2.