Distributed robust statistical learning: Byzantine mirror descent

Dongsheng Ding, Xiaohan Wei, Mihailo R. Jovanovic

Abstract: We consider the distributed statistical learning problem in a high-dimensional adversarial scenario. At each iteration, m worker machines compute stochastic gradients and send them to a master machine. However, an α-fraction of m worker machines, called Byzantine machines, may act adversarially and send faulty gradients. To guard against faulty information sharing, we develop a distributed robust learning algorithm based on mirror descent. This algorithm is provably robust against Byzantine machines whenever α ∈ [0, 1/2). For smooth convex functions, we show that running the proposed algorithm for T iterations achieves a statistical error bound Õ(1/√mT + α/√T). This result holds for a large class of normed spaces and it matches the known statistical error bound for Byzantine stochastic gradient in the Euclidean space setting. A key feature of the algorithm is that the dimension dependence of the bound scales with the dual norm of the gradient; in particular, for probability simplex, we show that it depends logarithmically on the problem dimension d. Such a weak dependence is desirable in high-dimensional statistical learning and it has been known to hold for the classical mirror descent but it appears to be new for the Byzantine gradient scenario.