An outer-inner linearization method for non-convex and nondifferentiable composite regularization problems

Minh Pham, Xiaodong Lin, Andrzej Ruszczynski, Yu Du

Abstract: We propose a new outer–inner linearization method for non-convex statistical learning problems involving non-convex structural penalties and non-convex loss. Many important statistical problems fall in this category, including the robust M-estimators, generalized linear models, and different types of structured learning problems. Our method exploits the fact that many such loss and penalty functions can be represented as compositions of smooth concave functions and nonsmooth convex functions. It linearizes the outer concave functions and solves the resulting convex, but still nonsmooth, subproblems by a special alternating linearization method. We provide proof of convergence to a stationary point of the method under mild conditions. Finally, numerical examples involving non-convex structural penalties and non-convex loss functions demonstrate the efficacy and accuracy of the method.