An adaptive sampling augmented Lagrangian method for stochastic optimization with deterministic constraints

Raghu Bollapragada, Cem Karamanli, Brendan Keith, Boyan Lazarov, Socratis Petrides, Jingyi Wang

Abstract: The primary goal of this paper is to provide an efficient solution algorithm based on the augmented Lagrangian framework for optimization problems with a stochastic objective function and deterministic constraints. Our main contribution is combining the augmented Lagrangian framework with adaptive sampling, resulting in an efficient optimization methodology validated with practical examples. To achieve the presented efficiency, we consider inexact solutions for the augmented Lagrangian subproblems, and through an adaptive sampling mechanism, we control the variance in the gradient estimates. Furthermore, we analyze the theoretical performance of the proposed scheme by showing equivalence to a gradient descent algorithm on a Moreau envelope function, and we prove sublinear convergence for convex objectives and linear convergence for strongly convex objectives with affine equality constraints. The worst-case sample complexity of the resulting algorithm, for an arbitrary choice of penalty parameter in the augmented Lagrangian function, is O(ϵ−3−δ)<math><mi mathvariant="script" is="true">O</mi><mo stretchy="false" is="true">(</mo><msup is="true"><mrow is="true"><mi is="true">ϵ</mi></mrow><mrow is="true"><mo linebreak="badbreak" linebreakstyle="after" is="true">−</mo><mn is="true">3</mn><mo linebreak="badbreak" linebreakstyle="after" is="true">−</mo><mi is="true">δ</mi></mrow></msup><mo stretchy="false" is="true">)</mo></math>, where ϵ>0<math><mi is="true">ϵ</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">&gt;</mo><mn is="true">0</mn></math> is the expected error of the solution and δ>0<math><mi is="true">δ</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">&gt;</mo><mn is="true">0</mn></math> is a user-defined parameter. If the penalty parameter is chosen to be O(ϵ−1)<math><mi mathvariant="script" is="true">O</mi><mo stretchy="false" is="true">(</mo><msup is="true"><mrow is="true"><mi is="true">ϵ</mi></mrow><mrow is="true"><mo linebreak="badbreak" linebreakstyle="after" is="true">−</mo><mn is="true">1</mn></mrow></msup><mo stretchy="false" is="true">)</mo></math>, we demonstrate that the result can be improved to O(ϵ−2)<math><mi mathvariant="script" is="true">O</mi><mo stretchy="false" is="true">(</mo><msup is="true"><mrow is="true"><mi is="true">ϵ</mi></mrow><mrow is="true"><mo linebreak="badbreak" linebreakstyle="after" is="true">−</mo><mn is="true">2</mn></mrow></msup><mo stretchy="false" is="true">)</mo></math>, which is competitive with the other methods employed in the literature. Moreover, if the objective function is strongly convex with affine equality constraints, we obtain O(ϵ−1log⁡(1/ϵ))<math><mi mathvariant="script" is="true">O</mi><mo stretchy="false" is="true">(</mo><msup is="true"><mrow is="true"><mi is="true">ϵ</mi></mrow><mrow is="true"><mo linebreak="badbreak" linebreakstyle="after" is="true">−</mo><mn is="true">1</mn></mrow></msup><mi mathvariant="normal" is="true">log</mi><mo is="true">⁡</mo><mo stretchy="false" is="true">(</mo><mn is="true">1</mn><mo stretchy="false" is="true">/</mo><mi is="true">ϵ</mi><mo stretchy="false" is="true">)</mo><mo stretchy="false" is="true">)</mo></math> complexity. Finally, we empirically verify the performance of our adaptive sampling augmented Lagrangian framework in machine learning optimization and engineering design problems, including topology optimization of a heat sink with environmental uncertainty.