Neural Fixed-Point Acceleration for Convex Optimization

Shobha Venkataraman, Brandon Amos

Keywords: convex optimization, meta-learning, acceleration

TL;DR: We improve the solution to fixed-point computations of convex optimization solvers with learning

Abstract:

Fixed-point iterations are at the heart of numerical computing and are often a computational bottleneck in real-time applications, which typically instead need a fast solution of moderate accuracy. Classical acceleration methods for fixed-point problems focus on designing algorithms with theoretical guarantees that apply to any fixed-point problem. We present neural fixed-point acceleration, a framework to automatically learn to accelerate convex fixed-point problems that are drawn from a distribution, using ideas from meta-learning and classical acceleration algorithms. We apply our framework to SCS, the state-of-the-art solver for convex cone programming, and design models and loss functions to overcome the challenges of learning over unrolled optimization and acceleration instabilities. Our work brings neural acceleration into any optimization problem expressible with CVXPY. This is relevant to AutoML as we (meta-)learn improvements to a convex optimization solver that replaces an acceleration component that is traditionally hand-crafted.

Upon acceptance, we will openly release the source code containing our batched and differentiable PyTorch implementation of SCS with neural acceleration and all of the supplementary files necessary to fully reproduce our results.

Ethics Statement: We do not see any ethical issues with our work as we do not study new applications. We focus on developing learning techniques to replace traditionally designed acceleration algorithms in fixed-point solvers in convex optimization problems. It has the potential to enable faster and more practical solutions to applications that can use these fixed-point solvers, but does not enable the solution of a new class of optimization problems.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2107.10254/code)