Optimization with Access to Auxiliary Information

Published: 03 Apr 2024, Last Modified: 03 Apr 2024Accepted by TMLREveryoneRevisionsBibTeX
Abstract: We investigate the fundamental optimization question of minimizing a \emph{target} function $f(x)$, whose gradients are expensive to compute or have limited availability, given access to some \emph{auxiliary} side function $h(x)$ whose gradients are cheap or more available. This formulation captures many settings of practical relevance, such as i) re-using batches in SGD, ii) transfer learning, iii) federated learning, iv) training with compressed models/dropout, etcetera. We propose two generic new algorithms that apply in all these settings; we also prove that we can benefit from this framework under the Hessian similarity assumption between the target and side information. A benefit is obtained when this similarity measure is small; we also show a potential benefit from stochasticity when the auxiliary noise is correlated with that of the target function.
Submission Length: Regular submission (no more than 12 pages of main content)
Code: https://github.com/elmahdichayti/OptAuxInf
Supplementary Material: pdf
Assigned Action Editor: ~Lorenzo_Orecchia1
Submission Number: 1599