Error-Feedback Meets Stochastic Approximation with Two Time Scales

Pengfei Wang, Nenggan Zheng

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: stochastic approximation, error-feedback, two time scales, structured perturbations

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Two-time-scale stochastic approximation is a recursive algorithm for solving a system of two equations. The method has found broad applications in many areas including machine learning and reinforcement learning. Recent works have revealed that single-time-scale stochastic approximation (especially its variant stochastic gradient descent in optimization) is robust to structured perturbations such as compression, local updates, and delays, but it is not well-understood in the two-time-scale case. Almost nothing is known about the analogous question: Is two-time-scale stochastic approximation also robust to similar structured perturbations? In this paper, we study error-feedback-based two-time-scale stochastic approximation. We propose a unified theory of two-time-scale stochastic approximation based on error-feedback to analyze the impact of different forms of structured perturbations. We show that two-time-scale stochastic approximation is robust to structured perturbations. In particular, two-time-scale stochastic approximation with different forms of structured perturbations exhibits the same non-asymptotic theoretical guarantees as its single-time-scale counterpart without structured perturbations. We further show that the convergence rate in all cases consists of two terms, where only the higher-order term is affected by structured perturbations. This is especially important for distributed parallel implementations of two-time-scale stochastic approximation algorithms.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4606