Clip21: Error Feedback for Gradient Clipping
Supplementary Material: pdf
Primary Area: optimization
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: clipping, error feedback, gradient-based methods, communication-efficient learning, optimization
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Motivated by the increasing importance of deep neural network training, we study distributed gradient methods with gradient clipping, i.e., clipping applied to the gradients computed from local information at the nodes. While gradient clipping enforces the convergence of gradient-based methods that minimize rapidly growing functions, it also induces bias which causes serious convergence issues specific to the distributed setting. Inspired by recent progress in the error-feedback literature which is focused on taming the bias/error introduced by communication compression operators such as Top-$k$ (Richtárik et al, 2021), and mathematical similarities between the clipping operator and contractive compression operators, we design Clip21 -- the first provably effective and practically useful error feedback mechanism for distributed methods with gradient clipping. We prove that our method converges at the same $\mathcal{O}({1}/{K})$ rate as distributed gradient descent in the smooth nonconvex regime, which improves the previous best $\mathcal{O}({1}/{\sqrt{K}})$ rate which was obtained under significantly stronger assumptions.
Our method converges significantly faster in practice than competing methods.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6746
Loading