Compressing Gradient Optimizers via Count-Sketches

Ryan Spring, Anastasios Kyrillidis, Vijai Mohan, Anshumali Shrivastava

Abstract: Many popular first-order optimization methods accelerate the convergence rate of deep learning models. However, these algorithms require auxiliary variables, which cost additional memory proportion...