(Distributed) Fractional Gradient Descent with Matrix Stepsizes for Non-Convex Optimisation

Alokendu Mazumder, Keshav Vyas, Punit Rathore

Keywords: compressed fractional gradient descent, matrix stepsize, distributed setting

Abstract: Fractional derivatives generalise integer-order derivatives, making them relevant for studying their convergence in descent-based optimisation algorithms. However, existing convergence analysis of fractional gradient descent is limited in both methods and settings. This paper bridges these gaps by establishing convergence guarantees for fractional gradient descent on a broader class of non-convex functions, known as matrix-smooth functions. We leverage the matrix smoothness properties of the function to prove convergence and accelerate the fractional gradient descent iterates. We propose two novel stochastic fractional descent algorithms, named Compressed Fractional Gradient Descent (CFGD), incorporating a matrix-valued stepsize to minimise matrix-smooth non-convex objectives. Our theoretical analysis covers both single-node and distributed settings and shows that matrix stepsizes better capture the structure of the objective, leading to faster convergence than scalar stepsizes. Additionally, we highlight the importance of matrix stepsizes to leverage model structure effectively. To the best of our knowledge, this is the first work to introduce fractional gradient descent in a federated/distributed setting.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 7