Projection-Free Algorithms for Minimax Problems

Khanh-Hung Giang-Tran, Soroosh Shafiee, Nam Ho-Nguyen

Abstract: This paper addresses constrained smooth saddle-point problems in settings where projection onto the feasible sets is computationally expensive. We bridge the gap between projection-based and projection-free optimization by introducing a unified dual dynamic smoothing framework that enables the design of efficient single-loop algorithms. Within this framework, we establish convergence results for nonconvex-concave and nonconvex-strongly concave settings. Furthermore, we show that this framework is naturally applicable to convex-concave problems, providing a unified analysis across varying payoff structures. We propose and analyze three algorithmic variants based on the application of a linear minimization oracle over the minimization variable, the maximization variable, or both. Notably, our analysis yields anytime convergence guarantees without requiring a pre-specified iteration horizon. These results significantly narrow the performance gap between projection-free and projection-based methods for minimax optimization.

Lay Summary:

How much time do learning algorithms have to waste on math bottlenecks just to stay within the rules of a problem? We wanted to answer this question by introducing a single-loop optimization framework that lets machine learning models learn faster and more flexibly without expensive computational baggage.

Our paper presents a way to bypass the costly projection step, which is the mathematical process traditionally used to keep variables inside their boundaries. We replace it with cheaper, straight line shortcuts called Linear Minimization Oracles, or LMOs. While projection free methods exist , adapting them to two player minimax games where players have opposite goals has historically required slow, complex nested loops. We solve this by using a dynamic smoothing technique that allows both players to update their strategies simultaneously in a single, efficient loop. Crucially, our algorithms maintain convergence guarantees continuously without needing a pre set timer or fixed iteration budget.

Our findings have implications for how we train complex models under data uncertainty or distribution shifts. We prove that swapping out tedious boundary checks for our straight line framework can nearly double the training iterations completed in the same timeframe without sacrificing performance.

Primary Area: Optimization->Non-Convex

Keywords: Minimax optimization, Projection-free methods, Single-loop algorithms, Nonconvex-concave optimization, Dual dynamic smoothing, Anytime convergence.

Submission Number: 13172