SDEs for Minimax Optimization
CoRR(2024)
Abstract
Minimax optimization problems have attracted a lot of attention over the past
few years, with applications ranging from economics to machine learning. While
advanced optimization methods exist for such problems, characterizing their
dynamics in stochastic scenarios remains notably challenging. In this paper, we
pioneer the use of stochastic differential equations (SDEs) to analyze and
compare Minimax optimizers. Our SDE models for Stochastic Gradient
Descent-Ascent, Stochastic Extragradient, and Stochastic Hamiltonian Gradient
Descent are provable approximations of their algorithmic counterparts, clearly
showcasing the interplay between hyperparameters, implicit regularization, and
implicit curvature-induced noise. This perspective also allows for a unified
and simplified analysis strategy based on the principles of Itô calculus.
Finally, our approach facilitates the derivation of convergence conditions and
closed-form solutions for the dynamics in simplified settings, unveiling
further insights into the behavior of different optimizers.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined