Faster Margin Maximization Rates for Generic and Adversarially Robust Optimization Methods
arxiv(2023)
摘要
First-order optimization methods tend to inherently favor certain solutions
over others when minimizing an underdetermined training objective that has
multiple global optima. This phenomenon, known as implicit bias, plays a
critical role in understanding the generalization capabilities of optimization
algorithms. Recent research has revealed that in separable binary
classification tasks gradient-descent-based methods exhibit an implicit bias
for the ℓ_2-maximal margin classifier. Similarly, generic optimization
methods, such as mirror descent and steepest descent, have been shown to
converge to maximal margin classifiers defined by alternative geometries. While
gradient-descent-based algorithms provably achieve fast implicit bias rates,
corresponding rates in the literature for generic optimization methods are
relatively slow. To address this limitation, we present a series of
state-of-the-art implicit bias rates for mirror descent and steepest descent
algorithms. Our primary technique involves transforming a generic optimization
algorithm into an online optimization dynamic that solves a regularized
bilinear game, providing a unified framework for analyzing the implicit bias of
various optimization methods. Our accelerated rates are derived by leveraging
the regret bounds of online learning algorithms within this game framework. We
then show the flexibility of this framework by analyzing the implicit bias in
adversarial training, and again obtain significantly improved convergence
rates.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要