Revisiting Random Weight Perturbation for Efficiently Improving Generalization
arxiv(2024)
摘要
Improving the generalization ability of modern deep neural networks (DNNs) is
a fundamental challenge in machine learning. Two branches of methods have been
proposed to seek flat minima and improve generalization: one led by
sharpness-aware minimization (SAM) minimizes the worst-case neighborhood loss
through adversarial weight perturbation (AWP), and the other minimizes the
expected Bayes objective with random weight perturbation (RWP). While RWP
offers advantages in computation and is closely linked to AWP on a mathematical
basis, its empirical performance has consistently lagged behind that of AWP. In
this paper, we revisit the use of RWP for improving generalization and propose
improvements from two perspectives: i) the trade-off between generalization and
convergence and ii) the random perturbation generation. Through extensive
experimental evaluations, we demonstrate that our enhanced RWP methods achieve
greater efficiency in enhancing generalization, particularly in large-scale
problems, while also offering comparable or even superior performance to SAM.
The code is released at https://github.com/nblt/mARWP.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要