A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization
arxiv(2022)
摘要
An open problem in differentially private deep learning is hyperparameter
optimization (HPO). DP-SGD introduces new hyperparameters and complicates
existing ones, forcing researchers to painstakingly tune hyperparameters with
hundreds of trials, which in turn makes it impossible to account for the
privacy cost of HPO without destroying the utility. We propose an adaptive HPO
method that uses cheap trials (in terms of privacy cost and runtime) to
estimate optimal hyperparameters and scales them up. We obtain state-of-the-art
performance on 22 benchmark tasks, across computer vision and natural language
processing, across pretraining and finetuning, across architectures and a wide
range of ε∈ [0.01,8.0], all while accounting for the privacy
cost of HPO.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要