Differentially Private Latent Diffusion Models
arxiv(2023)
摘要
Diffusion models (DMs) are widely used for generating high-quality
high-dimensional images in a non-differentially private manner. To address this
challenge, recent papers suggest pre-training DMs with public data, then
fine-tuning them with private data using DP-SGD for a relatively short period.
In this paper, we further improve the current state of DMs with DP by adopting
the Latent Diffusion Models (LDMs). LDMs are equipped with powerful pre-trained
autoencoders that map the high-dimensional pixels into lower-dimensional latent
representations, in which DMs are trained, yielding a more efficient and fast
training of DMs. In our algorithm, DP-LDMs, rather than fine-tuning the entire
DMs, we fine-tune only the attention modules of LDMs at varying layers with
privacy-sensitive data, reducing the number of trainable parameters by roughly
90
The smaller parameter space to fine-tune with DP-SGD helps our algorithm to
achieve new state-of-the-art results in several public-private benchmark data
pairs.Our approach also allows us to generate more realistic, high-dimensional
images (256x256) and those conditioned on text prompts with differential
privacy, which have not been attempted before us, to the best of our knowledge.
Our approach provides a promising direction for training more powerful, yet
training-efficient differentially private DMs, producing high-quality
high-dimensional DP images.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要