Noise Space Optimization for GANs

Traditional Generative Adversarial Networks (GANs) sample from a continuous stochastic noise distribution, and then treating that sample as a constant, optimize the parameters of a generator network on the average loss over the sample. In other words, the generator must take samples from a continuous input distribution, and on average, match a non-continuous target distribution (the training data). The result is that, as the generator transitions from one realistic image to another, some regions of the noise space correspond to relatively high quality images and other regions correspond to relatively low quality images. To avoid these relatively low quality areas, and allow the generator to optimize the shape of its noise distribution in a data-aware way, we present a novel sampling procedure: alternating (a) the traditional approach of optimizing the generator while holding the noise space sample constant, but then also (b) moving points in the noise space to improve the loss, using the direction of the gradient, while holding the generator parameters constant. We demonstrate that this procedure, which we call noise space optimization, improves the overall quality of samples obtained from an identical model without it on a wide array of canonical datasets and training paradigms.