Learning Approximate Invariance Requires Far Fewer Data

Efficient learning, that is, learning with small datasets is difficult for current deep learning models. Invariance has been conjectured to be the key for its generalization potential. One of the most used procedure to learn invariant models is data augmentation (DA). Data augmentation can be performed offline by augmenting the data before any training, or it can be performed online during training. However, applying those technique won’t yield better generalization gains every time. We frame this problem as the stability of generalization gains made by invariance inducing techniques. In this study we introduced a new algorithm to train an approximate invariant priors before posterior training of Bayesian Neural Network (BNN). Furthermore, we compared the generalization stability of our invariance inducing algorithm with online DA and offline DA on MNIST and Fashion MNIST with three perturbation processes: rotation, noise, and rotation+noise. Results showed that learning approximate invariant priors requires less exposure to the perturbation process, but it leads BNN to more stable generalization gains during posterior training. Finally, we also show that invariance inducing techniques enhance uncertainty in Bayesian Neural Networks.