Realmvp: A Change Of Variables Method For Rectangular Matrix-Vector Products

Rectangular matrix-vector products (MVPs) are used extensively throughout machine learning and are fundamental to neural networks such as multi-layer perceptrons. However, rectangular MVPs are notably missing not used as normalizing flow transforms. This paper identifies this methodological gap and plugs it with a tall and wide MVP change of variables formula. Our theory builds up to a scalable algorithm that envelops existing dimensionality increasing flow methods such as augmented flows (Huang et al., 2020). We show that tall MVPs are closely related to the stochastic inverse of wide MVPs and empirically demonstrate that they improve density estimation over existing dimension changing methods.