Adversarial examples in random neural networks with general activations

A substantial body of empirical work documents the lack of robustness in deep learning models to adversarial examples. Recent theoretical work proved that adversarial examples are ubiquitous in two-layers networks with sub-exponential width and ReLU or smooth activations, and multi-layer ReLU networks with sub-exponential width. We present a result of the same type, with no restriction on width and for general locally Lipschitz continuous activations. More precisely, given a neural network $f(,\cdot,;\mathbf{\theta})$ with random weights $\mathbf{\theta}$, and feature vector $\mathbf{x}$, we show that an adversarial example $\mathbf{x}'$ can be found with high probability along the direction of the gradient $\nabla\_{\mathbf{x}}f(\mathbf{x};\mathbf{\theta})$. Our proof is based on a Gaussian conditioning technique. Instead of proving that $f$ is approximately linear in a neighborhood of $\mathbf{x}$, we characterize the joint distribution of $f(\mathbf{x};\mathbf{\theta})$ and $f(\mathbf{x}';\mathbf{\theta})$ for $\mathbf{x}' = \mathbf{x}-s(\mathbf{x})\nabla\_{\mathbf{x}}f(\mathbf{x};\mathbf{\theta})$, where $s(\mathbf{x}) = \operatorname{sign}(f(\mathbf{x}; \mathbf{\theta})) \cdot s\_d$ for some positive step size $s\_d$.