Depth Separation for Neural Networks

Let f:𝕊^d-1×𝕊^d-1→𝕊 be a function of the form f(𝐱,𝐱') = g(⟨𝐱,𝐱'⟩) for g:[-1,1]→ℝ. We give a simple proof that shows that poly-size depth two neural networks with (exponentially) bounded weights cannot approximate f whenever g cannot be approximated by a low degree polynomial. Moreover, for many g's, such as g(x)=sin(π d^3x), the number of neurons must be 2^Ω(dlog(d)). Furthermore, the result holds w.r.t. the uniform distribution on 𝕊^d-1×𝕊^d-1. As many functions of the above form can be well approximated by poly-size depth three networks with poly-bounded weights, this establishes a separation between depth two and depth three networks w.r.t. the uniform distribution on 𝕊^d-1×𝕊^d-1.