Depth Separation in Norm-Bounded Infinite-Width Neural Networks
CoRR(2024)
Abstract
We study depth separation in infinite-width neural networks, where complexity
is controlled by the overall squared ℓ_2-norm of the weights (sum of
squares of all weights in the network). Whereas previous depth separation
results focused on separation in terms of width, such results do not give
insight into whether depth determines if it is possible to learn a network that
generalizes well even when the network width is unbounded. Here, we study
separation in terms of the sample complexity required for learnability.
Specifically, we show that there are functions that are learnable with sample
complexity polynomial in the input dimension by norm-controlled depth-3 ReLU
networks, yet are not learnable with sub-exponential sample complexity by
norm-controlled depth-2 ReLU networks (with any value for the norm). We also
show that a similar statement in the reverse direction is not possible: any
function learnable with polynomial sample complexity by a norm-controlled
depth-2 ReLU network with infinite width is also learnable with polynomial
sample complexity by a norm-controlled depth-3 ReLU network.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined