When Are Bias-Free ReLU Networks Like Linear Networks?
arxiv(2024)
摘要
We investigate the expressivity and learning dynamics of bias-free ReLU
networks. We firstly show that two-layer bias-free ReLU networks have limited
expressivity: the only odd function two-layer bias-free ReLU networks can
express is a linear one. We then show that, under symmetry conditions on the
data, these networks have the same learning dynamics as linear networks. This
allows us to give closed-form time-course solutions to certain two-layer
bias-free ReLU networks, which has not been done for nonlinear networks outside
the lazy learning regime. While deep bias-free ReLU networks are more
expressive than their two-layer counterparts, they still share a number of
similarities with deep linear networks. These similarities enable us to
leverage insights from linear networks, leading to a novel understanding of
bias-free ReLU networks. Overall, our results show that some properties
established for bias-free ReLU networks arise due to equivalence to linear
networks, and suggest that including bias or considering asymmetric data are
avenues to engage with nonlinear behaviors.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要