Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks
CoRR(2024)
摘要
Recently, neural networks utilizing periodic activation functions have been
proven to demonstrate superior performance in vision tasks compared to
traditional ReLU-activated networks. However, there is still a limited
understanding of the underlying reasons for this improved performance. In this
paper, we aim to address this gap by providing a theoretical understanding of
periodically activated networks through an analysis of their Neural Tangent
Kernel (NTK). We derive bounds on the minimum eigenvalue of their NTK in the
finite width setting, using a fairly general network architecture which
requires only one wide layer that grows at least linearly with the number of
data samples. Our findings indicate that periodically activated networks are
notably more well-behaved, from the NTK perspective, than ReLU
activated networks. Additionally, we give an application to the memorization
capacity of such networks and verify our theoretical predictions empirically.
Our study offers a deeper understanding of the properties of periodically
activated neural networks and their potential in the field of deep learning.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要