Tropical Expressivity of Neural Networks
CoRR(2024)
摘要
We propose an algebraic geometric framework to study the expressivity of
linear activation neural networks. A particular quantity that has been actively
studied in the field of deep learning is the number of linear regions, which
gives an estimate of the information capacity of the architecture. To study and
evaluate information capacity and expressivity, we work in the setting of
tropical geometry – a combinatorial and polyhedral variant of algebraic
geometry – where there are known connections between tropical rational maps
and feedforward neural networks. Our work builds on and expands this connection
to capitalize on the rich theory of tropical geometry to characterize and study
various architectural aspects of neural networks. Our contributions are
threefold: we provide a novel tropical geometric approach to selecting sampling
domains among linear regions; an algebraic result allowing for a guided
restriction of the sampling domain for network architectures with symmetries;
and an open source library to analyze neural networks as tropical Puiseux
rational maps. We provide a comprehensive set of proof-of-concept numerical
experiments demonstrating the breadth of neural network architectures to which
tropical geometric theory can be applied to reveal insights on expressivity
characteristics of a network. Our work provides the foundations for the
adaptation of both theory and existing software from computational tropical
geometry and symbolic computation to deep learning.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要