Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models
arxiv(2024)
摘要
Activity and parameter sparsity are two standard methods of making neural
networks computationally more efficient. Event-based architectures such as
spiking neural networks (SNNs) naturally exhibit activity sparsity, and many
methods exist to sparsify their connectivity by pruning weights. While the
effect of weight pruning on feed-forward SNNs has been previously studied for
computer vision tasks, the effects of pruning for complex sequence tasks like
language modeling are less well studied since SNNs have traditionally struggled
to achieve meaningful performance on these tasks. Using a recently published
SNN-like architecture that works well on small-scale language modeling, we
study the effects of weight pruning when combined with activity sparsity.
Specifically, we study the trade-off between the multiplicative efficiency
gains the combination affords and its effect on task performance for language
modeling. To dissect the effects of the two sparsities, we conduct a
comparative analysis between densely activated models and sparsely activated
event-based models across varying degrees of connectivity sparsity. We
demonstrate that sparse activity and sparse connectivity complement each other
without a proportional drop in task performance for an event-based neural
network trained on the Penn Treebank and WikiText-2 language modeling datasets.
Our results suggest sparsely connected event-based neural networks are
promising candidates for effective and efficient sequence modeling.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要