Knowledge Distillation For Spiking Neural Network.

The spiking neural network (SNN) is an effective computing model that mimics the brain's complex information processing and transmission systems. The spike-based computing paradigm is used to emulate the highly efficient processing capabilities of the human brain. Hence, it may be argued that SNNs exhibit superior biological interpretability in comparison to conventional artificial neural networks (ANN). However, similar to ANNs, the enhancement in performance of SNNs is likewise influenced by the increase in the number of network layers, and there is also a corresponding increase in the resources required for task execution. Efficiently doing complicated tasks with little resources is of utmost importance. Knowledge distillation(KD) is a widely used technique within the domain of transfer learning that can enhance the efficiency of a smaller model by transferring the learned information from a larger model to facilitate the training of the smaller target model. However, there is an absence of research on its implementation in SNNs. This paper utilizes the well-known KD algorithm in ANNs to implement it in SNNs, with the objective of exploring the application of the KD technique in the context of SNN s and then evaluating its effectiveness in the task of image classification. We conducted three separate exploratory endeavors. 1. Utilize the ANN as the teacher model to train a student model including an SNN architecture (referred to as ANN-KD-SNN). 2. Investigate the effects of changing time steps on the distillation process within the ANN-KD-SNN paradigm. 3. Exploit the SNN as the teacher model to facilitate the training of a student model, which is also an SNN (referred to as SNN-KD-SNN). Our experiments demonstrate the necessity and viability of employing KD in SNNs