Neural Architecture Search for Low-Precision Neural Networks

In our work, we extend the search space of the differentiable Neural Architecture Search (NAS) by adding bitwidth. The extended NAS algorithm is performed directly with low-precision from scratch without the proxy of full-precision. With our low-precision NAS, we can search for low- and mixed-precision network architectures of Convolutional Neural Networks (CNNs) under specific constraints, such as power consumption. Experiments on the ImageNet dataset demonstrate the effectiveness of our method, where the searched models achieve better accuracy (up to 1.2 percentage point) with smaller model sizes (up to 27% smaller) and lower power consumption (up to 27% lower) compared to the state-of-art methods. In our low-precision NAS, sharing of convolution is developed to speed up training and decrease memory consumption. Compared to the FBNet-V2 implementation, our solution reduces training time and memory cost by nearly 3x and 2x, respectively. Furthermore, we adapt the NAS to train the entire supernet instead of a subnet in each iteration to address the insufficient training issue. Besides, we also propose the forward-and-backward scaling method, which addresses the issue by eliminating the vanishing of the forward activations and backward gradients.