ConvDarts: a fast and exact convolutional algorithm selector for deep learning frameworks

Convolution is one of the most time-consuming operations in training deep neural networks. Existing convolutional algorithms, such as FFT, GEMM, Winograd and their varieties, have different performances in time and space. However, there is no best algorithm for all convolution configurations (parameters of convolutional operations). This paper addresses the problem of convolutional algorithm selection for given configurations and proposes a fast and exact selector ConvDarts. We propose an informed cache that is preset with common convolution configurations and their optimal algorithm indices. A lightweight machine learning model is also used to predict the optimal convolutional algorithm for cache missing configurations. Compared with the heuristics and profiling approaches exploited in cuDNN, it not only reduces the training time of classical deep learning networks but also reduces the required memory space. The selector ConvDarts proposed in this paper provides more possibilities for the training of network models in resource-constrained environments.