LongTail-Bench: A Benchmark Suite for Domain-Specific Operators in Deep Learning

Deep neural net works have brought significant innovations in many domains, such as computer vision, natural language processing, speech recognition, and intelligent decision. To this end, many new operators have also been developed to attain better accuracy in the specific domain, such as the operators about anchors in object detection and operators about agent-environment interaction in reinforcement learning. In this paper, we identify that the new domain-specific operators, which have no corresponding implementation in the device compute library (such as cuDNN, and PyTorch ATen) and have to resort to the Python interpreter, will introduce large amounts of challenges in deep learning training and deployment. We name these operators long-tail operators, inspired by the meaning of the long-tail phenomenon in business and statistics.As such, researchers have developed kinds of deep learning compilers, such as XLA, TorchScript, $\mathcal{J}{A X}$, and TVM, to solve the challenges incurred by long-tail operators. Since there are a lot of complex syntax features in the long-tail operators, providing a well-designed benchmark suite to assess and profile the deep learning compilers is of much importance. Unfortunately, there have been no representative benchmark suites that can take a solid understanding and analysis of long-tail operators. In this paper, we propose LongTail-Bench ¹ , a benchmark suite for domain-specific operators in deep learning. To help researchers study deep learning compilers, LongTail-Bench collects more than 100 representative operators. We first perform a comprehensive analysis and show that LongTail-Bench can cover a wide range of syntax features. Then, we conduct a throughout evaluation of the mainstream deep learning compilers from multiple aspects, including training performance, deploy performance, comparison with handwritten code, and performance on the emerging deep learning accelerator. Finally, we take a research example to demonstrate that our benchmark suite can help to support research. ¹ Corresponding Author