Analyzing Data Reference Characteristics of Deep Learning Workloads for Improving Buffer Cache Performance

Due to the recent growing data size of deep learning workloads, loading data from storage is increasingly becoming a performance bottleneck for neural network systems. In this article, we analyze the data reference characteristics of neural network workloads and observe that they are significantly different from conventional desktop workloads. In particular, during the training phase of deep learning, data blocks are referenced in a fully random manner, which significantly degrades the performance of a buffer cache. To handle this situation, this article suggests a new data shuffling scheme that aims to accelerate data loading in deep neural networks. Unlike the default shuffling method used in PyTorch that randomly shuffles full dataset in every epoch, the proposed scheme defines a shuffling unit called bundle, and enhances the locality of data references to improve buffer cache performances. Specifically, the proposed scheme performs data shuffling by the unit of a bundle, and the bundles used in each epoch are arranged alternately, thereby improving the locality of references at the viewpoint of the buffer cache. Based on simulation and measurement studies, we show that the hit rate of the buffer cache is improved by 37.2%, and the data loading time is also shortened by 11.4% without degrading the model's training efficiency.