Evaluation of Volta based DGX 1 System Using DNN Workloads

Multi-GPU systems are being widely used to train deep neural networks (DNNs) as GPUs can significantly reduce the training time. Data parallelism is a popular choice to train DNNs on a multi-GPU system. GPUs in a multi-GPU system repeatedly perform Forward Propagation (FP), Backward Propagation (BP) and, Weight Update (WU) to train a DNN. During the WU stage, GPUs communicate with each other. To improve communication time, different data transfer mechanisms and libraries have been introduced by NVIDIA, and adopted by high-level frameworks to train DNNs. We evaluate the use of two of the most popular communication methods (peer-to-peer (P2P) data transfer and NCCL library-based communication) for training DNNs on a DGX-1 multi-GPU system. We profile and analyze the training of five DNNs using 1, 2, 4 and 8 GPUs. Our analyses provide insights into the software– and hardware–level limiting factors for training DNNs on a multi-GPU system.