THperf: Enabling Accurate Network Latency Measurement for Tianhe-2 System.

To prevent the network from becoming a potential performance bottleneck, two approaches commonly adopted in recent High-Performance Computing systems are offloading the network stack and Remote Direct Memory Access. Notably, these technologies require precise measurement of network latency due to the increasing granularity of congestion control mechanisms. This demand is more urgent for the Tianhe system because of the huge volume and performance requirements. However, the Tianhe system lacks an accurate network latency measurement tool due to its customizable nature. In this paper, we propose THperf, a network round-trip latency measurement tool for the Tianhe network architecture that can accurately obtain network latency information without the need for hardware support. We perform accurate network latency measurement at various traffic sizes with THperf and analyze the obtained the latency as a percentage of all latency. In addition, we compare THperf to other measurement tools and discover that THperf can eliminate a significant amount of latency. Using THperf, we further measure and compare the performance of different Submission Queues in the Tianhe network, demonstrating that the hardware descriptor submission method outperforms the software descriptor submission method in terms of latency. We believe that THperf can be used as a controlled benchmark for more granular congestion control algorithms.