HCMonitor: An accurate measurement system for high concurrent network services

This article aims to enhance the monitoring accuracy of high concurrent network services. As modern network services grow rapidly in data centers, tail latency has become one of the most crucial deciding factors on user experience. Latency measurement and anomaly detection are essential in evaluating service performance. Existing monitoring tools can be divided into two categories according to estimation methods. First, approaches based on sample traffic sample network packets to unburden the measurement. Second, approaches based on full traffic like wrk, analyze all of the packets from the kernel network stack and load the client-side overhead into response delay. Therefore, we propose a high-performance monitor system named HCMonitor, which computes the server-side response latency and the round-trip time of per-request. It can afford full traffic monitoring on the basis of userspace, "zero copy" and pipeline. By switch mirroring, the measured latency eliminates the kernel network stack overhead and the queuing delay of the client-side. Such measurement results in improved accuracy, online analysis, anomaly detection, real-time display and transparent to network services. Our evaluations show HCMonitor obtains a higher throughput compared with tcpdump by over 200 times. Compared with wrk, the tail latency accuracy shows an increase by up to 72%-76% in high concurrent networks.