Cooperative Detection of Camouflaged Malicious TLS Traffic.

Recently, advanced hackers have exhibited a propensity to camouflage their attack traffic as legitimate Transport Layer Security (TLS) flows to evade detection. As an illustration, a hacker may first compromise a regular application server, and then leverage the server’s private keys and certificates to establish a TLS connection for transmitting attack commands or extracting stolen data. Detecting such malicious TLS connections poses a significant challenge, as they closely resemble normal traffic. This research aims to address this challenge through a novel cooperative detection technique. The proposed method constructs multiple behavior models for each legitimate server based on observed network traffic to characterize its normal applications. By comparing these constructed behavior models, the proposed method can efficiently detect malicious traffic since the behavior of malware is significantly different from that of normal applications. While the behavior models may be incomplete due to insufficient traffic data, which can lead to false positives, we further design a cooperative mechanism to enhance the detection effectiveness. Experimental results demonstrate that the proposed method achieves substantially higher performance compared to the prevalent anomaly detection approaches.