The use of Flow Features in Lossy Network Traffic Compression for Network Intrusion Detection Applications

In distributed network intrusion detection applications, it is necessary to transmit data from the remote sensors to the central analysis systems (CAS). Transmitting all the data captured by the sensor would place an unacceptable demand on the bandwidth available to the site. Most applications address this problem by sending only alerts or summaries; however, these alone do not always provide the analyst with enough information to truly understand what is happening on the network. Lossless compression techniques alone are not sufficient to address the bandwidth demand; therefore, some form of lossy compression must be employed. Working on the theory that a network flow that is malicious will manifest this maliciousness early, we explore the impact of compressing network traffic by stopping the transmission of packets in a flow once a given threshold either in number of packets or number of bytes have been transmitted.