Evaluating Deep Packet Inspection in Large-scale Data Processing

The Internet has evolved to the point that gigabytes and even terabytes of data are generated and processed on a daily basis. Such a stream of data is characterised by high volume, velocity and variety and is referred to as Big Data. Traditional data processing tools can no longer be used to process big data, because they were not designed to handle such a massive amount of data. This problem concerns also cyber security, where tools like intrusion detection systems employ classification algorithms to analyse the network traffic. Achieving a high accuracy attack detection becomes harder when the amount of data increases and the algorithms must be efficient enough to keep up with the throughput of a huge data stream. Due to the challenges posed by a big data environment, some monitoring systems have already shifted from deep packet inspection to flow-level inspection. The goal of this paper is to evaluate the applicability of an existing intrusion detection technique that performs deep packet inspection in a big data setting. We have conducted several experiments with Apache Spark to assess the performance of the technique when classifying anomalous packets, showing that it benefits from the use of Spark.