Unstructured Log Analysis for System Anomaly Detection—A Study

Nowadays, with the rapid pace of innovation, a typical production infrastructure is getting huge, complicated and difficult to manage. Hence, incident detection and action have become a challenge to the operations and information security (InfoSec) teams. As we are moving toward deployments of complicated or complex large-scale micro-service architectures, the kind of data generated from all those systems is huge. So, it becomes very difficult to identify if anything goes wrong in underlying systems, i.e., the system is vulnerable to various attacks. Keeping track of the flow of traffic and user activities on a large scale, complicated environment is becoming very costly and unmanageable. Traditional systems and ways are becoming inefficient for zero-day security issues.  So it is highly recommended to develop a system that is capable of raising an alarm for any detected anomaly after performing an automatic analysis of the generated logs. This study is conducted to review the research work on unstructured log analysis for the purpose of monitoring the system and anomaly detection. We have identified the datasets used for this purpose and also pointed out the challenges involved in unstructured data analysis.