Threshold based Technique to Detect Anomalies using Log Files.

Every action carried out on computer systems can be captured using log files. Proper scanning of log files can divulge security breaches. However, a large-scale data processing engine should analyze the log files due to the voluminous events in log files. This paper proposes an anomaly detection approach using a threshold to discriminate between regular and aberrant log files. The experiments are performed on HDFS, a publicly available log dataset. The system's efficacy is evaluated using Robust Random Cut Forest (RRCF), an unsupervised tree-based approach where we achieved precision 97.10% and F1-score 98.47% results. Hadoop framework is utilized to run the experiments due to its capability of parallel processing tasks in less time, even on large datasets.