PLELog: semi-supervised log-based anomaly detection via probabilistic label estimation

ABSTRACTPLELog is a novel approach for log-based anomaly detection via probabilistic label estimation. It is designed to effectively detect anomalies in unlabeled logs and meanwhile avoid the manual labeling effort for training data generation. We embed semantic information within log events as fixed-length vectors and apply HDBSCAN to automatically cluster log sequences. After that, we also propose a Probabilistic Label Estimation approach to automatically label log sequences, which can reduce the noises introduced by error labeling and put "labeled" instances into an attention-based GRU network for training. We conducted an empirical study to evaluate the effectiveness of PLELog on two open-source log data (i.e., HDFS and BGL). The results demonstrate the effectiveness of PLELog. In particular, PLELog has been applied to two real-world systems from a university and a large corporation, and the results further demonstrate its practicability.