LayerLog: Log sequence anomaly detection based on hierarchical semantics

System logs record the running status of systems, and log anomaly detection can help locate anomalies timely to reduce error time and ensure normal operation. Logs in text format contain abundant semantic information. Still, the methods based on log count vector and log key sequence do not take into account the semantics of log data, causing a high missing detection rate. And existing log semantics-based methods fail to fully consider the semantic information among words, logs, and log sequences. Besides, current methods require the usage of Log Parser in preprocessing, which could negatively impact detection accuracy and cause semantic loss. In this paper, we discover the three -layered structure of log data, named the "Word Log-Log Sequence"hierarchy, and propose LayerLog, a novel framework for log sequence anomaly detection based on the hierarchical semantics of log data. Without Log Parser in preprocessing phase, LayerLog can effectively extract semantic features from each layer and is the first framework to consider the semantics of words, logs, and log sequence. What is more, LayerLog can detect execution order anomaly, operation anomaly, and incomplete anomaly of log sequence simultaneously in an end-to-end way. We have evaluated sufficient experiments on two commonly-used public datasets and the experimental results confirm the effectiveness of LayerLog. (c) 2022 Elsevier B.V. All rights reserved.