Anomaly Detection using Distributed Log Data: A Lightweight Federated Learning Approach

Large-scale software systems are generally deployed on distributed machines. Logs are usually collected from those machines for comprehensive and accurate system fault analysis. However, there are potential challenges during log transmission from distributed machines to third-party data analytics services. First, uploading massive raw logs causes tremendous bandwidth consumption. Moreover, user privacy contained in logs is easy to get leaked during transmission. To address these issues, we introduce federated learning for anomaly detection using distributed log data. However, gradient updates of model parameters transmitted between the server (third-party data analytics services) and participants (distributed machines) in federated learning have been proved of possible recovery by attackers, so encryption of gradient updates is necessary for enhanced privacy protection. Considering that encryption time is proportional to the number of parameters, we propose a lightweight federated learning method for anomaly detection, named FLOGCNN, using distributed log data. The sever in FLOGCNN aggregates gradient updates according to the sample size of participants to generate an integrated model. For local training, participants apply an anomaly detection model based on one-dimensional convolution with much fewer parameters. Extensive experiments are conducted for FLOGCNN using open log datasets. Results demonstrate that FLOGCNN outperforms baseline methods on anomaly detection and reduces 97.08% parameters in comparison with one baseline method. Furthermore, we perform exploratory experiments on lightweight models and results manifest that logs with simple semantic information are suitable for lightweight anomaly detection models.