A novel framework for image-based malware detection with a deep neural network

AbstractAbstractThe rapid growth in the number of malware and its variants has seriously affected the security of the Internet. In recent years, deep learning combined with visualization technology has been proven to have good results in malware detection. In this paper, we propose a novel visual malware detection framework based on deep neural networks. Firstly, executable file samples are collected and converted into bytes files and asm files through disassembly technology. In this way, a balanced experimental dataset with our labeled normal software dataset and a widely used malware dataset (BIG 2015) is constructed. Secondly, visualization technology combined with data augmentation is used to further convert the samples into three-channel RGB images, so as to extract high-dimensional intrinsic features from data samples. Finally, we present a deep neural network architecture, i.e. SERLA (SEResNet50 + Bi-LSTM + Attention) to improve the performance of the detection method. After performance evaluation, the results show that our model stands out among other neural network models and state-of-the-art methods for malware detection and classification. Furthermore, our study verifies the superiority of three-channel RGB images compared to grayscale images in malware detection, compares the contribution of different channels, and indicates that data augmentation technology can contribute to malware recognition using visualization technology. This paper provides new ideas and methods for other researchers to carry out malware detection and classification.