An empirical study of problems and evaluation of IoT malware classification label sources

With the proliferation of malware on IoT devices, research on IoT malicious code has also become more mature. Most studies use learning models to detect or classify malware. Therefore, ensuring high-quality labels for malware samples is crucial to maintaining research accuracy. Researchers typically submit malware samples to Anti-Virus (AV) engines to obtain labels, but different engines have varying rules for detecting maliciousness and variants. This study aims to improve future IoT malware research accuracy by investigating label quality. We address three label-related issues, including Anti-Virus detection technology, naming rules, and label expiration. Additionally, we examine multiple sources of malware labels, including 63 studies on IoT, Windows, and Android malware, as well as commonly used tools such as AVClass and Anti-Virus engines. To evaluate and recommend label sources, we construct classification models using an IoT malware dataset obtained from VirusShare, which is labeled with common tools and Anti-Virus engines and classified based on ELF features. For family classification, we investigate four methods and tools, and for variant hierarchy classification, we compare label overlaps with sample clustering from 12 Anti-Virus engines. Based on our findings, we recommend AVClass for obtaining labels for IoT family classification. For small-scale malware families at the variant level, we recommend using the labels from Ad-Aware, BitDefender, and Emsisoft engines. For large-scale malware families, we advise employing labels from Jiangmin, NANO-Antivirus, and Avira engines, serving as a valuable guide for future IoT malware research.