Meizodon: Security Benchmarking Framework for Static Android Malware Detectors

Many Android applications are uploaded to app stores every day. A relatively small fraction of these applications, or apps, is malware. Several research teams developed tools which automate malware detection for apps, to keep up with the never-ending stream of uploaded apks (Android PacKages). Every tool seemed better than the last, some even claiming accuracy scores well over 90%. However, all of these designs were tested against test sets containing only self-written apks, synthetic malicious apks, or otherwise statistically unsound samples. Many of these tools are open source. We propose Meizodon, a novel framework to install Android static security analysis tools and run them efficiently in a distributed fashion, in equal environments and against a suitable dataset. This allows us to make a fair and statistically sound comparison of the most recent and best known tools, on real, 'practical' malware: malware created by malware creators, not by researchers, and found in the wild. From the results, we conclude that Android static security analysis tools do show great promise to classify apks in practice, but are not quite there yet. We demonstrate that Meizodon allows us to efficiently test analysis tools, and find that the accuracy of tested analysis tools is low (F1 scores are just over 58%), and analysis fails for many apks. Additionally, we investigate why accuracy is low, and why so many analyses result in errors.