FA-Fuzz: A Novel Scheduling Scheme Using Firefly Algorithm for Mutation-Based Fuzzing

Mutation-based fuzzing has been widely used in both academia and industry. Recently, researchers observe that the mutation scheduling scheme affects the efficiency of fuzzing. Accordingly, they propose PSO algorithm or machine learning-based technique to optimize the scheduling process. However, these methods fail to consider the fact that the optimal operator distribution of different seeds is different, even for the same program. In this paper, we propose a novel general scheduling scheme, named FA-fuzz, to find the optimal selecting probability distribution of mutation operators, which is based on the observations that the effective mutation operators are different for different seeds. Specifically, our method is based on the firefly algorithm. The positions of fireflies are mapped to the selection probability distribution of different mutation operators. The brightness of fireflies is expressed as the efficiency of discovering unique testcases. We implement prototype systems on multiple state-of-art fuzzers, and perform evaluations on two datasets. Our proposed method improves both the number of unique paths and unique bugs on real-world datasets. In addition, we discover 30 zero-day vulnerabilities in eight real-world programs, which demonstrate the effectiveness of FA-fuzz.