Exploring Effective Fuzzing Strategies to Analyze Communication Protocols

While coverage-based greybox fuzzing has gained great success in the field of vulnerability detection due to its simplicity and efficiency, it could become less powerful when applied directly to protocol fuzzing due to the unique challenges of protocol fuzzing. In particular, (1) The implementation of protocols usually involves multiple program binaries, i.e., multiple fuzzing entries; (2) The communication among multiple ends contains more than one packet, which are not necessarily dependent upon each other, i.e., fuzzing single (usually the first) packet can only achieve extremely limited code coverage. In this paper, we study such challenges and demonstrate the limitation of current non-stateful greybox fuzzer. In order to achieve higher code coverage, we design and implement a stateful protocol fuzzer, yFuzz , to explore the code related to different protocol states. yFuzz is built on AFL (a mainstream greybox fuzzer), and incorporates a stateful fuzzer (which contains a state switching engine) together with a multi-state forkserver (which enables multi-state program forking) to consistently and flexibly fuzz different states of a compiler-instrumented protocol program. Our experimental results on OpenSSL show that yFuzz improves the code coverage by 73% and increases the number of identified unique crashes by 100% when comparing against AFL fuzzing the first packet during a protocol handshake.