Testing Ethereum Smart Contracts: A Comparison of Symbolic Analysis and Fuzz Testing Tools

ABSTRACT Ethereum smart contract exploits have inflicted enormous monetary damage due to vulnerabilities introduced accidentally by the contract authors. Many of these errors can now be detected automatically by a growing number of security analysis tools that specifically target the most common vulnerabilities present in the Ethereum smart contract ecosystem. The aim of this work is to identify state-of-the-art security analysis tools that assist auditors in automatically testing and verifying real-world contracts. We compare two such symbolic executioners, Manticore (which we also extend) and Mythril, and one fuzz tester, Echidna, to evaluate their effectiveness when analysing a set of challenge contracts hosted online, as well as twenty of the most popular ERC-20 tokens found on the main Ethereum network. Our results showed that the tools were able to solve 24 of the 39 challenge contracts and both symbolic tools achieved on average more than 80% code coverage on successful evaluations of the popular ERC-20 token data. Code coverage plateaued after the second symbolic transaction, suggesting a good performance target for continuous integration environments.