Mumak: Efficient and Black-Box Bug Detection for Persistent Memory

The advent of Persistent Memory (PM) opens the door to novel application designs that explore its performance and durability benefits. However, there is no free lunch, and to program PM applications, developers need to be aware of potential inconsistent application state upon machine or application crashes. To overcome this difficulty, several tools have been proposed to detect the presence of the so-called crash-consistency bugs. While these are effective in detecting a variety of bugs, they present several key limitations, namely relying on application-specific semantics, requiring the programmer to manually annotate the program or modify the PM library, and relying on techniques with poor scalability, making them impractical for production code. In this paper, we introduce Mumak, a tool that detects bugs in PM applications in an efficient and black-box manner. Our key insight to reduce the search space is to use a two-pronged approach with a first pass that is highly efficient by focusing only on key, error-prone code points without exhaustively testing all possible persistence orderings, and a second pass based on heuristics that try to compensate the shortcomings of the initial approach. Furthermore, we avoid application-specific knowledge or annotations by relying on the application's own recovery procedure as an (imperfect) consistency oracle. Our experimental results, with different applications and libraries, show that Mumak has bug coverage on par with the other state-of-the-art tools, while being up to 25x faster. We also found four new crash-consistency bugs, two in PMDK and two in Montage, three of which have already been acknowledged and fixed by the developers.