Detecting and Reacting to Anomalies in Relaxed Uses of Raft

The Raft consensus algorithm is used in many popular distributed key–value stores to offer strong consistency. Due to the cost of implementing strong consistency, its performance characteristics may not meet the requirements of some users. To satisfy these users, many distributed key–value stores allow users to bypass Raft when serving read requests. Unfortunately, yet predictably, this introduces anomalies. While this is a tradeoff many users may be willing to make, the effects of the tradeoff are not properly accounted for: i.e., it is impossible to know how much consistency is being traded away for the increased speed. We propose the use of reflective consistency—a design space of consistency models used to expose anomaly statistics to the system and its users—to regain transparency in the tradeoff space. This work presents the complete lifecycle of implementing an instance of reflective consistency. We first describe how a popular feature in strongly consistent distributed key–value stores causes anomalies. We then design a reflective consistency implementation which can quantify the existence of anomalies. Finally, we evaluate the implementation, showing that, with nearly zero overhead, users are able to regain control over the anomaly behavior of their distributed storage systems.