Diagnostic visualization for non-expert machine learning practitioners: A design study.

As machine learning (ML) becomes increasingly popular, developers without deep experience in ML who we will refer to as ML practitioners are facing the need to diagnose problems with ML models. Yet successful diagnosis requires high-level expertise that practitioners lack. As in many complex data oriented domains, visualization could help. This two-phase study explored the design of visualizations to aid ML diagnosis. In phase 1, twelve ML practitioners were asked to diagnose a model using ten state-of-the-art visualizations; seven design themes were identified. In phase 2, several design themes were embodied in an interactive visualization. The visualization was used to engage practitioners in a participatory design exercise that explored how they would carry out multi-step diagnosis using the visualization. Our findings provide design implications for tools that better support ML diagnosis by non-expert practitioners.