Vis Repligogy: Towards a Culture of Facilitating Replication Studies in Visualization Pedagogy and Research

Data visualization is a young and growing field with many unresolved, debated, and untested theories that need to be tested, validated, and strengthened. Replication studies are a common way to achieve this validation by investigating the credibility, rigor, and generalizability of previously published research. The execution of replication studies is crucial to strengthen the field of data visualization and to ensure that the field’s foundations are solid with accurate methods and theories. Although visualization researchers acknowledge the epistemological significance of replications and its necessity to establish trust and reliability, the field has made little progress to support the publication of such studies and importantly provide methods to the community to encourage replications. In this paper, we contribute Vis Repligogy, a novel framework to incorporate replication studies within visualization course curricula that not only teaches students replication and evaluation methodologies but also results in executed replication studies to validate prior work. To validate the feasibility of the framework, we present the results of four replication studies that were conducted in a graduate data visualization course that implemented the Vis Repligogy framework. We also provide a meta-analysis of the four replication studies that replicated and validated, a seminal work in data visualization investigating the effects of visual embellishments on chart interpretability and memorability. Finally, we reflect on our experience of conducting the replication studies through the Vis Repligogy framework and reflect on the experience to provide useful recommendations for future use of the framework. We envision that this framework will encourage instructors to conduct replications in their own courses and will help facilitate more replication studies in the visualization community to support a culture shift where replications are commonplace.