Comparing trace visualizations for program comprehension through controlled experiments

For efficient and effective program comprehension, it is essential to provide software engineers with appropriate visualizations of the program's execution traces. Empirical studies, such as controlled experiments, are required to assess the effectiveness and efficiency of proposed visualization techniques. We present controlled experiments to compare the trace visualization tools Extravis and Explor Viz in typical program comprehension tasks. We replicate the first controlled experiment with a second one targeting a differently sized software system. In addition to a thorough analysis of the strategies chosen by the participants, we report on common challenges comparing trace visualization techniques. Besides our own replication of the first experiment, we provide a package containing all our experimental data to facilitate the verifiability, reproducibility and further extensibility of our presented results. Although subjects spent similar time on program comprehension tasks with both tools for a small-sized system, analyzing a larger software system resulted in a significant efficiency advantage of 28 percent less time spent by using Explor Viz. Concerning the effectiveness (correct solutions for program comprehension tasks), we observed a significant improvement of correctness for both object system sizes of 39 and 61 percent with Explor Viz.