How to Evaluate Mobile 360° Video Streaming Systems?

360° video brings an immersive experience to users by projecting panoramic content onto the display. Viewing 360° videos on untethered mobile platforms further enhances the immersive user experience. As a result, a large number of systems for optimized high-resolution 360° video streaming to mobile devices have been proposed over the past few years. In this paper, we review the diverse set of research methodologies in the system design and evaluation of recently proposed mobile 360° video streaming systems and discuss a number of pitfalls that prevent a fair and meaningful comparison among different systems. Our discussion suggests that there is an urgent need to redefine the design objectives and to develop an effective methodology for a meaningful evaluation and comparison of different systems. We finish with a set of concrete guidelines on the design and evaluation methodology of future mobile 360° video systems.