Studying Children’s Object Interaction in Virtual Reality: A Manipulative Gesture Taxonomy for VR Hand Tracking

In this paper, we propose a taxonomy for the classification of children’s gestural input elicited from spatial puzzle play in VR hand tracking. The taxonomy builds on the existing manipulative gesture taxonomy in human-computer interaction, and offers two main analytical categories; Goal-directed actions and Hand kinematics as complementary dimensions for analysing gestural input. Based on our study with eight children (aged between 7-14), we report the qualitative results for describing the categories for analysis and quantitative results for their frequency in occurring in children’s interaction with the objects during the spatial task. This taxonomy is an initial step towards capturing the complexity of manipulative gestures in relation to mental rotation actions, and helps designers and developers to understand and study children’s gestures as an input for object interaction as well as an indicator for spatial thinking strategies in VR hand tracking systems.