When Gestures and Words Synchronize: Exploring A Human Lecturer's Multimodal Interaction for the Design of Embodied Pedagogical Agents

Embodied Pedagogical Agents (EPAs) possess significant potential for using multimodal interactions to effectively convey social and educational information. While previous studies have examined the impact of EPAs on learning performance, the incorporation of meaningful gestures into EPAs remains relatively unexplored. This study employs a case study approach and systemic functional-multimodal discourse analysis to investigate the multimodal interactions of a human lecturer during teaching and explores the relationship between in-class meaningful gestures and corresponding words. We analyze the alignment between gestures, words, and contextual meanings, with a specific focus on personal pronouns, adjectives, adverbs, and verbs. The findings contribute to the understanding of lecturers' multimodal orchestration in classroom settings. Additionally, we provide a preliminary design guideline for EPAs that integrates meaningful word-synchronized gestures, using multimodal information to enhance students' embodied learning experiences. This research contributes to the advancement of embodied pedagogical agent design and the broader field of multimodal interaction.