Deep Transfer Learning for Recognizing Functional Interactions via Head Movements in Multiparty Conversations

ABSTRACT Head movements play various functions in multiparty conversations. To date, convolutional neural networks (CNNs) have been proposed to recognize the functions of individual interlocutors’ head movements. This paper extends the concept of head-movement functions to the interaction functions between speaker and listener, which are performed through their head movements, e.g., a listener’s back-channel nodding in response to a speaker’s rhythmic movements. Then, we propose transfer strategies to build deep neural networks (DNNs) to recognize these interaction functions by reusing pretrained CNNs for individual head-movement functions. One of the proposed strategies uses CNNs as the feature extractor and identifies the interaction function with another classifier using the extracted features. Compared with the baseline model that employs the logical product of the output of two individual CNNs, the transferred DNNs outperform the baseline model in four out of five interaction functions. For example, the F-measure is improved by 13.9 points for the interaction of a listener’s positive emotion in response to a speaker’s rhythmic movements. These results confirm the potential of the proposed transfer strategies for recognizing interaction functions based on head movements.