Multi-model Emotion Recognition Using Hybrid Framework of Deep and Machine Learning

Automated emotion research has grown significantly over the last two decades, with direct application across domains, such as medical science and chat-bot, all requiring an understanding of emotional psychology. Emotion is identified by various sources of emotion such as text, images, gesture, and video. The fusion process of various sources of information is called multi-model emotion recognition. In the proposed framework, a multi-model deep neural network is trained and further the input data is passed with the learned (pre-trained) model to obtain the feature vector. Further, the machine learning classifiers (such as SVM, Decision tree, Random forest, and XGBoost) are used to develop an emotion recognition model using the deep learning-based features. The best weighted-accuracy (WA) and unweighted-accuracy (UWA) are 71 and 66%, respectively, using Random forest classifier with the combination speech, text, and mocap.