Facial Expression Recognition of Animated Characters using Deep Learning

We apply computer vision and deep learning techniques to detect occurrences of a character from animated feature films and subsequently classify its emotions. For children on the autism spectrum, being able to identify facial expressions plays a vital role in the improvement of social responsiveness. Animated Language Learning (ALL), an Ireland based company, use manually labelled images from animated feature films, in interactive software, to help teach speech and language skills to children with autism. Our motivation is to help enable the automated detection of characters and labelling of emotions to improve their workflow. To detect occurrences of a character, our method uses a custom Haar classifier. For this paper, we focus on one character. We manually delete false positives and then manually label the occurrences with five emotion labels. We analyse the inter-annotator agreement and find that this emotion classification task is difficult even for humans. We benchmark two new custom CNNs with a performance analysis against pre-trained state-of-the-art deep CNNs. The models were firstly trained and validated on images from the Toy Story movies 1 to 3, and then tested on Toy Story 4 and some related Pixar short films. As an added step, we also visualise the outputs of key neurons to help understand the best model. The experimental results show that one of our developed models out-performs the best baseline model with an accuracy value of 77.65%.