On Multi-stream Classification of Two Person Interactions in Video with Skeleton-Based Features.

A method of human skeleton-tracking and -refinement, and feature extraction for two-person interaction recognition in video is proposed. Its purpose is to properly reassign the same person-representing skeletons, approximate the missing joints and extract meaningful relational features. In addition, based on the created feature streams, two different multi-stream deep neural networks are designed to perform data transformation and interaction classification. They provide different relations between model complexity and performance quality. The first one is an ensemble of “weak” pose-based action classifiers, which are trained on different time-phases of an interaction. At the same time, the overall classification result is a time-driven aggregation of weighted combinations of their results. In the second approach, three input feature streams were created, which fed a triple-stream LSTM network. Both network models were trained and tested on the interaction subset of the NTU RGB+D data set, showing comparable performance with the best reported CNN- and Graphic CNN-based classifiers.