Unsupervised Sequence Alignment between Video and Human Center of Pressure

Center of pressure (COP) estimation with images/videos as input achieves accurate precision with the development of the human skeleton joint extraction tasks. As a supervised learning task, correct labels acquired from COP with regard to the input images/videos are significant. Thus, synchronization between these two different types of sequences is necessary. If these two different modalities are misaligned, the downstream tasks' precision is affected significantly due to the inaccurate labels from the COP sequence. In this paper, we used a synchronized dataset and unsupervised deep learning to train an Alignment Network to align video and COP sequences on another unsynchronized dataset where each sequence starts at a different time and has different frame rates. On the synchronized dataset, the Alignment Network removes 84.4% of temporal offset. On the unsynchronized dataset, we proposed a simple yet effective Differential Network to simulate one practical downstream task. We used the differential Network to estimate the sway level of COP. Results show that this method achieved significant improvement (over 20% improvement on three sway level cases) over the misaligned dataset.