Inverse articulated-body dynamics from video via variational sequential Monte Carlo

Convolutional neural networks for pose estimation are continuously improving in identifying joints of moving agents from video. However, state-of-the-art algorithms offer no insight into the underlying mechanics of articulated limbs. "Seeing" the mechanics of movement is of major importance for fields like neuroscience, studying how the brain controls movement, and engineering, e.g., using vision to correct for errors in the action of a robotic manipulator. In the pipeline proposed here, we use a convolutional network to track joint positions, and embed these as the joints of a linked robotic manipulator. We develop a probabilistic physical model whose states specify second-order rigid-body dynamics and the torques applied to each actuator. Observations are generated by mapping the joint angles through the forward kinematics function to Cartesian coordinates. For nonlinear state estimation and parameter learning, we build on variational Sequential Monte Carlo (SMC), a differentiable variant of the classical SMC method leveraging variational inference. We extend with a distributed nested SMC algorithm, which, at inference time, wraps multiple independent SMC samplers within an outer-level importance sampler. We extract mechanical quantities from simulated data and newly acquired videos of mice and humans, offering a novel tool for studying e.g. biological motor control.