SparseSoftDECA — Efficient High-Resolution Physics-Based Facial Animation from Sparse Landmarks

Facial animation on computationally limited systems still heavily relies on linear blendshape models. Nonetheless, these models exhibit common issues like volume loss, self-collisions, and inaccuracies in soft tissue elasticity. Furthermore, personalizing blendshapes models demands significant effort, but there are limited options for simulating or manipulating physical and anatomical characteristics afterwards. Also, second-order dynamics can only be partially represented.For many years, physics-based facial simulations have been explored as an alternative to linear blendshapes, however, those remain cumbersome to implement and result in a high computational burden. We present a novel deep learning approach that offers the advantages of physics-based facial animations while being effortless and fast to use on top of linear blendshapes. For this, we design an innovative hypernetwork that efficiently approximates a physics-based facial simulation while generalizing over the extensive DECA model of human identities, facial expressions, and a wide range of material properties that can be locally adjusted without re-training.In addition to our previous work, we also demonstrate how the hypernetwork can be applied to facial animation from a sparse set of tracked landmarks. Unlike before, we no longer require linear blendshapes as the foundation of our system but directly operate on neutral head representations. This application is also used to complement an existing framework for commodity smartphones that already implements high resolution scanning of neutral faces and expression tracking.