Accurate markerless jaw tracking for facial performance capture

Abstract

We present the first method to accurately track the invisible jaw based solely on the visible skin surface, without the need for any markers or augmentation of the actor. As such, the method can readily be integrated with off-the-shelf facial performance capture systems. The core idea is to learn a non-linear mapping from the skin deformation to the underlying jaw motion on a dataset where ground-truth jaw poses have been acquired, and then to retarget the mapping to new subjects. Solving for the jaw pose plays a central role in visual effects pipelines, since accurate jaw motion is required when retargeting to fantasy characters and for physical simulation. Currently, this task is performed mostly manually to achieve the desired level of accuracy, and the presented method has the potential to fully automate this labour intense and error prone process.

Comparison between Occlusal Errors of Single Posterior Crowns Adjusted Using Patient Specific Motion or Conventional Methods

“…Conventional mandibular motion tracking based on optical tracking devices has some disadvantages, such as increased cost, the time required for preparing devices, and the existence of a learning curve [7]. In addition, the existence of the marker makes clinical procedures time-consuming and more technique-sensitive [58]. The marker attached to the teeth might interfere with the movement of the mandible [57].…”

3D Morphable Face Models—Past, Present, and Future

“…While most hair acquisition focuses on static reconstruction, some do capture hair in motion or estimate physical properties for hair simulation [Hu et al 2017a]. Especially challenging is the acquisition of partially or completely hidden properties, such a the tongue [Hewer et al 2018], the skull [Achenbach et al 2018;, or the jaw [Zoss et al 2019[Zoss et al , 2018, where oftentimes specialized imaging systems are required, such as Computer Tomography (CT), Magnetic Resonance Imaging (MRI), or Electromagnetic Articulography (EMA). Lastly, even skin itself requires specialized treatment in some areas, such as lips [Garrido et al 2016b] or eyelids [Bermano et al 2015], where the local appearance and deformation exceed the capabilities of the more generic methods.…”

Visual Analytics in Dental Aesthetics

“…The aforementioned approaches require that a significant part of the teeth is visible and they address different applications. Zoss et al [ZBGB19] track the jaw movements, based solely on the visible skin surface. Their approach requires training data and may provide inaccurate results if the patient has unique jaw movements, which cannot be triangulated inside the convex hull of the training set.…”