Video-Based Remote Physiological Measurement via Cross-Verified Feature Disentangling

Abstract

Remote physiological measurements, e.g., remote photoplethysmography (rPPG) based heart rate (HR), heart rate variability (HRV) and respiration frequency (RF) measuring, are playing more and more important roles under the application scenarios where contact measurement is inconvenient or impossible. Since the amplitude of the physiological signals is very small, they can be easily affected by head movements, lighting conditions, and sensor diversities. To address these challenges, we propose a cross-verified feature disentangling strategy to disentangle the physiological features with non-physiological representations, and then use the distilled physiological features for robust multi-task physiological measurements. We first transform the input face videos into a multi-scale spatial-temporal map (MSTmap), which can suppress the irrelevant background and noise features while retaining most of the temporal characteristics of the periodic physiological signals. Then we take pairwise MSTmaps as inputs to an autoencoder architecture with two encoders (one for physiological signals and the other for non-physiological information) and use a cross-verified scheme to obtain physiological features disentangled with the non-physiological features. The disentangled features are finally used for the joint prediction of multiple physiological signals like average HR values and rPPG signals. Comprehensive experiments on different large-scale public datasets of multiple physiological measurement tasks as well as the cross-database testing demonstrate the robustness of our approach.

Deep Learning Methods for Remote Heart Rate Measurement: A Review and Future Research Agenda

“…Another type of HR signal representation is the spatio-temporal map ( Figure 18 ) used for HR estimation in References [ 66 , 67 , 68 , 69 , 70 , 71 ]. Generally, an ROI selection step was involved in the construction of these spatio-temporal maps.…”

“…There are several methods proposed for handling these challenges. For example, the utilization of different HR signal representations, such as spectrum images [ 64 ] and spatio-temporal maps [ 66 , 67 , 68 , 69 , 70 , 71 ], as well as the use of attention mechanism [ 27 , 28 , 30 , 32 , 34 , 35 , 49 , 52 , 67 ], can deal with illumination variations and motion noise. STVEN [ 30 ] was designed to improve the robustness of HR measurement under video compression.…”

Personal‐specific gait recognition based on latent orthogonal feature space

“…Recently, in the study of human facial action unit detection, Niu et al have proposed a method based on local relationship learning with person-specific shape regularisation [19], which can effectively extract the pattern features of the same expression, reduce the influence of specific person, and improve the generalisation ability of the model. Niu et al also proposed a cross-verified feature disentangling strategy to disentangle the physiological features with non-physiological representations which can improve the robust ability of physiological measurement [20].…”

Turnip: Time-Series U-Net With Recurrence For NIR Imaging PPG

“…Recent work on iPPG estimation from RGB videos has used end-to-end DNNs [2,3,4,5,6]. In fewer cases, a modular approach is used [7,8]. Specifically, [7] uses an approach similar in spirit to ours.…”

“…In fewer cases, a modular approach is used [7,8]. Specifically, [7] uses an approach similar in spirit to ours. However, previous DNN methods can't handle NIR monochromatic videos and are not tested on real-world in-car data, which is the focus of our work.…”