Face2PPG: An Unsupervised Pipeline for Blood Volume Pulse Extraction From Faces

Abstract

Photoplethysmography (PPG) signals have become a key technology in many fields, such as medicine, well-being, or sports. Our work proposes a set of pipelines to extract remote PPG signals (rPPG) from the face robustly, reliably, and configurable. We identify and evaluate the possible choices in the critical steps of unsupervised rPPG methodologies. We assess a state-of-the-art processing pipeline in six different datasets, incorporating important corrections in the methodology that ensure reproducible and fair comparisons. In addition, we extend the pipeline by proposing three novel ideas; 1) a new method to stabilize the detected face based on a rigid mesh normalization; 2) a new method to dynamically select the different regions in the face that provide the best raw signals, and 3) a new RGB to rPPG transformation method, called Orthogonal Matrix Image Transformation (OMIT) based on QR decomposition, that increases robustness against compression artifacts. We show that all three changes introduce noticeable improvements in retrieving rPPG signals from faces, obtaining state-of-the-art results compared with unsupervised, non-learning-based methodologies and, in some databases, very close to supervised, learning-based methods. We perform a comparative study to quantify the contribution of each proposed idea. In addition, we depict a series of observations that could help in future implementations.

iBVP Dataset: RGB-Thermal rPPG Dataset with High Resolution Signal Quality Labels

“…The majority of existing rPPG datasets have ground-truth PPG signals acquired using a finger probe or wrist watch, making it challenging to match the morphology as well as phase [49] of ground-truth and extracted rPPG signals from facial video frames. The morphology of PPG waveform is site-dependent [50] and therefore it is crucial to acquire ground-truth signals for rPPG from a site that is closest to the face.…”

Improving Remote Photoplethysmography Performance through Deep-Learning-Based Real-Time Skin Segmentation Network

“…In recent years, studies have explored the challenges of rPPG, such as changes in ambient light and the movement of subjects. Representative rPPG extraction methods include principal component analysis (PCA) [6], CHROM [7], POS [8], and OMIT [9]. The contributions of this study are as follows:…”

“…OMIT [9] is a powerful rPPG algorithm for compression artifacts that is based on QR decomposition. Two methods are used for ROI selection: U-Net-based skin segmentation and patch selection based on facial landmark coordinates.…”

“…Second, glasses, hair, and beards can contaminate rPPG signals. To reduce the effect of these factors, studies have simply narrowed the face area or divided it into smaller areas without dividing the skin [9,11]. However, this approach reduces the ROI, thereby making rPPG more sensitive to noise [7].…”

Graph connection Laplacian allows for enhanced outcomes of consumer camera based photoplethysmography imaging

“…Starting from traditional spatial averaging [12], which primarily aims to increase the signal-to-noise ratio, different methods are used, differing from each other in their complexity or approach to signal extraction. At this point, we can mention the R-G algorithm [23] combining the red and green channels in the case of PPGI via an RGB camera, or the POS (Plane-Orthogonal-to-Skin) algorithm [24], which works with all three layers, or others such as, also in this case, the historically sorted G [12], which works only with the green layer or greyscale data, the aforementioned G-R [23], PCA [25], ICA [26], CHROM [27], PBV [28], 2SR [29], nonlinear-type time-frequency analysis like synchrosqueezing transform [30], the aforementioned POS [24] or Face2PPG [31]. Another approach is deep or machine learning methods [32], [33], which bring additional potential for feature extraction, classification and understanding of complex links within PPG signals.…”