Analysis of CNN-based remote-PPG to understand limitations and sensitivities

Abstract

Deep learning based on Convolutional Neural Network (CNN) has shown promising results in various vision-based applications, recently also in camera-based vital signs monitoring. The CNN-based Photoplethysmography (PPG) extraction has, so far, been focused on performance rather than understanding. In this paper, we try to answer four questions with experiments aiming at improving our understanding of this methodology as it gains popularity. We conclude that the network exploits the blood absorption variation to extract the physiological signals, and that the choice and parameters (phase, spectral content, etc.) of the reference-signal may be more critical than anticipated. The availability of multiple convolutional kernels is necessary for CNN to arrive at a flexible channel combination through the spatial operation, but may not provide the same motion-robustness as a multi-site measurement using knowledge-based PPG extraction. Finally, we conclude that the PPG-related prior knowledge is still helpful for the CNN-based PPG extraction. Consequently, we recommend further investigation of hybrid CNN-based methods to include prior knowledge in their design. IntroductionRemote Photoplethysmography (remote-PPG) is a contactless way to measure human cardiovascular activity by measuring the reflection variations of the skin registered by a video camera [1]. Over the last decade, various remote-PPG methods [2-7] have been proposed for PPG-signal extraction. The methods differ in their choice of assumptions [2-5] and use of handcrafted features (e.g. projected color features of CHROM [4] and POS [5]), while these choices affect their robustness with respect to illumination variations and subject motion. Related workRecently, the success of deep Convolutional Neural Network (CNN) methods that automatically learn relevant features from images/videos in various applications has inspired researchers to attempt CNN-based remote-PPG extraction [8][9][10][11][12]. Chen and McDuff [8] proposed a convolutional attention network consisting of two parallel models to extract the PPG signal from a video. The first model is a classical "appearance model" [13] that learns to find the skin regionof-interest (RoI), while the second parallel path fed with DC-normalized frame-differences from the RoI learns to extract the PPG signal, using a finger oximeter-derived signal as a reference. In [8], the second model is referred to as a "motion model", but we prefer to use the term "normalized frame difference model", since our work shall prove that it exploits the blood absorption variation rather than the skin motion as suggested by [8]. SynRhythm [9] is a general-to-specific transfer learning method. The authors directly convert the spatial-temporal features into heart rate based on the pre-trained network [14]. HR-CNN [10] consists of the extractor CNN and the HR-estimator CNN with different loss functions to predict the heart rate, rather than the PPG signal. PhysNet [11] is a 3D CNN which learns the temporal and spatial context features of f...

Few-shot pulse wave contour classification based on multi-scale feature extraction

“…On the premise of retaining the structure of the pre-processing layer and the reasoning layer, as well as the network parameters, the network layer is changed for comparison experiments. Zhan 34 et al extracted features through CNN for classification. Liu 35 et al extracted PWC timing features through parallel CNN structures, including signal segment features within a period and multi-period features representing cycle relationships; Ghosh 36 et al extracted PWC features based on LSTM to predict systolic and diastolic blood pressure.…”

“…Cardiovascular-disease classification experiments were carried out on PPG dataset and CNBP dataset. To evaluate the classification performance of the multi-scale feature-extraction model, three different classification methods including single neural network model 34 , parallel neural network model 35 , and LSTM model 36 were introduced for comparison. Comparative studies show that the multi-scale feature-extraction model outperforms the other classification methods in terms of identification accuracy, stability, and sensitivity, and the multi-scale feature-extraction model consumes less time for training.…”

“…On the premise of retaining the structure of the pre-processing layer and the reasoning layer, as well as the network parameters, the network layer is changed for comparison experiments. Zhan 34 36 et al extracted PWC features based on LSTM to predict systolic and diastolic blood pressure. The training process of different network models is shown in Figure 7.…”

Long Short-Term Memory Deep-Filter in Remote Photoplethysmography

“…End-to-end approaches based on deep learning have also been used recently [7,22,6,4,32]. One of the main advantages of these CNN-based measurements is that it allows achieving good results without the need for the designer to analyze the problem in depth [33]. In addition, it is no longer necessary to use a pipeline-based framework where regions of interest (ROI) are first detected and tracked over frames, RGB channels are then combined to estimate the pulse signal, which is filtered and analyzed to extract physiological parameters such as heart rate or respiration rate.…”

“…As a consequence, CNN-based approaches are less prone to error propagation in the pipeline. However, as noted in [33], recent works in this field have focused on performance rather than understanding. Consequently, it is often hard to predict the limitations of the system, and it is well known that the training dataset used is critical.…”