Synthetic Data for Multi-Parameter Camera-Based Physiological Sensing

Abstract

Synthetic data is a powerful tool in training data hungry deep learning algorithms. However, to date, camerabased physiological sensing has not taken full advantage of these techniques. In this work, we leverage a high-fidelity synthetics pipeline for generating videos of faces with faithful blood flow and breathing patterns. We present systematic experiments showing how physiologically-grounded synthetic data can be used in training camera-based multi-parameter cardiopulmonary sensing. We provide empirical evidence that heart and breathing rate measurement accuracy increases with the number of synthetic avatars in the training set. Furthermore, training with avatars with darker skin types leads to better overall performance than training with avatars with lighter skin types. Finally, we discuss the opportunities that synthetics present in the domain of camera-based physiological sensing and limitations that need to be overcome.

Camera Measurement of Physiological Vital Signs

“…Synthetics pipelines have the advantage of allowing simulation of many different combinations of appearance types, contexts and physiological states example high heart rates or arrhythmia states for which it may be difficult to create to gather examples in a lab. Research has shown that greater and greater numbers of avatars in a synthetic training set can continue to boost performance up to a point [90]. However this is early work and there remains a "sim-to-real" gap in performance of these systems, models trained purely on synthetic data do not generalize perfectly to real videos.…”