Energy expenditure estimation using visual and inertial sensors

Abstract

Deriving a person's energy expenditure accurately forms the foundation for tracking physical activity levels across many health and lifestyle monitoring tasks. In this study, the authors present a method for estimating calorific expenditure from combined visual and accelerometer sensors by way of an RGB-Depth camera and a wearable inertial sensor. The proposed individual-independent framework fuses information from both modalities which leads to improved estimates beyond the accuracy of single modality and manual metabolic equivalents of task (MET) lookup table based methods. For evaluation, the authors introduce a new dataset called SPHERE_RGBD + Inertial_calorie, for which visual and inertial data are simultaneously obtained with indirect calorimetry ground truth measurements based on gas exchange. Experiments show that the fusion of visual and inertial data reduces the estimation error by 8 and 18% compared with the use of visual only and inertial sensor only, respectively, and by 33% compared with a MET-based approach. The authors conclude from their results that the proposed approach is suitable for home monitoring in a controlled environment.

Calorific Expenditure Estimation Using Deep Convolutional Network Features

“…However, studies on energy expenditure using visual sensors are still strictly limited, possibly due to the absence of labeled data, which is difficult to obtain -not least given the ethical requirements necessary to obtain it. Our work stems from the SPHERE project [33], which has created the circumstances to generate and release a suitable dataset for us and the vision community to explore [27,29,2]. The proposed method is built on several relevant subject areas in computer vision.…”

“…The recent work in [27] introduced a vision-based framework for estimating calorific expenditure in a home environment. This method has become a baseline method in the area [28,29]. An extension enables the estimation of physical activity intensity levels in real-time [28].…”

“…However, the light-weight features extracted from bounding boxes can only estimate calorific expenditure coarsely. To improve on this setup, a further step was taken to fuse visual sensor data with accelerometer data in order to reduce the estimation error [29]. In this paper, we consider the use of a single visual sensor only and compare a deep architecture against the baseline method.…”

“…Among these, tri-axial accelerometers are the most broadly used inertial sensors [8]. Whilst appropriate to inferring coarse categorisations of activity intensity levels, based on this data alone it is difficult to produce precise calorific expenditure values [29].…”

“…the SPHERE-calorie dataset [29,2], which contains accurate ground truth collected with gas-exchange measurements obtained from a calorimeter and more than ten hours of visual data. We evaluate the performance of our proposed method on this dataset and compare it against the state-ofthe-art that uses hand-crafted visual features [27].…”