Comparison of the Validity and Generalizability of Machine Learning Algorithms for the Prediction of Energy Expenditure: Validation Study

Abstract

Background Accurate solutions for the estimation of physical activity and energy expenditure at scale are needed for a range of medical and health research fields. Machine learning techniques show promise in research-grade accelerometers, and some evidence indicates that these techniques can be applied to more scalable commercial devices. Objective This study aims to test the validity and out-of-sample generalizability of algorithms for the prediction of energy expenditure in several wearables (ie, Fitbit Charge 2, ActiGraph GT3-x, SenseWear Armband Mini, and Polar H7) using two laboratory data sets comprising different activities. Methods Two laboratory studies (study 1: n=59, age 44.4 years, weight 75.7 kg; study 2: n=30, age=31.9 years, weight=70.6 kg), in which adult participants performed a sequential lab-based activity protocol consisting of resting, household, ambulatory, and nonambulatory tasks, were combined in this study. In both studies, accelerometer and physiological data were collected from the wearables alongside energy expenditure using indirect calorimetry. Three regression algorithms were used to predict metabolic equivalents (METs; ie, random forest, gradient boosting, and neural networks), and five classification algorithms (ie, k-nearest neighbor, support vector machine, random forest, gradient boosting, and neural networks) were used for physical activity intensity classification as sedentary, light, or moderate to vigorous. Algorithms were evaluated using leave-one-subject-out cross-validations and out-of-sample validations. Results The root mean square error (RMSE) was lowest for gradient boosting applied to SenseWear and Polar H7 data (0.91 METs), and in the classification task, gradient boost applied to SenseWear and Polar H7 was the most accurate (85.5%). Fitbit models achieved an RMSE of 1.36 METs and 78.2% accuracy for classification. Errors tended to increase in out-of-sample validations with the SenseWear neural network achieving RMSE values of 1.22 METs in the regression tasks and the SenseWear gradient boost and random forest achieving an accuracy of 80% in classification tasks. Conclusions Algorithms trained on combined data sets demonstrated high predictive accuracy, with a tendency for superior performance of random forests and gradient boosting for most but not all wearable devices. Predictions were poorer in the between-study validations, which creates uncertainty regarding the generalizability of the tested algorithms.

Do eating behavior traits predict energy intake and body mass index? A systematic review and meta‐analysis

“…The triple recall bias can also be extended to other psychological EBT, which raises concerns over their construct validity and questions whether EBT measuring what they were designed to measure? Interestingly, new developments in real time digital tracking of energy balance behaviors, 148 , 149 , 150 , 151 , 152 digital ecological momentary assessments and mobile video recording 153 could be a new avenue for more ecological assessments of EI to capture EBT.…”

Safely and autonomously cutting meat with a collaborative robot arm

“…Generalizability (i.e., effective performance across multiple tasks and settings) is commonly identified as an important target in robotic control literature 45 , and a critical goal in the meat processing context 46 . In tests on generalizability in other contexts, RF approaches show strong performance among various machine learning algorithms tested 47 . However, superficial data segmentation for training and testing is acknowledged as a methodological pitfall likely to overestimate model performance 48 .…”

Diet composition and energy intake in humans

“…The methodological constraints to such designs are considerable but not necessarily unsurmountable. New technologies, approaches to data analytics and wearable sensors may improve our understanding of food intake [118][119][120][121], patterns of physical activity and EE [122][123][124][125][126] and changes in body composition (and hence EB) in the environmental contexts were they occur, e.g. [127][128][129] to provide comparative measures across experimental environments (laboratory and real world) and to aggregate data across time windows relevant to behavioural trajectories that affect longer-term EB.…”