Volumetric Food Quantification Using Computer Vision on a Depth-Sensing Smartphone: Preclinical Study

Abstract

Background Quantification of dietary intake is key to the prevention and management of numerous metabolic disorders. Conventional approaches are challenging, laborious, and lack accuracy. The recent advent of depth-sensing smartphones in conjunction with computer vision could facilitate reliable quantification of food intake. Objective The objective of this study was to evaluate the accuracy of a novel smartphone app combining depth-sensing hardware with computer vision to quantify meal macronutrient content using volumetry. Methods The app ran on a smartphone with a built-in depth sensor applying structured light (iPhone X). The app estimated weight, macronutrient (carbohydrate, protein, fat), and energy content of 48 randomly chosen meals (breakfasts, cooked meals, snacks) encompassing 128 food items. The reference weight was generated by weighing individual food items using a precision scale. The study endpoints were (1) error of estimated meal weight, (2) error of estimated meal macronutrient content and energy content, (3) segmentation performance, and (4) processing time. Results In both absolute and relative terms, the mean (SD) absolute errors of the app’s estimates were 35.1 g (42.8 g; relative absolute error: 14.0% [12.2%]) for weight; 5.5 g (5.1 g; relative absolute error: 14.8% [10.9%]) for carbohydrate content; 1.3 g (1.7 g; relative absolute error: 12.3% [12.8%]) for fat content; 2.4 g (5.6 g; relative absolute error: 13.0% [13.8%]) for protein content; and 41.2 kcal (42.5 kcal; relative absolute error: 12.7% [10.8%]) for energy content. Although estimation accuracy was not affected by the viewing angle, the type of meal mattered, with slightly worse performance for cooked meals than for breakfasts and snacks. Segmentation adjustment was required for 7 of the 128 items. Mean (SD) processing time across all meals was 22.9 seconds (8.6 seconds). Conclusions This study evaluated the accuracy of a novel smartphone app with an integrated depth-sensing camera and found highly accurate volume estimation across a broad range of food items. In addition, the system demonstrated high segmentation performance and low processing time, highlighting its usability.

Digital Solutions to Diagnose and Manage Postbariatric Hypoglycemia

“…GoFood replaced GoCARB's SMV with a neural network ( 48 ). Today, also single-view methods (e.g., combined with depth data) are available ( 49 ). In this case, only a single image instead of a video or reference map is required.…”

“…An example of the technical workflow of such a single-view method is displayed in Figure 2 . Systems that additionally provide automated quantification of macronutrients were shown to provide accurate macronutrient estimations with absolute errors of 14% for weight and 15% for carbohydrate content ( 49 ). It is of note that the reported estimate is better than currently reported carbohydrate counting skills in the type 1 diabetes population ( 50 ) and was recently demonstrated to be comparable to dieticians' estimates ( 48 , 51 ).…”

Enhancing Food Intake Tracking in Long-Term Care with Automated Food Imaging and Nutrient Intake Tracking (AFINI-T) Technology

“…In a LTC setting, semantic segmentation is required when measuring food and nutrient intake across a plate as many residents do not consume their entire portion. For food volume estimation, template matching has been popular [29][30][31][32][33][34][35] . However, within the context of LTC, the same food can take on various shapes (e.g., banana in peel versus sliced banana, versus pureed banana) and 47% of the LTC population receives modified texture foods 36 which limits utility of template-matching in this context.…”

A Novel Approach to Dining Bowl Reconstruction for Image-Based Food Volume Estimation

“…Although food recognition has been extensively studied using deep learning techniques [11][12][13][14][15][16][17], estimating food volume from images remains a challenging problem [9][10][11]18]. Several sensor-based approaches have been reported [18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33]. A special imaging sensor called a depth sensor has been used to produce depth on a per-pixel basis from which food volume can be estimated [22][23][24][25][26].…”

“…Several sensor-based approaches have been reported [18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33]. A special imaging sensor called a depth sensor has been used to produce depth on a per-pixel basis from which food volume can be estimated [22][23][24][25][26]. Another effective approach uses a pair of stereo cameras separated by a distance.…”