Computer Vision-Based Carbohydrate Estimation for Type 1 Patients With Diabetes Using Smartphones

Abstract

Individuals with type 1 diabetes (T1D) use prandial insulin doses to balance the effects of a meal.1 The meal's carbohydrate (CHO) content is a key factor in determining the optimal dose and maintaining normal blood glucose levels. According to clinical studies in insulin-depended children and adolescents, an error of ± 10 grams in CHO counting does not affect postprandial glycemia, 2 yet an error of ± 20 grams substantially impairs the postprandial control. Diabetics have to attend courses on CHO counting based on empirical rules; however even well trained patients face difficulties, according to studies. [4][5][6] In Brazeau et al, 4 the average error in CHO counting by 50 T1D adults was 15.4 ± 7.8 grams, while in Bishop et al 5 only 11 of 48 adolescent T1D patients estimated daily CHO with an error less than 10 grams. In a study involving children with T1D and their caregivers, CHO estimations were inaccurate by at least ± 15 grams for 27% of the meals. 6 The global spread of diabetes-together with the proven inability of diabetics to accurately assess their diet-raised the urgent need for automated tools and services that will support T1D patients with CHO counting. Recently, the ubiquity of smartphones with enhanced capabilities, along with recent advances in computer vision, has permitted the development of image analysis applications for automatic assessment of food intake. 7 The input of such a system consists of a few images or a short video of the upcoming meal, as captured by the user's smartphone. The data are then processed-either locally or on a remote server-to automatically assess the Abstract Background: Individuals with type 1 diabetes (T1D) have to count the carbohydrates (CHOs) of their meal to estimate the prandial insulin dose needed to compensate for the meal's effect on blood glucose levels. CHO counting is very challenging but also crucial, since an error of 20 grams can substantially impair postprandial control. Method: The GoCARB system is a smartphone application designed to support T1D patients with CHO counting of nonpacked foods. In a typical scenario, the user places a reference card next to the dish and acquires 2 images with his/her smartphone. From these images, the plate is detected and the different food items on the plate are automatically segmented and recognized, while their 3D shape is reconstructed. Finally, the food volumes are calculated and the CHO content is estimated by combining the previous results and using the USDA nutritional database. Results: To evaluate the proposed system, a set of 24 multi-food dishes was used. For each dish, 3 pairs of images were taken and for each pair, the system was applied 4 times. The mean absolute percentage error in CHO estimation was 10 ± 12%, which led to a mean absolute error of 6 ± 8 CHO grams for normal-sized dishes. Conclusion:The laboratory experiments demonstrated the feasibility of the GoCARB prototype system since the error was below the initial goal of 20 grams. However, further improvements and evaluation are needed pr...

Two-View 3D Reconstruction for Food Volume Estimation

“…The system outperformed existing solutions having overall mean absolute percentage error ranging from 8.2% to 9.8% on the different datasets, with over 90% of volume estimate errors under 20%, all in roughly 5.5 seconds. The proposed method is currently part of a carbohydrate counting system for individuals with diabetes [38], and may also be used for general food assessment. It uses segmentation to calculate portion sizes, and, by using food recognition, the nutrient profile is built up.…”

GoCARB in the Context of an Artificial Pancreas

“…The current version of the system is a prototype that considers 9 broad food classes found in common central European meals. The system's mean absolute percentage error was 10 ± 12% (or mean absolute error of 6 ± 8 CHO grams) in a laboratory setup 15 and 28 ± 20.5% (or mean absolute error of 13.16 ± 10.16 CHO grams) when being used by T1D patients in a preclinical study. 19 …”

“…[15][16][17][18] The application requires as input a pair of images of the upcoming meal with a credit card-sized reference object placed next to it. The first image is acquired horizontally above the dish and the second at 20-30 degrees from the vertical axis crossing the center of the dish.…”

A Digital Ecosystem of Diabetes Data and Technology

“…60 Other smartphone apps also include photobased meal logging, and looking beyond meal images as a simple visual record, research is ongoing toward automated annotation of meal content from photos. 61,62 Someday, individuals may even carry miniaturized spectroscopy-based devices capable of quantitatively characterizing the content of everything they eat. 63,64 Meal information collected by these tools, if sufficiently accurate, could be employed by a next-generation bolus calculator (in this case, the burden of active meal-tracking with a camera or other device may be offset by alleviation of the mental math and guesswork currently associated with determining an insulin dose) or by a closed-loop artificial pancreas system to enhance the proactive delivery of insulin at meal times.…”