For nutrition practitioners and researchers, assessing dietary intake of children and adults with a high level of accuracy continues to be a challenge. Developments in mobile technologies have created a role for images in the assessment of dietary intake. The objective of this review was to examine peer-reviewed published papers covering development, evaluation and/or validation of image-assisted or image-based dietary assessment methods from December 2013 to January 2016. Images taken with handheld devices or wearable cameras have been used to assist traditional dietary assessment methods for portion size estimations made by dietitians (image-assisted methods). Image-assisted approaches can supplement either dietary records or 24-h dietary recalls. In recent years, image-based approaches integrating application technology for mobile devices have been developed (image-based methods). Image-based approaches aim at capturing all eating occasions by images as the primary record of dietary intake, and therefore follow the methodology of food records. The present paper reviews several image-assisted and image-based methods, their benefits and challenges; followed by details on an image-based mobile food record. Mobile technology offers a wide range of feasible options for dietary assessment, which are easier to incorporate into daily routines. The presented studies illustrate that image-assisted methods can improve the accuracy of conventional dietary assessment methods by adding eating occasion detail via pictures captured by an individual (dynamic images). All of the studies reduced underreporting with the help of images compared with results with traditional assessment methods. Studies with larger sample sizes are needed to better delineate attributes with regards to age of user, degree of error and cost.

Mobile telephones with an integrated camera can provide a unique mechanism for collecting dietary information that reduces burden on record keepers. Objectives for this study were: (1) to test whether participant's proficiency with the mobile telephone food record (mpFR) improved after training and repeated use, and (2) to measure changes in perceptions regarding use of the mpFR after training and repeated use. Seventy-eight adolescents (26 males, 52 females) ages 11–18 y were recruited to use the mpFR for one or two meals. Proficiency with the mpFR was defined as capturing a useful image for image analysis and self-reported ease of use. Positive changes in perceptions regarding use of the mpFR were assumed to equate to potentially improved proficiency with the mpFR. Participants received instruction for using the mpFR prior to their first meal, and captured an image of their meals before and after eating. Following the first meal, participants took part in an interactive session where they received additional training on capturing images in various snacking situations and responded to questions about user preferences. Changes in the participants' abilities to capture useful images and perceptions about the usability of the mpFR were examined using McNemar, Wilcoxon rank-sum test, and paired t-test. After using the mpFR, the majority of participants (79%) agreed that the software was easy to use. Eleven percent of participants agreed taking images before snacking would be easy. After additional training, the percent increased significantly to 32% (p<.0001). For taking images after snacking, there was also improvement (21% before training and 43% after, p<.0001). Adolescents readily adopt new technologies; however the mpFR design needs to accommodate the lifestyles of its users to ensure useful images and continuous use. Further, these results suggest that additional training in using a new technology may improve the accuracy among users.

BackgroundThe development of a mobile telephone food record has the potential to ameliorate much of the burden associated with current methods of dietary assessment. When using the mobile telephone food record, respondents capture an image of their foods and beverages before and after eating. Methods of image analysis and volume estimation allow for automatic identification and volume estimation of foods. To obtain a suitable image, all foods and beverages and a fiducial marker must be included in the image.ObjectiveTo evaluate a defined set of skills among adolescents and adults when using the mobile telephone food record to capture images and to compare the perceptions and preferences between adults and adolescents regarding their use of the mobile telephone food record.MethodsWe recruited 135 volunteers (78 adolescents, 57 adults) to use the mobile telephone food record for one or two meals under controlled conditions. Volunteers received instruction for using the mobile telephone food record prior to their first meal, captured images of foods and beverages before and after eating, and participated in a feedback session. We used chi-square for comparisons of the set of skills, preferences, and perceptions between the adults and adolescents, and McNemar test for comparisons within the adolescents and adults.ResultsAdults were more likely than adolescents to include all foods and beverages in the before and after images, but both age groups had difficulty including the entire fiducial marker. Compared with adolescents, significantly more adults had to capture more than one image before (38% vs 58%, P = .03) and after (25% vs 50%, P = .008) meal session 1 to obtain a suitable image. Despite being less efficient when using the mobile telephone food record, adults were more likely than adolescents to perceive remembering to capture images as easy (P < .001).ConclusionsA majority of both age groups were able to follow the defined set of skills; however, adults were less efficient when using the mobile telephone food record. Additional interactive training will likely be necessary for all users to provide extra practice in capturing images before entering a free-living situation. These results will inform age-specific development of the mobile telephone food record that may translate to a more accurate method of dietary assessment.

There is a growing concern about chronic diseases and other health problems related to diet including obesity and cancer. The need to accurately measure diet (what foods a person consumes) becomes imperative. Dietary intake provides valuable insights for mounting intervention programs for prevention of chronic diseases. Measuring accurate dietary intake is considered to be an open research problem in the nutrition and health fields. In this paper, we describe a novel mobile telephone food record that will provide an accurate account of daily food and nutrient intake. Our approach includes the use of image analysis tools for identification and quantification of food that is consumed at a meal. Images obtained before and after foods are eaten are used to estimate the amount and type of food consumed. The mobile device provides a unique vehicle for collecting dietary information that reduces the burden on respondents that are obtained using more classical approaches for dietary assessment. We describe our approach to image analysis that includes the segmentation of food items, features used to identify foods, a method for automatic portion estimation, and our overall system architecture for collecting the food intake information.

We propose a method for dietary assessment to automatically identify and locate food in a variety of images captured during controlled and natural eating events. Two concepts are combined to achieve this: a set of segmented objects can be partitioned into perceptually similar object classes based on global and local features; and perceptually similar object classes can be used to assess the accuracy of image segmentation. These ideas are implemented by generating multiple segmentations of an image to select stable segmentations based on the classifier’s confidence score assigned to each segmented image region. Automatic segmented regions are classified using a multichannel feature classification system. For each segmented region, multiple feature spaces are formed. Feature vectors in each of the feature spaces are individually classified. The final decision is obtained by combining class decisions from individual feature spaces using decision rules. We show improved accuracy of segmenting food images with classifier feedback.

The mobile Food Record (mFR) is an image-based dietary assessment method for mobile devices. The study primary aim was to test the accuracy of the mFR by comparing reported energy intake (rEI) to total energy expenditure (TEE) using the doubly labeled water (DLW) method. Usability of the mFR was assessed by questionnaires before and after the study. Participants were 45 community dwelling men and women, 21–65 years. They were provided pack-out meals and snacks and encouraged to supplement with usual foods and beverages not provided. After being dosed with DLW, participants were instructed to record all eating occasions over a 7.5 days period using the mFR. Three trained analysts estimated rEI from the images sent to a secure server. rEI and TEE correlated significantly (Spearman correlation coefficient of 0.58, p < 0.0001). The mean percentage of underreporting below the lower 95% confidence interval of the ratio of rEI to TEE was 12% for men (standard deviation (SD) ± 11%) and 10% for women (SD ± 10%). The results demonstrate the accuracy of the mFR is comparable to traditional dietary records and other image-based methods. No systematic biases could be found. The mFR was received well by the participants and usability was rated as easy.

Dietary intake provides valuable insights for mounting intervention programs for prevention of disease. With growing concern for adolescent obesity, the need to accurately measure diet becomes imperative. Assessment among adolescents is problematic as this group has irregular eating patterns and have less enthusiasm for recording food intake. Preliminary studies among adolescents suggest that innovative use of technology may improve the accuracy of diet information from young people. In this paper, we propose a novel food record method using a mobile device that will provide an accurate account of daily food and nutrient intake among adolescents. Our approach includes the use of image analysis tools for identification and quantification of food consumption. Images obtained before and after food is consumed can be used to estimate the diet of an individual. In this paper we describe our initial results and indicate the potential of the proposed system.

Obtaining accurate food portion estimation automatically is challenging since the processes of food preparation and consumption impose large variations on food shapes and appearances. The aim of this paper was to estimate the food energy numeric value from eating occasion images captured using the mobile food record. To model the characteristics of food energy distribution in an eating scene, a new concept of “food energy distribution” was introduced. The mapping of a food image to its energy distribution was learned using Generative Adversarial Network (GAN) architecture. Food energy was estimated from the image based on the energy distribution image predicted by GAN. The proposed method was validated on a set of food images collected from a 7-day dietary study among 45 community-dwelling men and women between 21–65 years. The ground truth food energy was obtained from pre-weighed foods provided to the participants. The predicted food energy values using our end-to-end energy estimation system was compared to the ground truth food energy values. The average error in the estimated energy was 209 kcal per eating occasion. These results show promise for improving accuracy of image-based dietary assessment.