Assessing Individual Dietary Intake in Food Sharing Scenarios with a 360 Camera and Deep Learning

Abstract

A novel vision-based approach for estimating individual dietary intake in food sharing scenarios is proposed in this paper, which incorporates food detection, face recognition and hand tracking techniques. The method is validated using panoramic videos which capture subjects' eating episodes. The results demonstrate that the proposed approach is able to reliably estimate food intake of each individual as well as the food eating sequence. To identify the food items ingested by the subject, a transfer learning approach is designed. 4, 200 food images with segmentation masks, among which 1, 500 are newly annotated, are used to fine-tune the deep neural network for the targeted food intake application. In addition, a method for associating detected hands with subjects is developed and the outcomes of face recognition are refined to enable the quantification of individual dietary intake in communal eating settings.

Counting Bites and Recognizing Consumed Food from Videos for Passive Dietary Monitoring

“…As using recipes facilitates dietary intake assessments, there is growing interest in developing cross-modal food image and recipe retrieval [44]- [49]. Recently, efforts to assess individual dietary intake in communal eating scenarios have also been made with the use of a 360 camera [50], [51]. Fine-grained food ingredient recognition has also been studied to enhance general food recognition [52], or to perform recipe retrieval [53], but so far studies have only been carried out in recognizing ingredients from food images rather than from dietary intake videos.…”

OREBA: A Dataset for Objectively Recognizing Eating Behavior and Associated Intake

“…Before the application of deep learning architectures, the traditional approach in this field reduced the dimensionality of the raw sensor data by extracting handcrafted features based on expert knowledge. Deep learning methods have been explored to detect individual intake gestures with inertial sensor data since 2017 [3] and with video data since 2018 [2], [4], [5], whereby large amounts of labeled examples are leveraged to let algorithms learn the features automatically. The most widely used approach in this space builds on convolutional neural networks (CNN) and long short-term memory (LSTM) models [16], however gated recurrent unit (GRU) models have also been applied, especially in the context of activity recognition in daily living [17].…”

“…However, collecting, synchronizing, and labeling data of eating occasions is a work-intensive process. Hence, there is a need for more public datasets to reduce barriers for researchers to create new machine learning models, and to objectively compare the performance of existing approaches [4], [5].…”

“…A related dataset is iHEARu-EAT[19], however we did not include it here since it does not focus on intake events.2 See http://www.skleinberg.org/data.html 3 See http://cecas.clemson.edu/~ahoover/cafeteria/. Recordings with missing annotations are excluded here 4. See https://mug.ee.auth.gr/intake-cycle-detection/…”

Single-Stage Intake Gesture Detection Using CTC Loss and Extended Prefix Beam Search

“…Early work on the Clemson dataset, established in 2012, used threshold values for detection from inertial data [4]. More recent developments include the use of machine learning to learn features automatically [5] and learning from video, which has become more practical with emerging spherical camera technology [6] [7]. Research on the OREBA dataset showed that frontal video data can exhibit even higher accuracies in detecting eating gestures than inertial data [8].…”