A Deep Network for Automatic Video-Based Food Bite Detection

Abstract

Past research has now provided compelling evidence pointing towards correlations among individual eating styles and the development of (un)healthy eating patterns, obesity and other medical conditions. In this setting, an automatic, noninvasive food bite detection system can be a really useful tool in the hands of nutritionists, dietary experts and medical doctors in order to explore real-life eating behaviors and dietary habits. Unfortunately, the automatic detection of food bites can be challenging due to occlusions between hands and mouth, use of different kitchen utensils and personalized eating habits. On the other hand, although accurate, manual bite detection is time-consuming for the annotator, making it infeasible for large scale experimental deployments or real-life settings. To this regard, we propose a novel deep learning methodology that relies solely on human body and face motion data extracted from videos depicting people eating meals. The purpose is to develop a system that can accurately, robustly and automatically identify food bite instances, with the long-term goal to complement or even replace manual bite-annotation protocols currently in use. The experimental results on a large dataset reveal the superb classification performance of the proposed methodology on the task of bite detection and paves the way for additional research on automatic bite detection systems.

Validation of a Deep Learning System for the Full Automation of Bite and Meal Duration Analysis of Experimental Meal Videos

“…These features were then used for even more accurate modeling of hand, head, and mouth movements on a 2D plane. While our previous results already achieved a near-perfect agreement with manual video-clip annotations (κ = 0.879) [28], our current results improve the rate of agreement (current κ = 0.894), pointing to superb algorithm performance. On that level, RABiD outperforms all previous comparable efforts of video-based and wrist-worn accelerometer meal analyses.…”

“…Regarding the internal RABiD outcome validation, we managed to increase our previously published performance [28], using higher resolution images for the extraction of skeletal features. These features were then used for even more accurate modeling of hand, head, and mouth movements on a 2D plane.…”

“…Our current effort focuses on the development of a tool that improves the already existing and well-established video meal analysis methodology that can be used for the automation of the behavioral meal annotation process, on the level of meal duration and meal bites. Our proposed methodology is supported by recent advances in the fields of deep learning and image processing for skeletal feature extraction that led to the detection and analysis of specific movements in videos [28] and has been developed retrospectively on top of existing video meal datasets collected in controlled settings. Thus, we designed and deployed the "Rapid Automatic Bite Detection system" (RABiD), attempting to eliminate the need of human annotation for meal duration and total meal bites.…”

PeRsOnalised nutriTion for hEalthy livINg: The PROTEIN project

“…This is highlighted through the consortium's deployment of novel wearable sensors, their creation of new nutrition and physical activity ontologies within the system and of unique user profile models. In addition to this, the consortium hopes to further the advances in technological sensors and their application to nutrition through the development of the following: a bowel sound/movement 'smart band', within-meal chew/bite detection algorithms (Konstantinidis et al 2019(Konstantinidis et al , 2020, a food identification algorithm (Theodoridis et al 2020) and a food depth perception algorithm to calculate portion sizes (Graikos et al 2020). Given the increase in noncommunicable diseases and rising levels of obesity within the UK, modern technology applications such as the PROTEIN application could be useful to provide end users with easily accessible validated tools to make positive lifestyle changes.…”

Automatic Count of Bites and Chews From Videos of Eating Episodes

“…This model worked only to detect chewing events (classification) while not counting the number of chews. Recently, in [ 27 ] authors developed a deep network using features from both face and body to detect bite instances and count the number of bites, but did not count the number of chews in the eating episodes. A novel deep learning based algorithm named “Rapid Automatic Bite Detection” (RABid) [ 28 ] was developed by the authors that extracts and processes skeletal features from videos of eating episodes to measure meal duration and bites using long short-term memory (LSTM) network.…”