Automatic Count of Bites and Chews From Videos of Eating Episodes

Abstract

Methods for measuring of eating behavior (known as meal microstructure) often rely on manual annotation of bites, chews, and swallows on meal videos or wearable sensor signals. The manual annotation may be time consuming and erroneous, while wearable sensors may not capture every aspect of eating (e.g. chews only). The aim of this study is to develop a method to detect and count bites and chews automatically from meal videos. The method was developed on a dataset of 28 volunteers consuming unrestricted meals in the laboratory under video observation. First, the faces in the video (regions of interest, ROI) were detected using Faster R-CNN. Second, a pre-trained AlexNet was trained on the detected faces to classify images as a bite/no bite image. Third, the affine optical flow was applied in consecutively detected faces to find the rotational movement of the pixels in the ROIs. The number of chews in a meal video was counted by converting the 2-D images to a 1-D optical flow parameter and finding peaks. The developed bite and chew count algorithm was applied to 84 meal videos collected from 28 volunteers. A mean accuracy (±STD) of 85.4% (±6.3%) with respect to manual annotation was obtained for the number of bites and 88.9% (±7.4%) for the number of chews. The proposed method for an automatic bite and chew counting shows promising results that can be used as an alternative solution to manual annotation.

The potential of artificial intelligence in enhancing adult weight loss: a scoping review

“…Details on the use of AI for the self-regulation of weight lossrelated behaviours are shown in Table 2. Of the studies on enhancing self-monitoring, twenty-nine (43•9 %) were on eating behaviours (58)(59)(60)(61)(62)(63)(64)(65)(66)(67)(68)(69)(70)(71)(72)(73)(74)(75)(76) , seven (10•6 %) were on energy intake (34,(77)(78)(79)(80)(81)(82) , thirty-three (50 %) were on physical activity (26,(51)(52)(53)(54)(55)60,74, and nine (13•6 %) were on energy expenditure (83,85,92,(94)(95)(96)(97)100,101) . Of the studies on optimising goal setting, five were on optimising eating behaviour goals (e.g., eating at a certain time of the day and energy intake) (48,49,53) and six were on optimising physical activity...…”

“…The studies reported recognition accuracies ranging from 69•2 to 99•1 %. Machine recognition techniques used in the included studies were gesture (n 32) (51,56,58,(60)(61)(62)64,65,70,74,81,(83)(84)(85)(86)(87)(88)(89)(90)(91)(92)(93)(95)(96)(97)(98)(99)(100)(101)(102)(103)(104) , image (n 14) (34,63,(66)(67)(68)74,76,(78)(79)(80)88,93,94,101) , sound (n 7) (57)(58)(59)69,(71)(72)…”

“…Inertia: arm, hand-to-mouth, jaw movement (bite detection), physical activity Five force-sensitive resistors (100) Pressure Heart rate monitor (patch-type sensor or built-in smartphone/ smartwatch) (86) , photoplethysmography (107) , electrocardiogram (95) , ventilation sensor (96) Physiological: cardiovascular parameters mainly heart rate, ventilation Piezoelectric sensor (56,60,70,97) , electromyography (EMG) electrodes (temporal muscle activity during mastication) (58,60,64) Swallowing movement (jaw and skin), chewing cycle and food type Global positioning system (GPS) (52,87) Location Image recognition Video recorder (63,68) Images of food item/group/type Camera (34,66,67,74,78,79,93,94) Images of food size, shape, colour, portion and texture Sound recognition Ear microphone (worn on one's inner ear, outer ear, wrist, neck near the jaw/throat) (58,59,69,71,73) Chewing sound (bone-conducted food breakdown sounds) Speech recognition Microphone (smartphone in-build microphone) (75) Verbal food description or nutrition label Wireless signal recognition Signal receiver (109) Wireless signals neighbour (n 5) (86,85,88,91,103) , clustering (n 1) (106) and support vector machines (n 14) (60,68,…”

Determination of Chewing Count from Video Recordings Using Discrete Wavelet Decomposition and Low Pass Filtration

“…Similarly, Papapanagiotou et al [25] used convolutional neural networks to achieve a 98% accuracy and F1-score of 88.3%. Recently, Hossain et al [26] used a similar approach to detect faces, which they followed by transfer learning using AlexNet to classify images as bite or not, and used affine optical flow to detection rotational movement in the detect faces. They reported a mean accuracy of 88.9 ± 7.4% for chew count.…”

Short and Long-Term Innovations on Dietary Behavior Assessment and Coaching: Present Efforts and Vision of the Pride and Prejudice Consortium

“…Whereas cameras provide opportunities to improve dietary intake assessment, advanced video image analysis techniques offer new opportunities in terms of automated detection of eating behaviors such as emotion detection (adults and children), acceptance and rejection behavior of infants and automated oral processing behaviors such as chews and bites in different age groups [78,79]. Deep learning models can be based on extracted data from the video, for example facial landmarks, and training the model on annotated events and their time.…”