ResSeg: Residual encoder-decoder convolutional neural network for food segmentation

Abstract

This paper presents the implementation and evaluation of different convolutional neural network architectures focused on food segmentation. To perform this task, it is proposed the recognition of 6 categories, among which are the main food groups (protein, grains, fruit, and vegetables) and two additional groups, rice and drink or juice. In addition, to make the recognition more complex, it is decided to test the networks with food dishes already started, i.e. during different moments, from its serving to its finishing, in order to verify the capability to see when there is no more food on the plate. Finally, a comparison is made between the two best resulting networks, a SegNet with architecture VGG-16 and a network proposed in this work, called Residual Segmentation Convolutional Neural Network or ResSeg, with which accuracies greater than 90% and interception-over-union greater than 75% were obtained. This demonstrates the ability, not only of SegNet architectures for food segmentation, but the use of residual layers to improve the contour of the segmentation and segmentation of complex distribution or initiated of food dishes, opening the field of application of this type of networks to be implemented in feeding assistants or in automated restaurants, including also for dietary control for the amount of food consumed.

Food Region Extraction Based on Saliency Detection Model

“…We employed SegNet [38], which was applied to the food extraction in Refs. [14,15], as the food extraction using low-training-cost DNN. SegNet is trained by UNIMIB2016 [39], which includes 1017 annotated food images available on the Internet.…”

“…Refs. [12][13][14][15] proposed food extraction using DNN, requiring pixel-wise annotated food images as a training dataset. The food regions include various shapes, colors, and textures [12,16].…”

“…[4]. Although the food extraction using DNN [12][13][14][15] can recognize food type at pixel level, it requires the pixel-wise annotated training dataset, which is extremely laborious and costly to be prepared [21].…”

Fuzzy encoding with hybrid pooling for visual dictionary in food recognition

“…Image description is a process to extract the visual content of foods. In particular, local feature is more suitable to represent food features as the properties of local features that capture minuscule parts of the food beside its robustness towards illumination, scale, rotation, and orientation which made it capable to deal with the cluttered appearance of foods [1,5,6]. The interest points that were detected and described have produced a high volume and diverse features that require the features to be transformed into another more simplified representation by using certain feature encoding technique.…”

Algorithm of detection, classification and gripping of occluded objects by CNN techniques and Haar classifiers

“…Besides CNN, other artificial intelligence methods have been developed for the classification of patterns, such as the fast R-CNN [15] and the DAG-CNN [16,17], in the first case is former a stage of extraction of a Regions of Interest (ROIs) that is responsible for of detecting desired elements in the input image, extracting them and entering them into a CNN for classification, as explained in [18], while the last consists of a branched structure where each branch contains a sequence of convolutional layers whose filters vary of dimension, to extract characteristics of greater and smaller size of the input image, and in the end to unify the results to give a classification, as indicated in [19].…”