Visual Aware Hierarchy Based Food Recognition

Abstract

Food recognition is one of the most important components in image-based dietary assessment. However, due to the different complexity level of food images and inter-class similarity of food categories, it is challenging for an image-based food recognition system to achieve high accuracy for a variety of publicly available datasets. In this work, we propose a new two-step food recognition system that includes food localization and hierarchical food classification using Convolutional Neural Networks (CNNs) as the backbone architecture. The food localization step is based on an implementation of the Faster R-CNN method to identify food regions. In the food classification step, visually similar food categories can be clustered together automatically to generate a hierarchical structure that represents the semantic visual relations among food categories, then a multi-task CNN model is proposed to perform the classification task based on the visual aware hierarchical structure. Since the size and quality of dataset is a key component of data driven methods, we introduce a new food image dataset, VIPER-FoodNet (VFN) dataset, consists of 82 food categories with 15k images based on the most commonly consumed foods in the United States. A semi-automatic crowdsourcing tool is used to provide the ground-truth information for this dataset including food object bounding boxes and food object labels. Experimental results demonstrate that our system can significantly improve both classification and recognition performance on 4 publicly available datasets and the new VFN dataset.

Improving Dietary Assessment Via Integrated Hierarchy Food Classification

“…Recently, modern deep learning techniques [6]- [8] have enabled advances of image-based dietary assessment methods [9]- [17] which take only food or eating scene images These authors contributed equally. Fig.…”

“…1: An example of misclassification. Comparison between flat CNN (non-hierarchical), visual based hierarchy [9] and our proposed method. Energy and nutrients ground truth are obtained from the USDA FNDDS database [18] using 100g food sample.…”

“…Therefore, under the circumstance of noticeable classification error rate, a better mistake will enable a more accurate match of the food to a similar food within a food and nutrient database to improve the estimation of energy or dietary components within the food compared with a ground truth. Figure 1 shows an example of a misclassified food image where our proposed method gives the best prediction result in terms of energy and macronutrients values compared with flat (nonhierarchical) CNN and visual based hierarchy method [9]. The concept of hierarchy has been studied in deep learning based image classification [19], [20], which showed promising results.…”

“…The concept of hierarchy has been studied in deep learning based image classification [19], [20], which showed promising results. For food image classification, both [9], [10] proposed hierarchy based multi-task learning to improve the classification accuracy. However, a number of challenges remain to be addressed.…”

“…In addition, food names are not unique for different geographic regions and are deeply rooted in local culture, making it challenging to build an adaptive semantic hierarchical structure. Recently, Mao et al [9] proposed to build a visual hierarchical structure by extracting the visual features from CNNs and then measure the similarity score between food categories, which can be easily applied to different food image datasets. However, both methods achieved better mistakes measured by using visual information only, which is not sufficient for assessing dietary intake as illustrated in Figure 1.…”