Multi-Task Learning for Calorie Prediction on a Novel Large-Scale Recipe Dataset Enriched with Nutritional Information

Ruede, Robin; Heusser, Verena; Frank, Lukas; Roitberg, Alina; Haurilet, Monica; Stiefelhagen, Rainer

doi:10.1109/icpr48806.2021.9412839

Cited by 13 publications

(9 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Energy expenditure prediction. Visual estimation of calorie values has been mainly investigated in food image analysis (i.e., tracking the amount of caloric intake) [34,40,48]. Energy expenditure induced by physical activity is mostly studied from an egocentric perspective featuring data from wearable sensors, such as accelometors or heart rate monitors [2,5,18,26,37,39,41,55,64], with a recent survey provided in [70].…”

Section: Related Workmentioning

confidence: 99%

Should I take a walk? Estimating Energy Expenditure from Video Data

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We explore the problem of automatically inferring the amount of kilocalories used by human during physical activity from his/her video observation. To study this underresearched task, we introduce Vid2Burn -an omni-source benchmark for estimating caloric expenditure from video data featuring both, high-and low-intensity activities for which we derive energy expenditure annotations based on models established in medical literature.In practice, a training set would only cover a certain amount of activity types, and it is important to validate, if the model indeed captures the essence of energy expenditure, (e.g., how many and which muscles are involved and how intense they work) instead of memorizing fixed values of specific activity categories seen during training. Ideally, the models should look beyond such category-specific biases and regress the caloric cost in videos depicting activity categories not explicitly present during training. With this property in mind, Vid2Burn is accompanied with a crosscategory benchmark, where the task is to regress caloric expenditure for types of physical activities not present during training. An extensive evaluation of state-of-the-art approaches for video recognition modified for the energy expenditure estimation task demonstrates the difficulty of this problem, especially for new activity types at test-time, marking a new research direction. Dataset and code are available at https://github.com/KPeng9510/ Vid2Burn.

show abstract

Section: Related Workmentioning

confidence: 99%

Should I take a walk? Estimating Energy Expenditure from Video Data

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…There are still some limitations for multi-stage VBDA. First, these methods need to be defined and optimized individually at each stage, while their accuracy remains a challenge and has an impact on subsequent operations (Ruede et al, 2021). Second, the method mainly relies on pixel-wise annotations of large amounts of data and additional information in the food images, such as volume.…”

Section: End-to-end Architecturementioning

confidence: 99%

“…The results proved that the multi-task CNN outperformed the single-task CNN. Ruede et al (2021) designed the network from a different perspective. They introduced a framework for retrieving nutritional information of recipes by matching ingredients and their mass to a nutrient database using phrase embedding.…”

Section: End-to-end Approachesmentioning

confidence: 99%

“…Many researches were done using this approach. For example, Ruede et al (2021) collected the dataset from a German recipe website, namely the pic2kcal dataset, which contains cooking instructions, ingredients and images. Situju et al (2019) collected large datasets of food images with calorie and salinity annotations based on cooking food websites.…”

Section: Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

A review on vision-based analysis for automatic dietary assessment

Wang,

Min,

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: Maintaining a healthy diet is vital to avoid health-related issues, e.g., undernutrition, obesity and many non-communicable diseases. An indispensable part of the health diet is dietary assessment. Traditional manual recording methods are burdensome and contain substantial biases and errors. Recent advances in Artificial Intelligence, especially computer vision technologies, have made it possible to develop automatic dietary assessment solutions, which are more convenient, less time-consuming and even more accurate to monitor daily food intake. Scope and approach: This review presents one unified Vision-Based Dietary Assessment (VBDA) framework, which generally consists of three stages: food image analysis, volume estimation and nutrient derivation. Vision-based food analysis methods, including food recognition, detection and segmentation, are systematically summarized, and methods of volume estimation and nutrient derivation are also given. The prosperity of deep learning makes VBDA gradually move to an end-to-end implementation, which applies food images to a single network to directly estimate the nutrition. The recently proposed end-to-end methods are also discussed. We further analyze existing dietary assessment datasets, indicating that one large-scale benchmark is urgently needed, and finally highlight key challenges and future trends for VBDA. Key findings and conclusions: After thorough exploration, we find that multi-task end-to-end deep learning approaches are one important trend of VBDA. Despite considerable research progress, many challenges remain for VBDA due to the meal complexity. We also provide the latest ideas for future development of VBDA, e.g., fine-grained food analysis and accurate volume estimation. This survey aims to encourage researchers to propose more practical solutions for VBDA.

show abstract

“…Another set of techniques utilize the abundance of web data for recipes to help understand images of dishes [4,14,17,16]. These approaches leverage web labels to train embedding models that retrieve similar recipes, or use the attribute labels for classification.…”

Section: Related Workmentioning

confidence: 99%

Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food

Thames

Karpur

Norris

et al. 2021

Preprint

View full text Add to dashboard Cite

Understanding the nutritional content of food from visual data is a challenging computer vision problem, with the potential to have a positive and widespread impact on public health. Studies in this area are limited to existing datasets in the field that lack sufficient diversity or labels required for training models with nutritional understanding capability. We introduce Nutrition5k, a novel dataset of 5k diverse, real world food dishes with corresponding video streams, depth images, component weights, and high accuracy nutritional content annotation. We demonstrate the potential of this dataset by training a computer vision algorithm capable of predicting the caloric and macronutrient values of a complex, real world dish at an accuracy that outperforms professional nutritionists. Further we present a baseline for incorporating depth sensor data to improve nutrition predictions. We will publicly release Nutrition5k in the hope that it will accelerate innovation in the space of nutritional understanding.

show abstract

Multi-Task Learning for Calorie Prediction on a Novel Large-Scale Recipe Dataset Enriched with Nutritional Information

Cited by 13 publications

References 20 publications

Should I take a walk? Estimating Energy Expenditure from Video Data

Should I take a walk? Estimating Energy Expenditure from Video Data

A review on vision-based analysis for automatic dietary assessment

Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food

Contact Info

Product

Resources

About