An intelligent crowd-worker selection approach for reliable content labeling of food images

Rabbi, Mashfiqui; Costa, Jean; Okeke, Fabian; Schachere, Max; Zhang, Mi; Choudhury, Tanzeem

doi:10.1145/2811780.2811955

Cited by 9 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address the need for constructing large-scale food datasets with food images that provide comprehensive ground truth information, a solution is to merge food images sourced from the internet or from nutrition studies with manual annotation from crowd-sourcing platforms. Amazon Mechanical Turk (AMT) has been used for food image collection and annotation tasks [19,20], although AMT is not tailored for building large food image datasets efficiently with proper labels. This inefficiency may be partly attributed to its high cost and dependency on crowdsource workers unfamiliar with the context in which the data were collected (eg, restaurant food vs homemade meal).…”

Section: Technology-enabled Domains For Measuring Calorie and Macronumentioning

confidence: 99%

Counting Bites With Bits: Expert Workshop Addressing Calorie and Macronutrient Intake Monitoring

et al. 2019

View full text Add to dashboard Cite

BackgroundConventional diet assessment approaches such as the 24-hour self-reported recall are burdensome, suffer from recall bias, and are inaccurate in estimating energy intake. Wearable sensor technology, coupled with advanced algorithms, is increasingly showing promise in its ability to capture behaviors that provide useful information for estimating calorie and macronutrient intake.ObjectiveThis paper aimed to summarize current technological approaches to monitoring energy intake on the basis of expert opinion from a workshop panel and to make recommendations to advance technology and algorithms to improve estimation of energy expenditure.MethodsA 1-day invitational workshop sponsored by the National Science Foundation was held at Northwestern University. A total of 30 participants, including population health researchers, engineers, and intervention developers, from 6 universities and the National Institutes of Health participated in a panel discussing the state of evidence with regard to monitoring calorie intake and eating behaviors.ResultsCalorie monitoring using technological approaches can be characterized into 3 domains: (1) image-based sensing (eg, wearable and smartphone-based cameras combined with machine learning algorithms); (2) eating action unit (EAU) sensors (eg, to measure feeding gesture and chewing rate); and (3) biochemical measures (eg, serum and plasma metabolite concentrations). We discussed how each domain functions, provided examples of promising solutions, and highlighted potential challenges and opportunities in each domain. Image-based sensor research requires improved ground truth (context and known information about the foods), accurate food image segmentation and recognition algorithms, and reliable methods of estimating portion size. EAU-based domain research is limited by the understanding of when their systems (device and inference algorithm) succeed and fail, need for privacy-protecting methods of capturing ground truth, and uncertainty in food categorization. Although an exciting novel technology, the challenges of biochemical sensing range from a lack of adaptability to environmental effects (eg, temperature change) and mechanical impact, instability of wearable sensor performance over time, and single-use design.ConclusionsConventional approaches to calorie monitoring rely predominantly on self-reports. These approaches can gain contextual information from image-based and EAU-based domains that can map automatically captured food images to a food database and detect proxies that correlate with food volume and caloric intake. Although the continued development of advanced machine learning techniques will advance the accuracy of such wearables, biochemical sensing provides an electrochemical analysis of sweat using soft bioelectronics on human skin, enabling noninvasive measures of chemical compounds that provide insight into the digestive and endocrine systems. Future computing-based researchers should focus on reducing the burden of wearable sensors, aligning data across m...

show abstract

Section: Technology-enabled Domains For Measuring Calorie and Macronumentioning

confidence: 99%

Counting Bites With Bits: Expert Workshop Addressing Calorie and Macronutrient Intake Monitoring

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Annotation tools and platforms, such as LabelMe [30] and Amazon Mechanical Turk (AMT) [1], are widely used to perform image annotation such as creating bounding boxes to indicate the pixel locations and label of the objects within. In [29], a machine learning method is proposed to identify high performing worker on AMT in order to achieve high accuracy. In [32], a deep neural network is developed for food object detection to remove noisy images which do not contain foods during pre-processing, alleviating the burden placed on human annotators.…”

Section: Introductionmentioning

confidence: 99%

An Integrated System for Mobile Image-Based Dietary Assessment

Shao¹,

Han²,

He³

et al. 2021

Preprint

View full text Add to dashboard Cite

Accurate assessment of dietary intake requires improved tools to overcome limitations of current methods including user burden and measurement error. Emerging technologies such as imagebased approaches using advanced machine learning techniques coupled with widely available mobile devices present new opportunities to improve the accuracy of dietary assessment that is costeffective, convenient and timely. However, the quality and quantity of datasets are essential for achieving good performance for automated image analysis. Building a large image dataset with high quality groundtruth annotation is a challenging problem, especially for food images as the associated nutrition information needs to be provided or verified by trained dietitians with domain knowledge. In this paper, we present the design and development of a mobile, image-based dietary assessment system to capture and analyze dietary intake, which has been deployed in both controlled-feeding and community-dwelling dietary studies. Our system is capable of collecting high quality food images in naturalistic settings and provides groundtruth annotations for developing new computational approaches.

show abstract

“…For example, in [33], AMT is used as the crowd-sourcing service to select relevant images and add bounding boxes to the selected food images. In [35], a machine learning method is proposed to identify high performing worker on AMT in order to achieve high accuracy. However, the use of AMT is time consuming and expensive.…”

Section: Introductionmentioning

confidence: 99%

Semi-Automatic Crowdsourcing Tool for Online Food Image Collection and Annotation

Shao

Mao

Zhu

2019

2019 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Assessing dietary intake accurately remains an open and challenging research problem. In recent years, image-based approaches have been developed to automatically estimate food intake by capturing eat occasions with mobile devices and wearable cameras. To build a reliable machine-learning models that can automatically map pixels to calories, successful imagebased systems need large collections of food images with high quality groundtruth labels to improve the learned models. In this paper, we introduce a semi-automatic system for online food image collection and annotation. Our system consists of a web crawler, an automatic food detection method and a web-based crowdsoucing tool. The web crawler is used to download large sets of online food images based on the given food labels. Since not all retrieved images contain foods, we introduce an automatic food detection method to remove irrelevant images. We designed a web-based crowdsourcing tool to assist the crowd or human annotators to locate and label all the foods in the images. The proposed semi-automatic online food image collection system can be used to build large food image datasets with groundtruth labels efficiently from scratch.

show abstract

An intelligent crowd-worker selection approach for reliable content labeling of food images

Cited by 9 publications

References 6 publications

Counting Bites With Bits: Expert Workshop Addressing Calorie and Macronutrient Intake Monitoring

Counting Bites With Bits: Expert Workshop Addressing Calorie and Macronutrient Intake Monitoring

An Integrated System for Mobile Image-Based Dietary Assessment

Semi-Automatic Crowdsourcing Tool for Online Food Image Collection and Annotation

Contact Info

Product

Resources

About