Food Volume Estimation Based on Deep Learning View Synthesis from a Single Depth Map

Abstract

An objective dietary assessment system can help users to understand their dietary behavior and enable targeted interventions to address underlying health problems. To accurately quantify dietary intake, measurement of the portion size or food volume is required. For volume estimation, previous research studies mostly focused on using model-based or stereo-based approaches which rely on manual intervention or require users to capture multiple frames from different viewing angles which can be tedious. In this paper, a view synthesis approach based on deep learning is proposed to reconstruct 3D point clouds of food items and estimate the volume from a single depth image. A distinct neural network is designed to use a depth image from one viewing angle to predict another depth image captured from the corresponding opposite viewing angle. The whole 3D point cloud map is then reconstructed by fusing the initial data points with the synthesized points of the object items through the proposed point cloud completion and Iterative Closest Point (ICP) algorithms. Furthermore, a database with depth images of food object items captured from different viewing angles is constructed with image rendering and used to validate the proposed neural network. The methodology is then evaluated by comparing the volume estimated by the synthesized 3D point cloud with the ground truth volume of the object items.

Current Developments in Digital Quantitative Volume Estimation for the Optimisation of Dietary Assessment

“…The increasing availability and accuracy of depth sensing technology in recent years has enabled the ability to detect form and shape and to establish the scale of objects beyond the limits of red/green/blue (RGB) images. Depth cameras such as the Microsoft Kinect or Intel RealSense have been used in fields of engineering and medicine, and are now being integrated into the field of nutrition [107][108][109][110]. Modern smartphones are also being outfitted with multiple cameras that also allow for stereo vision and depth perception.…”

“…Modern smartphones are also being outfitted with multiple cameras that also allow for stereo vision and depth perception. Given the ability to detect distance accurately, these modern methods using depth perception have enabled researchers to circumvent the need for a fiducial marker entirely [107][108][109]111].…”

“…nutrition [107][108][109][110]. Modern smartphones are also being outfitted with multiple cameras that also allow for stereo vision and depth perception.…”

“…A depth map of an object can be determined by various methods such as stereoscopic vision, structured light, depth sensing or deep learning methods. As mentioned earlier, these optical approaches are able to determine the relative distance of objects within the picture allowing for the 3-dimensional surface area of food to be calculated in pictures without the need for a fiducial marker [108,111]. The height is defined based on distance from the relative plane to the identified Y-axis coordinates and the volume can be calculated respectively.…”

Estimating Weight of Unknown Objects Using Active Thermography

“…Volume estimation may rely on off-the-shelf techniques to contactlessly measure the volume of a sample. Existing methods, such as [22], can be applied to measure the volume of unknown samples. This is considered beyond the scope of this research and thus the current study focuses on estimating the density of the sampled material.…”