See the Glass Half Full: Reasoning About Liquid Containers, Their Volume and Content

Mottaghi, Roozbeh; Schenck, Connor; Fox, Dieter; Farhadi, Ali

doi:10.1109/iccv.2017.207

Cited by 54 publications

(54 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…28 The Containers Of liQuid contEnt (COQE) dataset contains images with liquids in containers but with no semantic or instance segmentation maps. 2 The COCO dataset 31 is the largest and most general image segmentation dataset and contains several subclasses of vessels, such as cups, jars, and bottles. We speculated that training with Vector-LabPics and related vessel classes from the COCO dataset could improve the accuracy of our nets.…”

Section: Training With Additional Datasetsmentioning

confidence: 99%

Computer Vision for Recognition of Materials and Vessels in Chemistry Lab Settings and the Vector-LabPics Dataset

Eppel

Xu²,

Bismuth³

et al. 2020

Preprint

View full text Add to dashboard Cite

This work presents a machine learning approach for computer vision-based recognition of materials inside vessels in a chemistry lab and other settings. In addition, we release a dataset associated with the training of the model for further model development. The task to learn is finding the region, boundaries, and category for each material phase and vessel in an image. Handling materials inside mostly transparent containers is the main activity performed by human and robotic chemists in the laboratory. Visual recognition of vessels and their content is essential for performing this task. Modern machine vision methods learn recognition tasks by using datasets containing a large number of annotated images. This work presents the Vector-LabPics dataset, which consists of 2187 images of materials within mostly transparent vessels in a chemistry lab and other general settings. The images are annotated for both the vessels and the individual material phases inside them, and each instance is assigned one or more classes (liquid, solid, foam, suspension, powder, …). The fill level, labels, corks, and parts of the vessel are also annotated. Several convolutional nets for semantic and instance segmentation were trained on this dataset. @ The trained neural networks achieved good accuracy in detecting and segmenting vessels and material phases, and in classifying liquids and solids, but relatively low accuracy in segmenting multiphase systems such as phase-separating liquids. * Equal Contributions. # Corresponding author. @ The dataset and models used for this work are available here .

show abstract

Section: Training With Additional Datasetsmentioning

confidence: 99%

Computer Vision for Recognition of Materials and Vessels in Chemistry Lab Settings and the Vector-LabPics Dataset

Eppel

Xu²,

Bismuth³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Morris and Kutulakos [6] look at reconstructing a refractive surface, but this requires a pattern placed underneath the liquid surface. Mottaghi et al [7] use deep learning to infer volume characteristics of containers. Neither of these papers consider the problem of tracking a liquid level and pouring to a specific height.…”

Section: Related Workmentioning

confidence: 99%

Accurate Pouring with an Autonomous Robot Using an RGB-D Camera

Burgard

2018

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

Robotic assistants in a home environment are expected to perform various complex tasks for their users. One particularly challenging task is pouring drinks into cups, which for successful completion, requires the detection and tracking of the liquid level during a pour to determine when to stop. In this paper, we present a novel approach to autonomous pouring that tracks the liquid level using an RGB-D camera and adapts the rate of pouring based on the liquid level feedback. We thoroughly evaluate our system on various types of liquids and under different conditions, conducting over 250 pours with a PR2 robot. The results demonstrate that our approach is able to pour liquids to a target height with an accuracy of a few millimeters.

show abstract

“…Alayrac et al [6] model the interaction between actions and objects in a discrete manner. Some methods further demonstrate that liquid amount can be estimated by combining semantic segmentation CNN and LSTM [34,7]. In contrast, our main goal is not to explicitly recognize environmental states.…”

Section: Related Workmentioning

confidence: 99%

“…Our single pouring sequence consists of pouring liquid from the source container with initial liquid amount α to target container with β amount of liquid. Similar to [7], we roughly divide the container states into discrete labels. In successful sequences, the demonstrator tries to fill target container with the liquid in the source container without spilling out any liquid.…”

Section: Datasetmentioning

confidence: 99%

Liquid Pouring Monitoring via Rich Sensory Inputs

Tie-jun

Lin

Wang

et al. 2018

Computer Vision – ECCV 2018

View full text Add to dashboard Cite

Humans have the amazing ability to perform very subtle manipulation task using a closed-loop control system with imprecise mechanics (i.e., our body parts) but rich sensory information (e.g., vision, tactile, etc.). In the closed-loop system, the ability to monitor the state of the task via rich sensory information is important but often less studied. In this work, we take liquid pouring as a concrete example and aim at learning to continuously monitor whether liquid pouring is successful (e.g., no spilling) or not via rich sensory inputs. We mimic humans' rich sensories using synchronized observation from a chest-mounted camera and a wrist-mounted IMU sensor. Given many success and failure demonstrations of liquid pouring, we train a hierarchical LSTM with late fusion for monitoring. To improve the robustness of the system, we propose two auxiliary tasks during training: inferring (1) the initial state of containers and (2) forecasting the one-step future 3D trajectory of the hand with an adversarial training procedure. These tasks encourage our method to learn representation sensitive to container states and how objects are manipulated in 3D. With these novel components, our method achieves ∼ 8% and ∼ 11% better monitoring accuracy than the baseline method without auxiliary tasks on unseen containers and unseen users respectively.

show abstract

See the Glass Half Full: Reasoning About Liquid Containers, Their Volume and Content

Cited by 54 publications

References 43 publications

Computer Vision for Recognition of Materials and Vessels in Chemistry Lab Settings and the Vector-LabPics Dataset

Computer Vision for Recognition of Materials and Vessels in Chemistry Lab Settings and the Vector-LabPics Dataset

Accurate Pouring with an Autonomous Robot Using an RGB-D Camera

Liquid Pouring Monitoring via Rich Sensory Inputs

Contact Info

Product

Resources

About