Container Localisation and Mass Estimation with an RGB-D Camera

Abstract

In the research area of human-robot interactions, the automatic estimation of the mass of a container manipulated by a person leveraging only visual information is a challenging task. The main challenges consist of occlusions, different filling materials and lighting conditions. The mass of an object constitutes key information for the robot to correctly regulate the force required to grasp the container. We propose a single RGB-D camera-based method to locate a manipulated container and estimate its empty mass i.e., independently of the presence of the content. The method first automatically selects a number of candidate containers based on the distance with the fixed frontal view, then averages the mass predictions of a lightweight model to provide the final estimation. Results on the CORSMAL Containers Manipulation dataset show that the proposed method estimates empty container mass obtaining a score of 71.08% under different lighting or filling conditions.

The effects of selected object features on a pick-and-place task: A human multimodal dataset

“…The main insight is that a correct estimate of object properties would allow a robot to interact with a human more appropriately, especially when physical interactions are involved, for example, a handover. Researchers explored the usage of deep neural networks to estimate containers capacity, dimension, and weight while observing human manipulations with RGB-D (Wang et al 2022; Apicella et al 2022) or even simple RGB cameras (Pang et al 2021). However, a robot should also consider if an object requires particular care for being handled to avoid changing irreparably some of its properties.…”