The CORSMAL Benchmark for the Prediction of the Properties of Containers

Abstract

The contactless estimation of the weight of a container and the amount of its content manipulated by a person are key pre-requisites for safe human-to-robot handovers. However, opaqueness and transparencies of the container and the content, and variability of materials, shapes, and sizes, make this problem challenging. In this paper, we present a range of methods and an open framework to benchmark acoustic and visual perception for the estimation of the capacity of a container, and the type, mass, and amount of its content. The framework includes a dataset, specific tasks and performance measures. We conduct a fair and in-depth comparative analysis of methods that used this framework and audio-only or vision-only baselines designed from related works. Based on this analysis, we can conclude that audioonly and audio-visual classifiers are suitable for the estimation of the type and amount of the content using different types of convolutional neural networks, combined with either recurrent neural networks or a majority voting strategy, whereas computer vision methods are suitable to determine the capacity of the container using regression and geometric approaches. Classifying the content type and level using only audio achieves a weighted average F1-score up to 81% and 97%, respectively. Estimating the container capacity with vision-only approaches and filling mass with audio-visual approaches, multi-stage algorithms reaches up to 65% weighted average capacity and mass scores. These results show that there is still room of improvement for the design of future methods that will be ranked and compared on the individual leaderboards provided by our open framework.

The effects of selected object features on a pick-and-place task: A human multimodal dataset

“…Each subject performed 80 trials, ensuring 300 interactions for each of the 4 objects. Such participants' numerosity is generally higher than that of other object manipulation datasets in the literature, as highlighted by the review of Huang et al (2016) and, as recent examples, the works by Kratzer et al (2021); Xompero et al (2022) and Fan et al (2023).…”

“…The definition of carefulness is not straightforward, since many factors may solicit cautious actions: from the physical context where the action takes place, to the properties of the object involved such as its fragility, precarious balance, or content about to spill (Lastrico et al 2022). Given the difficulty in framing this feature, not many studies explicitly refer to it, although it is addressed, for example, in the handling of filled containers (Xompero et al 2022; Mottaghi et al 2017; Pang et al 2021; Sanchez-Matilla et al 2020; Yu et al 2015). From the same perspective, we define carefulness as the modulation of arm motion that minimizes liquid spilling during the manipulation of containers.…”

“…The literature presents different examples of datasets for the study of object manipulations (Huang et al 2016), most of the time focusing on the interaction between the humans and the objects in specific applications, for example, activities of daily living (Huang and Sun 2019), kitchen-related actions (Tenorth et al 2009; Stein and Mckenna 2013; Nicora et al 2020), or handovers (Carfì et al 2019). A recently published dataset, the CORSMAL Container Manipulation by Xompero et al (2022), collects actions such as pouring and handover initiations with containers with various shapes, materials, and content recorded with RGB-D cameras and microphones. However, the datasets currently available are more oriented to classical object recognition problems, proposing a high variability in the shapes and sizes of the objects examined, considering fewer sensors or a limited pool of participants; moreover, no one takes into consideration nor is aimed at modeling carefulness.…”