Refining the Fusion of Pepper Robot and Estimated Depth Maps Method for Improved 3D Perception

Abstract

As it is well known, some versions of the Pepper robot provide poor depth perception due to the lenses it has in front of the tridimensional sensor. In this paper, we present a method to improving that faulty 3D perception. Our proposal is based on a combination of the actual depth readings of Pepper and a deep learning-based monocular depth estimation. As shown, the combination of both of them provides a better 3D representation of the scene. In previous works we made an initial approximation of this fusion technique, but it had some drawbacks. In this paper we analyze the pros and cons of the Pepper readings, the monocular depth estimation method and our previous fusion method. Finally, we demonstrate that the proposed fusion method outperforms them all.

Point Clouds With Color: A Simple Open Library for Matching RGB and Depth Pixels from an Uncalibrated Stereo Pair

“…It is also interesting to mention the work of Bauer et al on improving the vision of the Pepper robot [14]. In their work, they seek to improve 3D perception by fusion method between the depth image raw data with the monocular depth prediction from the RGB image.…”

RoboCup@Home Education 2020 Best Performance: RoboBreizh, a modular approach

“…The first one computes visual information from 2D monocular cameras and 3D Depth sensors to build a 3D map. In [1], experiments have been conducted on Pepper's depth camera (the Asus Xtion) that lead to a Root Mean Square Error (RMSE) of 20.36mm for a 1m distance and 79.15mm for 3m distance. This error is found to stem from the robot's lenses [1].…”

“…In [1], experiments have been conducted on Pepper's depth camera (the Asus Xtion) that lead to a Root Mean Square Error (RMSE) of 20.36mm for a 1m distance and 79.15mm for 3m distance. This error is found to stem from the robot's lenses [1]. Other issues such as depth shadow or monocular depth estimation are also covered in this paper.…”

“…In our case, we experienced those issues by qualitative observations as shown in Figure 2. Although there are various calibration methods for this sensor, [24,1], they are complex and can not completely eliminate the error. Concerning the Laser-Based SLAM approach, Pepper's lasers are located in front, left and right of the robot with a VFOV of 40°and HFOV of 60°( Vertical/Horizontal Field of View).…”

Improving Visual Perception of a Social Robot for Controlled and In-the-wild Human-robot Interaction

“…with humans, social robots require robust visual perception [2], which remains challenging especially in real-world scenarios and public spaces with diverse user and environmental contexts [5,6]. This study aims to improve human perception capabilities of social robots, demonstrated on the Pepper humanoid robot as a representative commonly adopted in research and application [24,28,30].…”