Learning to Guide Human Attention on Mobile Telepresence Robots with 360 Vision

Abstract

Mobile telepresence robots (MTRs) allow people to navigate and interact with a remote environment that is in a place other than the person's true location. Thanks to the recent advances in 360 • vision, many MTRs are now equipped with an all-degree visual perception capability. However, people's visual field horizontally spans only about 120 • of the visual field captured by the robot. To bridge this observability gap toward human-MTR shared autonomy, we have developed a framework, called GHAL360, to enable the MTR to learn a goal-oriented policy from reinforcements for guiding human attention using visual indicators. Three telepresence environments were constructed using datasets that are extracted from Matterport3D and collected from a real robot respectively. Experimental results show that GHAL360 outperformed the baselines from the literature in the efficiency of a human-MTR team completing target search tasks. A demo video is available: https://youtu.be/aGbTxCGJSDM

Equirectangular Image Data Detection, Segmentation and Classification of Varying Sized Traffic Signs: A Comparison of Deep Learning Methods

“…There are many benefits of 360-degree field of view capture for ADAS [ 4 ], and it is widely used in other applications such as robotics perception [ 5 ], surveillance [ 6 ], virtual reality [ 7 ] and other applications [ 8 ]. A camera with a 360-degree field of view projected to a plane or that approximates the covering of a sphere before projection to a plane is defined as omnidirectional [ 9 ].…”

“…There are three main types of omnidirectional camera systems: dioptric, catadioptric and polydioptric [ 9 ], where a spherical structure of surrounds is typically produced before being projected to a plane using techniques such as cylindrical or equirectangular projections. In the case of [ 5 , 6 , 7 ], equirectangular was used, as it keeps horizontal and vertical 360 degree fields of view. Current image datasets such as The German Traffic Sign Detection Benchmark [ 10 ], German Traffic Sign Recognition Benchmark [ 11 ], TRMSD [ 12 ], COCO [ 13 ], ImageNet [ 14 ], Cityscapes [ 15 ], PASCAL VOC, [ 16 ] PASCAL-Context [ 17 ], and KITTI [ 18 ] do not include omnidirectional images.…”