Learning to Guide Human Attention on Mobile Telepresence Robots with 360° Vision

Chandan, Kishan; Albertson, Jack; Zhang, Xiaohan; Zhang, Xiaoyang; Liu, Yao; Zhang, Shiqi

doi:10.1109/iros51168.2021.9636607

Cited by 7 publications

(2 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In all the trials, the robot is provided with a domain map, a dataset of scene graphs, and a pretrained network for generating scene graphs. It receives 360-degree images of the environment as the input, where we leveraged our previous research on 360-degree robot vision [43], and the output is the location of the target object. Our first baseline method is a naive POMDP planner [38] (with uniform prior belief) where the robot action policy solely depends on the model of the world.…”

Section: Methodsmentioning

confidence: 99%

Reasoning with Scene Graphs for Robot Planning under Partial Observability

Amiri¹,

Chandan²,

Zhang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Robot planning in partially observable domains is difficult, because a robot needs to estimate the current state and plan actions at the same time. When the domain includes many objects, reasoning about the objects and their relationships makes robot planning even more difficult. In this paper, we develop an algorithm called scene analysis for robot planning (SARP) that enables robots to reason with visual contextual information toward achieving long-term goals under uncertainty. SARP constructs scene graphs, a factored representation of objects and their relations, using images captured from different positions, and reasons with them to enable context-aware robot planning under partial observability. Experiments have been conducted using multiple 3D environments in simulation, and a dataset collected by a real robot. In comparison to standard robot planning and scene analysis methods, in a target search domain, SARP improves both efficiency and accuracy in task completion. Supplementary material can be found at https://tinyurl.com/sarp22

show abstract

Section: Methodsmentioning

confidence: 99%

Reasoning with Scene Graphs for Robot Planning under Partial Observability

Amiri¹,

Chandan²,

Zhang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Ikei et al [49] proposed an all-around stereo live camera system in which the orientations of two 360 • camera lenses are fixed in world coordinates so that these cameras move with the position of the viewer's eyes. Chandan et al [50] developed a mobile telepresence robot (MTR) and equipped it with the ability to analyze 360 • scenes; it was capable of directing the user's attention to the area of interest, allowing the user to efficiently and accurately locate the target object in the remote environment.…”

Section: Movable Telepresence Robots (Mtrs)mentioning

confidence: 99%

Viewpoint-Controllable Telepresence: A Robotic-Arm-Based Mixed-Reality Telecollaboration System

Luo

Weng

Hao

et al. 2023

Sensors

View full text Add to dashboard Cite

In mixed-reality (MR) telecollaboration, the local environment is remotely presented to a remote user wearing a virtual reality (VR) head-mounted display (HMD) via a video capture device. However, remote users frequently face challenges in naturally and actively manipulating their viewpoints. In this paper, we propose a telepresence system with viewpoint control, which involves a robotic arm equipped with a stereo camera in the local environment. This system enables remote users to actively and flexibly observe the local environment by moving their heads to manipulate the robotic arm. Additionally, to solve the problem of the limited field of view of the stereo camera and limited movement range of the robotic arm, we propose a 3D reconstruction method combined with a stereo video field-of-view enhancement technique to guide remote users to move within the movement range of the robotic arm and provide them with a larger range of local environment perception. Finally, a mixed-reality telecollaboration prototype was built, and two user studies were conducted to evaluate the overall system. User study A evaluated the interaction efficiency, system usability, workload, copresence, and user satisfaction of our system from the remote user’s perspective, and the results showed that our system can effectively improve the interaction efficiency while achieving a better user experience than two traditional view-sharing techniques based on 360 video and based on the local user’s first-person view. User study B evaluated our MR telecollaboration system prototype from both the remote-user side and the local-user side as a whole, providing directions and suggestions for the subsequent design and improvement of our mixed-reality telecollaboration system.

show abstract