Asking for Help with the Right Question by Predicting Human Visual Performance

Abstract

Abstract-In this paper, we consider robotic surveillance tasks that involve visual perception. The robot has a limited access to a remote operator to ask for help. However, humans may not be able to accomplish the visual task in many scenarios, depending on the sensory input. In this paper, we propose a machine learning-based approach that allows the robot to probabilistically predict human visual performance for any visual input. Based on this prediction, we then present a methodology that allows the robot to properly optimize its field decisions in terms of when to ask for help, when to sense more, and when to rely on itself. The proposed approach enables the robot to ask the right questions, only querying the operator with the sensory inputs for which humans have a high chance of success. Furthermore, it allows it to autonomously locate the areas that need more sensing. We test the proposed predictor on a large validation set and show Normalized Mean Square Error of 0.0199, as well as a reduction of about an order of magnitude in error as compared to the state-of-the-art. We then run a number of robotic surveillance experiments on our campus as well as a larger-scale evaluation with real data/human feedback in a simulation environment. The results showcase the efficacy of our approach, indicating a considerable increase in the success rate of human queries (a few folds in several cases) and the overall performance (30%-41% increase in success rate).

Modeling Human Helpfulness with Individual and Contextual Factors for Robot Planning

“…One work models how humans interpret natural language help requests, and then generates the requests likely to result in desired human actions [28]. Another work models the accuracy of humans given a visual input and uses that model to determine when and how to ask for help [7].…”

Adviser Networks: Learning What Question to Ask for Human-In-The-Loop Viewpoint Estimation

“…Finally, there has been research on improving human-computer interaction and human-robot interaction through asking better questions in tasks such as navigation [24], task learning [25], and question-answering [26]. While this work shares our goal of learning to best leverage the human input, they differ in problem scope, task representation, and assumed knowledge of the task.…”

“…While this work shares our goal of learning to best leverage the human input, they differ in problem scope, task representation, and assumed knowledge of the task. Cai and Mostofi train a convolutional neural network to predict how easy it is for a human to detect an object in a specific image [24]. The trained network is then used by a robot as it navigates around the environment to determine when it would be useful to query a remote operator about its current surroundings.…”