A General Framework for Shared Control in Robot Teleoperation with Force and Visual Feedback

Abstract

In the last decade, the topic of human robot interaction has received increasing interest from research and industry, as robots must now interface with human users to accomplish complex tasks. In this scenario, robotics engineers are required to take the human component into account in the robot design and control. This is especially true in telerobotics, where interaction with the user plays an important role in the controlled system stability. By means of a thorough analysis and practical experiments, this contribution aims at giving a concrete idea of the aspects that need to be considered in the design of a complete control framework for teleoperated systems, that are able to seamlessly integrate with a human operator through shared control.

Still Not Solved: A Call for Renewed Focus on User-Centered Teleoperation Interfaces

“…The term, situation awareness, emerged from aviation psychology to describe the pilot’s understanding of tactical flight operations ( Durso and Gronlund, 1999 ), but it is applicable broadly to any cognitive activity and information processing, including teleoperation ( Durso and Gronlund, 1999 ; Yanco and Drury, 2004a ; Endsley, 2015 ). Situation awareness, in teleoperation, is an operator’s ability to perceive, understand, and reason about the environment around them and around their remote robot ( Endsley, 1988 ; Rakita et al, 2017 ; Rakita et al, 2018 ; Nicolis, 2020 ), which requires the operator to process a large amount of robot sensor data from the remote environment in real-time ( Goodrich et al, 2013 ).…”

“…A general trend is to process more raw data for the operation, and then create a display of those results, that is, easy for the operator to understand and reason about. For example, interfaces can highlight points of interest in ways that naturally draw the operator’s interest ( Rea et al, 2017b ), leverage people’s existing knowledge to summarize an off-screen’s object’s position and distance ( Seo et al, 2017b ), or combine multiple mediums like sound or haptics [multi-modal interfaces such as ( Hacinecipoglu et al, 2013 ; Suarez Fernandez et al, 2016 ; Nicolis, 2020 ; Seo et al, 2020 )]. Making robot sensor visualizations is not always obvious as sensors may detect qualities that are hard to visualize with a screen or speakers [such as haptic and force data ( Reveleau et al, 2015 ; Nicolis, 2020 )].…”

“…For example, interfaces can highlight points of interest in ways that naturally draw the operator’s interest ( Rea et al, 2017b ), leverage people’s existing knowledge to summarize an off-screen’s object’s position and distance ( Seo et al, 2017b ), or combine multiple mediums like sound or haptics [multi-modal interfaces such as ( Hacinecipoglu et al, 2013 ; Suarez Fernandez et al, 2016 ; Nicolis, 2020 ; Seo et al, 2020 )]. Making robot sensor visualizations is not always obvious as sensors may detect qualities that are hard to visualize with a screen or speakers [such as haptic and force data ( Reveleau et al, 2015 ; Nicolis, 2020 )]. Interfaces like those described here—and many others ( Nielsen et al, 2007 ; Yang et al, 2015 ; Seo et al, 2017a )—that fuse and interpret sensor data are essentially performing situation awareness processing for the operator, instead of having the operator analyze data or separate visualizations to come to conclusions themselves.…”

“…Historically, this has been almost no automation, with operators manually controlling each motor [e.g., early arms seen in ( Singh et al, 2013 ) or remote-controlled vehicles]. However, in recent years, research has increasingly focused on introducing more semi-autonomous behaviors, enabling simple inputs to result in complex behaviors ( Singh et al, 2013 ; Herlant et al, 2016 ; Kent et al, 2017 ; Li et al, 2017 ; Rakita et al, 2017 ; Rakita et al, 2019a ; Nicolis, 2020 ; Young and Peschel, 2020 ). In addition, high levels of automation are vital to assist teleoperators in managing multiple robots while performing their tasks ( Squire and Parasuraman, 2010 ; Wong and Seet, 2017 ).…”

“…User-centered work, in contrast, started from the abilities and needs of the operator and built interfaces with them in mind. Showcasing the potential of this approach, we highlight classic and recent examples for human-centered and non-expert teleoperation interface design [e.g., for manipulation ( Herlant et al, 2016 ; Kent et al, 2017 ; Li et al, 2017 ; Rakita et al, 2017 ), shared autonomy ( Jain et al, 2015 ; Aronson et al, 2018 ; Rakita et al, 2019b ; Jeon et al, 2020 ; Nicolis, 2020 ), camera control ( Rakita et al, 2018 ; Rakita et al, 2019a ), and social and psychological interfaces ( Rea et al, 2017a ; Rea and Young, 2018 ; Rea and Young, 2019a ; Rea et al, 2020 ; Seo et al, 2020 ; Valiton and Li, 2020 )], we call for teleoperation and robot researchers broadly to use more additional advanced applications of user-centered practices by starting with user-driven solutions in addition to existing technically-driven approaches.…”