Gesture-based robot control: Design challenges and evaluation with humans

Coronado, Enrique; Villalobos, Jessica; Bruno, Barbara; Mastrogiovanni, Fulvio

doi:10.1109/icra.2017.7989321

Cited by 39 publications

(30 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…using fiducial markers. There exist a number of hand gesturebased HRI frameworks [17], [18], [19] for terrestrial robots. In addition, recent visual hand gesture recognition techniques [26], [27] based on CNNs have been shown to be highly accurate and robust to noise and visual distortions [8].…”

Section: B Underwater Human-robot Communicationmentioning

confidence: 99%

Understanding human motion and gestures for underwater human–robot collaboration

Islam

Sattar³

2018

Journal of Field Robotics

View full text Add to dashboard Cite

In this paper, we present a number of robust methodologies for an underwater robot to visually detect, follow, and interact with a diver for collaborative task execution. We design and develop two autonomous diver-following algorithms, the first of which utilizes both spatial-and frequency-domain features pertaining to human swimming patterns in order to visually track a diver. The second algorithm uses a convolutional neural network-based model for robust tracking-by-detection. In addition, we propose a hand gesture-based human-robot communication framework that is syntactically simpler and computationally more efficient than the existing grammar-based frameworks. In the proposed interaction framework, deep visual detectors are used to provide accurate hand gesture recognition; subsequently, a finite-state machine performs robust and efficient gesture-to-instruction mapping. The distinguishing feature of this framework is that it can be easily adopted by divers for communicating with underwater robots without using artificial markers or requiring memorization of complex language rules. Furthermore, we validate the performance and effectiveness of the proposed methodologies through extensive field experiments in closed-and open-water environments. Finally, we perform a user interaction study to demonstrate the usability benefits of our proposed interaction framework compared to existing methods.

show abstract

Section: B Underwater Human-robot Communicationmentioning

confidence: 99%

Understanding human motion and gestures for underwater human–robot collaboration

Islam

Sattar³

2018

Journal of Field Robotics

View full text Add to dashboard Cite

show abstract

“…Modulating robot control based on human input in the form of speech, hand gestures, or keyboard interfaces has been explored extensively for terrestrial environments [3], [5], [20], [22]. However, most of these human-robot communication modules are not readily applicable in underwater applications due to environmental and operational constraints [7].…”

Section: Related Workmentioning

confidence: 99%

“…Controlling a robot using speech, direct input (e.g., a keyboard or joystick), or free-form gestures is a general paradigm [3], [5], [22] in the context of Human-Robot Interaction (HRI). Unlike relatively less challenging terrestrial environments, the use of keyboard or joystick interfaces or tactile sensors is unappealing in underwater applications since it entails costly waterproofing and introduces an additional point of failure.…”

Section: Introductionmentioning

confidence: 99%

Dynamic Reconfiguration of Mission Parameters in Underwater Human-Robot Collaboration

Islam

Sattar

2018

2018 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

This paper presents a real-time programming and parameter reconfiguration method for autonomous underwater robots in human-robot collaborative tasks. Using a set of intuitive and meaningful hand gestures, we develop a syntactically simple framework that is computationally more efficient than a complex, grammar-based approach. In the proposed framework, a convolutional neural network is trained to provide accurate hand gesture recognition; subsequently, a finite-state machine-based deterministic model performs efficient gestureto-instruction mapping and further improves robustness of the interaction scheme. The key aspect of this framework is that it can be easily adopted by divers for communicating simple instructions to underwater robots without using artificial tags such as fiducial markers or requiring memorization of a potentially complex set of language rules. Extensive experiments are performed both on field-trial data and through simulation, which demonstrate the robustness, efficiency, and portability of this framework in a number of different scenarios. Finally, a user interaction study is presented that illustrates the gain in the ease of use of our proposed interaction framework compared to the existing methods for the underwater domain.

show abstract

“…Traditionally, they have been used to perform high-speed, efficient and repetitive tasks in hazardous and industrial environments, often requiring few or no interactive capabilities [1,2] . These cases differ significantly from social robots, for which the main goals are to play useful social roles and engage different types of users through meaningful, natural, suitable, and safe interactions [3,4] . Despite being one of the most relevant emergent technologies according to the World Economic Fo-conventions are better approached by UX/UI designers, researchers, practitioners in social sciences.…”

Section: Introductionmentioning

confidence: 99%

Visual Programming Environments for End-User Development of intelligent and social robots, a systematic review

Coronado

Mastrogiovanni

Indurkhya

et al. 2020

Journal of Computer Languages

Self Cite

View full text Add to dashboard Cite

Robots are becoming interactive and robust enough to be adopted outside laboratories and in industrial scenarios as well as interacting with humans in social activities. However, the design of engaging robotbased applications requires the availability of usable, flexible and accessible development frameworks, which can be adopted and mastered by researchers and practitioners in social sciences and adult end users as a whole. This paper surveys Visual Programming Environments aimed at enabling a paradigm fostering the so-called End-User Development of applications involving robots with social capabilities. The focus of this article is on those Visual Programming Environments that are designed to support social research goals as well as to cater for professional needs of people not trained in more traditional text-based computer programming languages. This survey excludes interfaces aimed at supporting expert programmers, at allowing industrial robots to perform typical industrial tasks (such as pick and place operations), and at teaching children how to code. After having performed a systematic search, sixteen programming environments have been included in this survey. Our goal is two-fold: first, to present these software tools with their technical features and Authoring Artificial Intelligence modeling approaches, and second, to present open challenges in the development of Visual Programming Environments for end users and social researchers, which can be informative and valuable to the community. The results show that the most recent such tools are adopting distributed and Component-Based Software Engineering approaches and web technologies. However, few of them have been designed to enable the independence of end users from high-tech scribes. Moreover, findings indicate the need for (i) more objective and comparative evaluations, as well as usability and user experience studies with real end users; and (ii) validations of these tools for designing applications aimed at working "in-the-wild" rather than only in laboratories and structured settings.

show abstract

Gesture-based robot control: Design challenges and evaluation with humans

Cited by 39 publications

References 21 publications

Understanding human motion and gestures for underwater human–robot collaboration

Understanding human motion and gestures for underwater human–robot collaboration

Dynamic Reconfiguration of Mission Parameters in Underwater Human-Robot Collaboration

Visual Programming Environments for End-User Development of intelligent and social robots, a systematic review

Contact Info

Product

Resources

About