Hands-Free Assistive Manipulator Using Augmented Reality and Tongue Drive System

Chu, Fu-Jen; Xu, Ruinian; Zhang, Zhenxuan; Vela, Patricio A.; Ghovanloo, Maysam

doi:10.1109/iros.2018.8594508

Cited by 5 publications

(2 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Obviously such an approach is restricted to systems with a screen. In [10] a deep learning approach, namely the YOLO network, was used to generate 3D bounding boxes around objects, however an ARUCO marker was still used to transform the coordinate systems between the HMD and a robot arm. Some special cases exist.…”

Section: A Referencing In Augmented Realitymentioning

confidence: 99%

Referencing between a Head-Mounted Device and Robotic Manipulators

Puljiz¹,

Hein²,

Kröger³

2019

Preprint

View full text Add to dashboard Cite

Having a precise and robust transformation between the robot coordinate system and the AR-device coordinate system is paramount during human-robot interaction (HRI) based on augmented reality using Head mounted displays (HMD), both for intuitive information display and for the tracking of human motions. Most current solutions in this area rely either on the tracking of visual markers, e.g. QR codes, or on manual referencing, both of which provide unsatisfying results. Meanwhile a plethora of object detection and referencing methods exist in the wider robotic and machine vision communities. The precision of the referencing is likewise almost never measured. Here we would like to address this issue by firstly presenting an overview of currently used referencing methods between robots and HMDs. This is followed by a brief overview of object detection and referencing methods used in the field of robotics. Based on these methods we suggest three classes of referencing algorithms we intend to pursue -semi-automatic, on-shot; automatic, one-shot; and automatic continuous. We describe the general workflows of these three classes as well as describing our proposed algorithms in each of these classes. Finally we present the first experimental results of a semi-automatic referencing algorithm, tested on an industrial KUKA KR-5 manipulator.

show abstract

Section: A Referencing In Augmented Realitymentioning

confidence: 99%

Referencing between a Head-Mounted Device and Robotic Manipulators

Puljiz¹,

Hein²,

Kröger³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The combination of affordances, defined as relations between objects and the actions that a robot can apply to them [3], and AR head-mounted displays (HMD) UIs, is a design paradigm that has allowed untrained users fluent control of single and multi-robot assistive platforms [4], [5], [6]. By exploiting affordances, AR HMD UIs assist the user in the perception and planning tasks that are naturally involved in the control of assistive robots and have demonstrated benefits Fig.…”

Section: Introductionmentioning

confidence: 99%

Proactive Robot Assistance: Affordance-Aware Augmented Reality User Interfaces

Chacon-Quesada

Demiris

2022

IEEE Robot. Automat. Mag.

View full text Add to dashboard Cite

Assistive robots can increase the autonomy and quality of life of people with disabilities and Augmented Reality (AR) User Interfaces (UIs) have the potential to facilitate their use. In this paper, we argue that to fulfil this potential and accommodate a more diverse user base, AR UIs should proactively identify user affordances, i.e., action options that are possible in the current context. However, current AR UIs for the control of assistive robots do not readily combine atomic actions and therefore can only provide individual actions as options. To overcome these limitations, we propose Affordance-Aware Proactive Planning (AP) 2 , an algorithm that proactively identifies feasible sequences of atomic actions by leveraging large datasets of plans given using human language. (AP) 2 incorporates natural language procesing, and planning algorithms to provide the most relevant and feasible plans given the user's context and provides means to reduce the time required to generate and present these as options to the user. Our main contributions are: 1) we propose a method that allows affordance-aware AR UI for robot control to combine atomic actions and provide higher-level options to the user, 2) we provide a means for dynamically updating goal states and the amount of semantically relevant plans that are analysed to facilitate ways to improve interactivity for the user, and 3) we validate the applicability of the proposed architecture with an assistive mobile manipulator deployed in a bedroom environment and controlled using an AR UI.

show abstract