AR-mentor: Augmented reality based mentoring system

Abstract

AR-Mentor is a wearable real time Augmented Reality (AR) mentoring system that is configured to assist in maintenance and repair tasks of complex machinery, such as vehicles, appliances, and industrial machinery. The system combines a wearable Optical-See-Through (OST) display device with high precision 6-Degree-Of-Freedom (DOF) pose tracking and a virtual personal assistant (VPA) with natural language, verbal conversational interaction, providing guidance to the user in the form of visual, audio and locational cues. The system is designed to be heads-up and hands-free allowing the user to freely move about the maintenance or training environment and receive globally aligned and context aware visual and audio instructions (animations, symbolic icons, text, multimedia content, speech). The user can interact with the system, ask questions and get clarifications and specific guidance for the task at hand. A pilot application with AR-Mentor was successfully built to instruct a novice to perform an advanced 33-step maintenance task on a training vehicle. The initial live training tests demonstrate that AR-Mentor is able to help and serve as an assistant to an instructor, freeing him/her to cover more students and to focus on higher-order teaching.

AR Enabled IoT for a Smart and Interactive Environment: A Survey and Future Directions

“…Some research in the augmented reality (AR) field is about simulation related to the control of the object’s function and usability measurement [44]. AR also offers an excellent method of in situ object control (even for remote objects, using a remote-controlled camera) [1].…”

User Interactions for Augmented Reality Smart Glasses: A Comparative Evaluation of Visual Contexts and Interaction Gestures

“…Several research works have been conducted to provide visualization methods concerning visual contexts for practical task assistance in different environments [1,8,9,13]. Most smart AR glasses support see-through augmentation.…”

“…Previous studies proposed methods for task assistance in wearable AR. Zhu et al [9] proposed AR-Mentor, which is an AR system for assisting in maintenance and repair tasks. Their method combined AR smart glasses with six degrees of freedom (DOF) pose tracking and virtual personal assistance to instruct a novice in performing maintenance tasks.…”

“…Sci. 2019, 9,3171 3 of 27 2. Non-stereo rendering and indirect augmentation with video background 3.…”

Application of Augmented Reality and Robotic Technology in Broadcasting: A Survey

“…By using MSCKF (Sliding window of poses) to present state vector in EKF formulation, it overcomes the inconsistent problem of classical EKF-based visual tracking algorithm. Zhu, et al tracked the pose of HMD AR device in 6 DOF with EKF, and combined vision and IMU data in "tightly coupled" manner to optimize motion model [73].…”