Robotics‐based telepresence using multi‐modal interaction for individuals with visual impairments

Abstract

Although many assistive devices have been developed and utilized to aid in daily living, the most general assistive means for individuals with visual impairments are the walking cane and guide dogs. These assistive means are effective in assisting the user in navigating within an environment; however, the navigation space is limited to the proximal environment of the user. Thus, in this paper, we discuss a method to increase the range of accessibility to a remote environment through robotic embodiment that enables teleoperation and teleperception through multi-modal feedback. In order to transform remote spatial information into a nonvisual modality, we present a framework for utilizing an RGB-D-based depth camera, a mobile robot, and a haptic interface for 3D haptic rendering to accomplish the goal of haptic exploration of a remote environment. Experiments with three different control methods for robot interaction are designed for users with and without visual impairments. Several hypothesis are built to study the correlation between control/feedback modality and performance in telerobotic operations. Results show that users performed best when combining semi-autonomous navigation with 3D haptic exploration and also rated their experience with our system as fairly good.

Towards Tangible Vision for the Visually Impaired through 2D Multiarray Braille Display

“…There are methods that use haptic telepresence devices, which can express tactual perception using 2D+depth data. Park et al [9,10,11,12,13], who collaborated with our team, developed the robot system to remotely observe the visual attractions and evade obstacles at art galleries or museums. Figure 3 illustrates a telepresence robot and a haptic interaction system.…”

“…Further, it is difficult for them to obtain visual multimedia from information technology devices, such as PCs, smartphones, and televisions. Park, a co-researcher [9,10,11,12,13], developed methods for touching 3D shapes through depth capturing and its reconstruction. However, these methods are considerably slow for real-time haptic telepresence.…”

“…The studies conducted by Park [9,10,11,12,13] were focused on capturing 2D+ depth images and rendering using a haptic device. However, this study performs haptic rendering through a 2D+ depth video.…”

“…It reduces more than 50% of the bitrate when compared to existing video coding standards. Park et al [9] studied haptic rendering with a 320×240 point cloud, which was transformed into a 3D grid map, translated into a disparity map for visualization, and calibrated for distance in case of scaling. In contrast, this research provides the depth map from IR sensors at approximately 30 fps.…”

Special focus section on applications of signal processing and control to assistive technology

“…Park and Howard discuss the use of robotics and an RGB‐D depth camera to give visually impaired people access to a remote environment through a 3D haptic interface. They also present the results of end‐users tests of the system.…”