Human-Computer Synergies in Prosthetic Interactions

Abstract

Remote sighted assistance provides prosthetic support to people with visual impairments (PVI) through internet-mediated conversational interactions. In these interactions, PVI broadcast live video to remotely-located, sighted people who engage in speech interactions with PVI to create prosthetic support. These interactions can be quite nuanced, creative, and effective. In this paper, we present a design investigation of remote sighted assistance (RSA) in which computer vision capabilities are integrated into the prosthetic interaction, supporting the human participants in various ways. Our study involved creating design scenarios to identify and concretize future possibilities in order to articulate and analyze design rationale for these scenarios, that is to say, strengths and challenges of RSA integrated with CV. We discuss implications for the design of the next generation of remote sighted assistance.

Helping Helpers: Supporting Volunteers in Remote Sighted Assistance with Augmented Reality Maps

“…To address the above challenges, our prior work has envisioned assisting agents with computer vision (CV) technologies, particularly 3D map construction, object annotations, real-time localization, and video stream augmentation [33,63,91]. We also designed a series of low-fdelity prototypes (e.g., pictorials) illustrating how CV technologies can be adopted in RSA services [91].…”

Feasibility of Interactive 3D Map for Remote Sighted Assistance

“…As RSA services have gained popularity, a growing number of researchers started to study different aspects of these services. Kamikubo et al [26] and Lee et al [27,19,28,29] studied the interaction paradigm between RSA agents and visually impaired users and reported several challenges for RSA agents. Some of the key challenges include agents' lack of confidence due to unfamiliarity of the users' current physical environment, not having indoor maps with finer details, continuously tracking and orienting the user in the map and within his or her surroundings, estimating objects' depth in users' camera feed and describing those in realtime, and developing mutual trust and synergy with the users.…”