RegionSpeak

Abstract

Blind people often seek answers to their visual questions from remote sources, however, the commonly adopted singleimage, single-response model does not always guarantee enough bandwidth between users and sources. This is es pecially true when questions concern large sets of informa tion, or spatial layout, e.g., where is there to sit in this area, what tools are on this work bench, or what do the buttons on this machine do? Our RegionSpeak system addresses this problem by providing an accessible way for blind users to (i) combine visual information across multiple photographs via image stitching, (ii) quickly collect labels from the crowd for all relevant objects contained within the resulting large visual area in parallel, and (iii) then interactively explore the spa tial layout of the objects that were labeled. The regions and descriptions are displayed on an accessible touchscreen in terface, which allow blind users to interactively explore their spatial layout. We demonstrate that workers from Amazon Mechanical Turk are able to quickly and accurately identify relevant regions, and that asking them to describe only one region at a time results in more comprehensive descriptions of complex images. RegionSpeak can be used to explore the spatial layout of the regions identified. It also demonstrates broad potential for helping blind users to answer difficult spa tial layout questions.

Investigating Cursor-based Interactions to Support Non-Visual Exploration in the Real World

“…This potentially helps users explore and under stand the spatial layout of a scene or document. The interaction is similar to the Explore by Touch in VoiceOver, TalkBack and RegionSpeak [28].…”

“…Slide Rule developed multi-touch gestures that could control touchscreens non-visually [10], which informed the VoiceOver screen reader on iOS, and the TalkBack screen reader on An droid. RegionSpeak [28] enables spatial exploration of the layout of objects in a photograph using a touchscreen. Users send a photo (or multiple stitched photos) to have the crowd label all of the objects in the photo.…”

“…To explore this concept, we implemented three cursor-based in teractions to help blind users attend to items within a complex visual scene (Figure 1), including: (i) window cursor, in which the user moves the device to scan the scene and receives infor mation at the center of the camera, similar to VizWiz::LocateIt [3]; (ii) finger cursor, in which the user moves their finger on the real world object they want to access and receives informa tion near their fingertip, similar to VizLens [8]; (iii) and touch cursor, in which the user moves their finger on the touchscreen and receives information of the relative location on the live camera image, similar to RegionSpeak [28].…”

The Effect of Computer-Generated Descriptions on Photo-Sharing Experiences of People with Visual Impairments

“…If a visually impaired person takes a photo and does not upload it immediately, it is hard for her to navigate through the album and find that photo independently, especially when many photos have accumulated in her album over time.Moreover, it is difficult to judge the quality of a photo, for example, to determine whether the photo is blurry, whether a person has her eyes closed in the photo, or whether the photo is aesthetically pleasing [30].Researchers and designers have tackled this problem, proposing techniques to help people with visual impairments access and understand the contents of a photo. Most of these efforts used human-powered services (e.g., crowd workers, friends) to provide photo descriptions or answer a photo-based question [13,15,54,55]. However, such systems are hard to scale and sustain due to the limited number of volunteers, the monetary cost of crowd workers, and the possible social costs involved when asking friends [18,22].…”

“…Researchers and designers have tackled this problem, proposing techniques to help people with visual impairments access and understand the contents of a photo. Most of these efforts used human-powered services (e.g., crowd workers, friends) to provide photo descriptions or answer a photo-based question [13,15,54,55]. However, such systems are hard to scale and sustain due to the limited number of volunteers, the monetary cost of crowd workers, and the possible social costs involved when asking friends [18,22].…”