ImageExplorer: Multi-Layered Touch Exploration to Encourage Skepticism Towards Imperfect AI-Generated Image Captions

Lee, Jae‐Wook; Herskovitz, Jaylin; Peng, Yi-Hao; Guo, Anhong

doi:10.1145/3491102.3501966

Cited by 21 publications

(21 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Morris et al looked at touchscreen-based image exploration as one of several techniques for providing BLV users with richer representations of images [37]. Systems, such as Lee et al 's ImageExplorer [26] and Microsoft's Seeing AI [35], use deep learning models to automatically detect important objects within an image and place bounding boxes around those objects. As users move their finger over the image and touch those bounding boxes, the app will verbally announce what those objects are.…”

Section: Presenting Images To Blv Users Viamentioning

confidence: 99%

“…Although not a complete replacement for physical tactile graphics, touchscreen-based image exploration systems present distinct benefits for BLV users, especially when compared with standard alt text and image descriptions. Prior work has found that these systems afford users a strong sense of the spatial layout and sizes of elements within the image [14,26,37]. Research has also found that allowing users to explore the image on their own terms via touch affords a sense of agency [26], and allows BLV users to more vividly imagine and understand what the image is showing [1].…”

Section: Presenting Images To Blv Users Viamentioning

confidence: 99%

“…Touchscreen-based image exploration systems have several advantages for BLV users over simple alt text, including granting users with a strong sense of the spatial layout of image elements, affording users with a strong sense of autonomy, and allowing BLV users to more vividly imagine what images are showing [1,26,37]. Yet, exploring images using these systems is a process requiring far more effort than simply hearing a description: Users must move their finger around the screen to look for elements on the image while trying to make sure that they have seen everything.…”

Section: Introductionmentioning

confidence: 99%

“…Yet, exploring images using these systems is a process requiring far more effort than simply hearing a description: Users must move their finger around the screen to look for elements on the image while trying to make sure that they have seen everything. Indeed, prior work has identified the effort required as a major issue facing these systems [26]. Little work, however, has been done to explore the specific interaction bottlenecks faced by BLV users in the image exploration process and propose potential solutions to alleviate these bottlenecks.…”

Section: Introductionmentioning

confidence: 99%

“…We first implemented a smartphone-based, touchscreen image exploration system -meant to faithfully recreate state-of-the-art methods [1,26,35,37] -and had two BLV users try the tool out on a variety of images while providing feedback as part of a formative study. We found that participants scanned the image in an inefficient manner, and they often missed elements of the image even when they were trying to find everything.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

ImageAssist: Tools for Enhancing Touchscreen-Based Image Exploration Systems for Blind and Low Vision Users

Nair,

Zhu,

Smith

2023

Preprint

View full text Add to dashboard Cite

Figure 1: Depictions of the state of the art in touchscreen-based image exploration alongside ImageAssist, a set of three tools we created to alleviate bottlenecks in this state-of-the-art/baseline system. The baseline system divides images into regions, and users touch the image to explore them. The tools that comprise ImageAssist are: A menu & beacon tool listing areas within the image and directing users to those areas; a hints tool providing recommendations and a volume-based prominence indicator to give users a sense of direction while exploring; and a zoom tool to quickly zoom into the four quadrants of the image to survey small areas.

show abstract

Section: Presenting Images To Blv Users Viamentioning

confidence: 99%

Section: Presenting Images To Blv Users Viamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

ImageAssist: Tools for Enhancing Touchscreen-Based Image Exploration Systems for Blind and Low Vision Users

Nair,

Zhu,

Smith

2023

Preprint

View full text Add to dashboard Cite

show abstract

Analysis and design framework for the development of indoor scene understanding assistive solutions for the person with visual impairment/blindness

Valipoor,

de Antonio,

Cabrera

2024

Multimedia Systems

View full text Add to dashboard Cite

This paper discusses the challenges of the current state of computer vision-based indoor scene understanding assistive solutions for the person with visual impairment (P-VI)/blindness. It focuses on two main issues: the lack of user-centered approach in the development process and the lack of guidelines for the selection of appropriate technologies. First, it discusses the needs of users of an assistive solution through state-of-the-art analysis based on a previous systematic review of literature and commercial products and on semi-structured user interviews. Then it proposes an analysis and design framework to address these needs. Our paper presents a set of structured use cases that help to visualize and categorize the diverse real-world challenges faced by the P-VI/blindness in indoor settings, including scene description, object finding, color detection, obstacle avoidance and text reading across different contexts. Next, it details the functional and non-functional requirements to be fulfilled by indoor scene understanding assistive solutions and provides a reference architecture that helps to map the needs into solutions, identifying the components that are necessary to cover the different use cases and respond to the requirements. To further guide the development of the architecture components, the paper offers insights into various available technologies like depth cameras, object detection, segmentation algorithms and optical character recognition (OCR), to enable an informed selection of the most suitable technologies for the development of specific assistive solutions, based on aspects like effectiveness, price and computational cost. In conclusion, by systematically analyzing user needs and providing guidelines for technology selection, this research contributes to the development of more personalized and practical assistive solutions tailored to the unique challenges faced by the P-VI/blindness.

show abstract

Ayn: Assistive Tool to Describe Images for Blind and Visually Impaired

Alqahtani,

Almutairi

et al. 2023

2023 Sixth International Conference of Women in Data Science at Prince Sultan University (WiDS PSU)

View full text Add to dashboard Cite

ImageExplorer: Multi-Layered Touch Exploration to Encourage Skepticism Towards Imperfect AI-Generated Image Captions

Cited by 21 publications

References 40 publications

ImageAssist: Tools for Enhancing Touchscreen-Based Image Exploration Systems for Blind and Low Vision Users

ImageAssist: Tools for Enhancing Touchscreen-Based Image Exploration Systems for Blind and Low Vision Users

Analysis and design framework for the development of indoor scene understanding assistive solutions for the person with visual impairment/blindness

Ayn: Assistive Tool to Describe Images for Blind and Visually Impaired

Contact Info

Product

Resources

About