Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics

Abstract

Few images on the Web receive alt-text descriptions that would make them accessible to blind and low vision (BLV) users. Imagebased NLG systems have progressed to the point where they can begin to address this persistent societal problem, but these systems will not be fully successful unless we evaluate them on metrics that guide their development correctly. Here, we argue against current referenceless metrics -those that don't rely on human-generated ground-truth descriptions -on the grounds that they do not align with the needs of BLV users. The fundamental shortcoming of these metrics is that they cannot take context into account, whereas contextual information is highly valued by BLV users. To substantiate these claims, we present a study with BLV participants who rated descriptions along a variety of dimensions. An indepth analysis reveals that the lack of contextawareness makes current referenceless metrics inadequate for advancing image accessibility, requiring a rethinking of referenceless evaluation metrics for image-based NLG systems.

Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM

“…Within the domain of Human-Computer Interaction, researchers have extensively delved into accessibility challenges prevalent in various categories of small-scale mobile applications [56,63,83,84,87,107,134], spanning domains like health [39,61,93,110,129], smart cities [95,127,132], and government engagement [13,66]. While these works scrutinize distinct facets of accessibility, a recurring theme has been the conspicuous absence of descriptions for image-based components [68,98,112,122]. The gravity of this issue has been explicitly recognized across studies.…”

“It’s Kind of Context Dependent”: Understanding Blind and Low Vision People’s Video Accessibility Preferences Across Viewing Scenarios

“…Prior work has also emphasized that BLV people's preferences for image descriptions vary based on an image's context [17]. Specifically, preferences differ based on the source or content of the image [4,21,49,60,63,78,79,111,114]. For example, Stangl et al [111,114] found that BLV people wanted different details for images associated with different sources and user goals.…”