Why Does a Visual Question Have Different Answers?

Abstract

Visual question answering is the task of returning the answer to a question about an image. A challenge is that different people often provide different answers to the same visual question. To our knowledge, this is the first work that aims to understand why. We propose a taxonomy of nine plausible reasons, and create two labelled datasets consisting of ∼45,000 visual questions indicating which reasons led to answer differences. We then propose a novel problem of predicting directly from a visual question which reasons will cause answer differences as well as a novel algorithm for this purpose. Experiments demonstrate the advantage of our approach over several related baselines on two diverse datasets. We publicly share the datasets and code at https://vizwiz.org.

"I Hope This Is Helpful"

“…On a higher level, it is important to note that what makes a "good" caption for users who are blind may not be ultimately deemed synonymous with what is deemed a "good" caption for AI researchers (as discussed in section 2.3). Our findings support previous work indicating that workers' subjective, different interpretations may be desirable [8,10,25,42,78,101,103]).…”

Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision

“…Driven by VQA, several datasets have been proposed to minimize the bias observed in natural images (Goyal et al, 2017;Ray et al, 2019); to force models to "reason" over a joint visual and linguistic input Suhr et al, 2019); to deal with objects' attributes and relations (Krishna et al, 2017); to encompass more diverse (Zhu et al, 2016) and goal-oriented questions and answers (Gurari et al, 2018). At the same time, some work proposed higher-level evaluations of VQA models and showed their limitations (Hodosh and Hockenmaier, 2016;Shekhar et al, 2017); similarly, recent attention has been paid to understand what makes a question "difficult" for a model (Bhattacharya et al, 2019;Terao et al, 2020). Despite impressive progress, current approaches to VQA do not tackle one crucial limitation of the task: the answer to a question is given by the alignment of language and vision rather than their complementary integration.…”

Unanswerable Questions About Images and Texts

“…Other studies that have addressed the issues of unanswerable visual questions include Toor et al ( 2017 ) and Bhattacharya et al ( 2019 ).…”