Understanding Knowledge Gaps in Visual Question Answering: Implications for Gap Identification and Testing

Bajaj, Goonmeet; Bandyopadhyay, Bortik; Schmidt, Daniel F.; Maneriker, Pranav; Myers, Christopher W.; Parthasarathy, S.

doi:10.1109/cvprw50498.2020.00201

Cited by 3 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, this section summarizes the current research status of knowledge existing forms in Kb-VQA and displays them in Table 2. [11], Cudic [29], Lioutas [30], Liu [31], Lu [32], Mun [33], Wu [12] 2 Image knowledge Bai [14], Gordon [15], Li [16], Rahman [34] 3 Internal reasoning knowledge Zhang [17], Liang [18], Kim [19], Zhang [35], Bajaj [36] 4 External knowledge Zhang [25], Yu [24], Marino [37], Su [23], Aditya [10], Zhu [22], Wu [38], Wang [39,40], Wu [21], Zhu [20] 5 Unknown knowledge No…”

Section: Knowledge Pyramidmentioning

confidence: 99%

The Core of Smart Cities: Knowledge Representation and Descriptive Framework Construction in Knowledge-Based Visual Question Answering

Wang

Wu²,

Wang

2022

Sustainability

View full text Add to dashboard Cite

Visual question answering (VQA), which is an important presentation form of AI-complete task and visual Turing tests, coupled with its potential application value, attracted widespread attention from both researchers in computer vision and natural language processing. However, there are no relevant research regarding the expression and participation methods of knowledge in VQA. Considering the importance of knowledge for answering questions correctly, this paper analyzes and researches the stratification, expression and participation process of knowledge in VQA and proposes a knowledge description framework (KDF) to guide the research of knowledge-based VQA (Kb-VQA). The KDF consists of a basic theory, implementation methods and specific applications. This paper focuses on describing mathematical models at basic theoretical levels, as well as the knowledge hierarchy theories and key implementation behaviors established on this basis. In our experiment, using the statistics of VQA’s accuracy in the relevant literature, we propose a good corroboration of the research results from knowledge stratification, participation methods and expression forms in this paper.

show abstract

Section: Knowledge Pyramidmentioning

confidence: 99%

The Core of Smart Cities: Knowledge Representation and Descriptive Framework Construction in Knowledge-Based Visual Question Answering

Wang

Wu²,

Wang

2022

Sustainability

View full text Add to dashboard Cite

show abstract

“…This mapping allows for questions to be tagged with multiple KGs. The detailed mapping for question metadata fields to KGs and details about the KG identification method can be found in Bajaj et al (2020) and a sample mapping is presented in Table 2.…”

Section: Analysis Of Knowledge Gaps In the Gqa Datasetmentioning

confidence: 99%

“…See Bajaj et al. (2020) and Schmidt (2020) for an initial taxonomic representation of KGs. In Table 1, we expand our understanding of this initial representation by providing a mapping between the gaps that have been identified (Bajaj et al., 2020) and the cognitive processes or capacities we hypothesize to be involved in their resolution.…”

Section: Knowledge Gaps and Related Processesmentioning

confidence: 99%

“…A taxonomy of KGs and their associated capacities can also help researchers to focus on particular gaps or be informed on what types of gaps might arise as they research and develop particular cognitive capacities. See Bajaj et al (2020) and Schmidt (2020) for an initial taxonomic representation of KGs. In Table 1, we expand our understanding of this initial representation by providing a mapping between the gaps that have been identified (Bajaj et al, 2020) and the cognitive processes or capacities we hypothesize to be involved in their resolution.…”

Section: Explanatorymentioning

confidence: 99%

See 1 more Smart Citation

Knowledge Gaps: A Challenge for Agent‐Based Automatic Task Completion

Bajaj

Current

Schmidt

et al. 2021

Topics in Cognitive Science

View full text Add to dashboard Cite

The study of human cognition and the study of artificial intelligence (AI) have a symbiotic relationship, with advancements in one field often informing or creating new work in the other. Human cognition has many capabilities modern AI systems cannot compete with. One such capability is the detection, identification, and resolution of knowledge gaps (KGs). Using these capabilities as inspiration, we examine how to incorporate detection, identification, and resolution of KGs in artificial agents. We present a paradigm that enables research on the understanding of KGs for visual-linguistic communication. We leverage and enhance and existing KG taxonomy to identify possible KGs that can occur for visual question answer (VQA) tasks and use these findings to develop a classifier to identify questions that could be engineered to contain specific KG types for other VQA datasets. Additionally, we examine the performance of different VQA models through the lens of KGs.

show abstract