A Competence-Aware Curriculum for Visual Concepts Learning via Question Answering

Li, Qing; Huang, Siyuan; Hong, Yining; Zhu, Song‐Chun

doi:10.1007/978-3-030-58536-5_9

Cited by 20 publications

(11 citation statements)

References 75 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Researchers have proposed to combine statistical learning and symbolic reasoning, with pioneer efforts devoted to different directions, including representation learning and reasoning (Sun, 1994;Garcez et al, 2008;Manhaeve et al, 2018), abductive learning (Li et al, 2020a;Dai et al, 2019;Zhou, 2019), knowledge abstraction (Hinton et al, 2006;Bader et al, 2009), etc. There also have been recent works on the application of neural-symbolic methods, such as neural-symbolic visual reasoning and program synthesis (Yi et al, 2018;Mao et al, 2018;Li et al, 2020b;Parisotto et al, 2016), semantic parsing (Liang et al, 2016;Yin et al, 2018), and math word problems (Lample & Charton, 2020;Lee et al, 2020). Current neural-symbolic approaches often require a perfect domain-specific language, including both the syntax and semantics of the targeted domain.…”

Section: Neural-symbolic Integrationmentioning

confidence: 99%

A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

Li¹,

Huang²,

Hong³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Inspired by humans' remarkable ability to master arithmetic and generalize to unseen problems, we present a new dataset, HINT, to study machines' capability of learning generalizable concepts at three different levels: perception, syntax, and semantics. In particular, concepts in HINT, including both digits and operators, are required to learn in a weakly-supervised fashion: Only the final results of handwriting expressions are provided as supervision. Learning agents need to reckon how concepts are perceived from raw signals such as images (i.e., perception), how multiple concepts are structurally combined to form a valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics). With a focus on systematic generalization, we carefully design a fivefold test set to evaluate both the interpolation and the extrapolation of learned concepts. To tackle this challenging problem, we propose a neuralsymbolic system by integrating neural networks with grammar parsing and program synthesis, learned by a novel deduction-abduction strategy. In experiments, the proposed neural-symbolic system demonstrates strong generalization capability and significantly outperforms end-to-end neural methods like RNN and Transformer. The results also indicate the significance of recursive priors for extrapolation on syntax and semantics.

show abstract

Section: Neural-symbolic Integrationmentioning

confidence: 99%

A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

Li¹,

Huang²,

Hong³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Visual reasoning aims to reason about object properties and their relationships in given images, usually evaluated as the question-answering accuracy (Johnson et al, 2017a;Hudson & Manning, 2018;Mascharka et al, 2018;Hu et al, 2018). Recently, there has been an increasing amount of work has been focusing on using neuro-symbolic frameworks to bridge visual concept learning and visual reasoning (Yi et al, 2018;Li et al, 2020). The high-level idea is to disentangle concept learning: association of linguistic units with visual representations, and reasoning: the ability to count objects or make queries.…”

Section: Related Workmentioning

confidence: 99%

FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations

Mei¹,

Mao²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a meta-learning framework for learning new visual concepts quickly, from just one or a few examples, guided by multiple naturally occurring data streams: simultaneously looking at images, reading sentences that describe the objects in the scene, and interpreting supplemental sentences that relate the novel concept with other concepts. The learned concepts support downstream applications, such as answering questions by reasoning about unseen images. Our model, namely FALCON, represents individual visual concepts, such as colors and shapes, as axis-aligned boxes in a high-dimensional space (the "box embedding space"). Given an input image and its paired sentence, our model first resolves the referential expression in the sentence and associates the novel concept with particular objects in the scene. Next, our model interprets supplemental sentences to relate the novel concept with other known concepts, such as "X has property Y" or "X is a kind of Y". Finally, it infers an optimal box embedding for the novel concept that jointly 1) maximizes the likelihood of the observed instances in the image, and 2) satisfies the relationships between the novel concepts and the known ones. We demonstrate the effectiveness of our model on both synthetic and real-world datasets.

show abstract

“…Module networks Architectures based on neural module networks (Andreas et al, 2015) have been studied for vision-and-language learning tasks, both in visual grounding (Hu et al, 2016;Yu et al, 2018) and in VQA context (Li et al, 2020;Mao et al, 2019;Yi et al, 2018). Similar to our approach, module networks break the overall architecture into separate modules, each responsible for grounding a specific concept in the input query.…”

Section: Related Workmentioning

confidence: 99%

Sim-To-Real Transfer of Visual Grounding for Human-Aided Ambiguity Resolution

Tziafas¹,

Kasaei²

2022

Preprint

View full text Add to dashboard Cite

Service robots should be able to interact naturally with non-expert human users, not only to help them in various tasks, but also to receive guidance in order to resolve ambiguities that might be present in the instruction. We consider the task of visual grounding, where the agent segments an object from a crowded scene given a natural language description. Modern holistic approaches to visual grounding usually ignore language structure and struggle to cover generic domains, therefore relying heavily on large datasets. Additionally, their transfer performance in RGB-D datasets suffers due to high visual discrepancy between the benchmark and the target domains. Modular approaches marry learning with domain modeling and exploit the compositional nature of language to decouple visual representation from language parsing, but either rely on external parsers or are trained in an end-to-end fashion due to the lack of strong supervision. In this work, we seek to tackle these limitations by introducing a fully decoupled modular framework for compositional visual grounding of entities, attributes and spatial relations. We exploit rich scene graph annotations generated in a synthetic domain and train each module independently in simulation. Our approach is evaluated both in simulation and in two real RGB-D scene datasets. Experimental results show that the decoupled nature of our framework allows for easy integration with domain adaptation approaches for Sim-To-Real visual recognition, offering a data-efficient, robust, and interpretable solution to visual grounding in robotic applications. * We thank NVIDIA Corporation for their generous donation of GPUs which was partially used in this research.

show abstract

A Competence-Aware Curriculum for Visual Concepts Learning via Question Answering

Cited by 20 publications

References 75 publications

A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

A HINT from Arithmetic: On Systematic Generalization of Perception, Syntax, and Semantics

FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic descriptions, and Conceptual Relations

Sim-To-Real Transfer of Visual Grounding for Human-Aided Ambiguity Resolution

Contact Info

Product

Resources

About