An integrated system for interactive continuous learning of categorical knowledge

Abstract

This article presents an integrated robot system capable of interactive learning in dialogue with a human. Such a system needs to have several competencies and must be able to process different types of representations. In this article, we describe a collection of mechanisms that enable integration of heterogeneous competencies in a principled way. Central to our design is the creation of beliefs from visual and linguistic information, and the use of these beliefs for planning system behaviour to satisfy internal drives. The system is able to detect gaps in its knowledge and to plan and execute actions that provide information needed to fill these gaps. We propose a hierarchy of mechanisms which are capable of engaging in different kinds of learning interactions, e.g. those initiated by a tutor or by the system itself. We present the theory these mechanisms are build upon and an instantiation of this theory in the form of an integrated robot system. We demonstrate the operation of the system in the case of learning conceptual models of objects and their visual properties

Learning how to Learn: An Adaptive Dialogue Agent for Incrementally Learning Visually Grounded Word Meanings

“…Ideally, such interactive systems ought to be able to handle natural, spontaneous human dialogue. However, most work on interactive language grounding learn their systems from synthetic, hand-made dialogues or simulations which lack both in variation and the kinds of dialogue phenomena that occur in everyday conversation; they thus lead to systems which are not robust and cannot handle everyday conversation (Yu et al, 2016c;Skocaj et al, 2016;Yu et al, 2016a). In this paper, we try to change this by training an adaptive learning agent from human-human dialogues in a visual attribute learning task.…”

“…from images or videos an-notated with descriptions or definite reference expressions as in (Kennington and Schlangen, 2015;Socher et al, 2014)) or from live interaction as in, e.g. (Skocaj et al, 2016;Yu et al, 2015Yu et al, , 2016cDas et al, 2017Das et al, , 2016de Vries et al, 2016;Thomason et al, 2015Thomason et al, , 2016Tellex et al, 2013). The latter, which we do here, is clearly more appropriate for multimodal systems or robots that are expected to continuously, and incrementally learn from the environment and their users.…”

“…a dictionary of pre-trained classifiers) as in e.g. Silberer and Lapata (2014); Thomason et al (2016); Kollar et al (2013); Matuszek et al (2014); and 2) the agent as a child: without any prior knowledge of perceptual categories, the agent must learn both the perceptual categories themselves and also how NL expressions map to these (Skocaj et al, 2016;Yu et al, 2016c). Here, we concentrate on the latter scenario, where a system learns to identify and describe visual attributes (colour and shape in this case) through interaction with human tutors, incrementally, over time.…”

“…Previous work has approached the grounding problem using a variety of resources and approaches, for instance, either using annotated visual datasets (Silberer and Lapata, 2014;Socher et al, 2014;Naim et al, 2015;Al-Omari et al, 2016;Tellex et al, 2014;Matuszek et al, 2012Matuszek et al, , 2014, or through interactions with other agents or real humans (Kollar et al, 2013;Tellex et al, 2013;Thomason et al, 2015Thomason et al, , 2016Skocaj et al, 2016;Yu et al, 2016c), where feedback from other agents is used to learn new concepts.…”

Situated Interaction with a Smart Environment: Challenges and Opportunities

“…While there have been many efforts to capture the notion of common ground in general [8] and in human-robot interaction settings [6], the computational management of situated spatial dialogue is still under-developed [15,34] and requires creative solutions for reference handling [23], including attempts to incorporate the human's gaze in the system's interpretation procedure [1], and strategies for handling errors [33]. One major challenge concerns the fundamental difference between human concepts represented by natural language, especially in the domain of space [2], and formal systems suited for computational purposes, e.g., spatial reasoning-even if based on qualitative rather than metric relations [25].…”