Context-based Word Acquisition for Situated Dialogue in a Virtual World

Abstract

To tackle the vocabulary problem in conversational systems, previous work has applied unsupervised learning approaches on co-occurring speech and eye gaze during interaction to automatically acquire new words. Although these approaches have shown promise, several issues related to human language behavior and human-machine conversation have not been addressed. First, psycholinguistic studies have shown certain temporal regularities between human eye movement and language production. While these regularities can potentially guide the acquisition process, they have not been incorporated in the previous unsupervised approaches. Second, conversational systems generally have an existing knowledge base about the domain and vocabulary. While the existing knowledge can potentially help bootstrap and constrain the acquired new words, it has not been incorporated in the previous models. Third, eye gaze could serve different functions in human-machine conversation. Some gaze streams may not be closely coupled with speech stream, and thus are potentially detrimental to word acquisition. Automated recognition of closely-coupled speech-gaze streams based on conversation context is important. To address these issues, we developed new approaches that incorporate user language behavior, domain knowledge, and conversation context in word acquisition. We evaluated these approaches in the context of situated dialogue in a virtual world. Our experimental results have shown that incorporating the above three types of contextual information significantly improves word acquisition performance.

Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots

“…Qu and Chai ( 2008 , 2010 ) proposed a learning method that automatically acquires novel words for an interactive system. They focused on the co-occurrence between word-sequences and entity-sequences tracked by eye-gaze in lexical acquisition.…”

Dialogue with Robots to Support Symbiotic Autonomy

“…The problem of tackling the vocabulary in conversational systems has been addressed in [24]. They propose approaches that incorporate user language behavior, domain knowledge, and conversation context in word acquisition, evaluating such methods in the context of situated dialogue in a virtual world.…”

Learning to Mediate Perceptual Differences in Situated Human-Robot Dialogue

“…Recent years have seen an increasing amount of work that grounds language to shared environment (Mooney 2008;Qu and Chai 2010;Chen and Mooney 2011;Tellex et al 2011;Matuszek et al 2012). Specifically, for grounding referring expressions to objects in the environment, different approaches have been developed (Gorniak and Roy 2004;Siebert and Schlangen 2008;DeVault and Stone 2009;Krishnamurthy and Kollar 2013).…”