VOILA: An Optimised Dialogue System for Interactively Learning Visually-Grounded Word Meanings (Demonstration System)

Abstract

We present VOILA: an optimised, multimodal dialogue agent for interactive learning of visually grounded word meanings from a human user. VOILA is: (1) able to learn new visual categories interactively from users from scratch; (2) trained on real human-human dialogues in the same domain, and so is able to conduct natural spontaneous dialogue; (3) optimised to find the most effective trade-off between the accuracy of the visual categories it learns and the cost it incurs to users. VOILA is deployed on Furhat 1 , a humanlike, multi-modal robot head with backprojection of the face, and a graphical virtual character.

Artificial intelligence and machine learning

“…Neuhofer et al (2015) suggest an agent capable of personalization through a continuous learning processes of guest information for digital platforms, which an example of such an agent. Other examples include agents capable of making recommendations on music platforms (Liebman et al, 2015), regulating heat pump thermostats (Ruelens et al, 2015), acquiring collective knowledge across different tasks (Rostami et al, 2017), and learning the meanings of words (Yu et al, 2017). The choice of the learning type in agents (simple-reflex vs. learning agent) influences the agent's general overall design and the contribution of ML.…”

Machine Learning in Artificial Intelligence: Towards a Common Understanding

“…An example for such an agent is shown by Liebman et al, who build a self-learning agent for music playlist recommendations [60]. Other cases are for instance the regulation of heat pump thermostats [61], an agent to acquire collective knowledge over different tasks [62] or learning word meanings [63].…”