PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

Abstract

Figure 1. Example structure of how PUMICE learns the concepts and procedures in the command "If it's hot, order a cup of Iced Cappuccino." The numbers indicate the order of utterances. The screenshot on the right shows the conversational interface of PUMICE. In this interactive parsing process, the agent learns how to query the current temperature, how to order any kind of drink from Starbucks, and the generalized concept of "hot" as "a temperature (of something) is greater than another temperature". ABSTRACTNatural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multimodal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations. Users can also define new procedures and concepts by demonstrating and referring to contents within GUIs of existing mobile apps. We demonstrate this approach in PUMICE, an end-user programmable agent that implements this approach. A lab study with 10 users showed its usability.

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs

“…An instructable agent is a promising new type of frame-based agent that can learn intents for new tasks interactively from the end user's natural language instructions [4,37,61] and/or demonstrations [1,35,39,47]. It allows users to use agents for personalized tasks and tasks in "long-tail" domains, addressing the "out-of-domain" errors in human-agent conversations [37].…”

“…However, these ap proaches usually require users to verbally clarify or define these keywords. A previous study [39] found that when users tried to verbally explain a concept unknown to the system, they often introduced even more unknown concepts in their explanations. The agents also have problems understanding such explanations due to their limited capability of reasoning with natural language instructions and domain knowledge.…”

“…The system shows guesses of user goals, assists the user to provide more effective inputs, and engages in multi-turn dialogs with the user to resolve any uncertainties and ambiguities. This com bination of multi-modal and mixed-initiative approaches has been previously applied in the interactive task learning process in systems such as [1,30,36,39]. SOVITE bridges an important gap in these systems, as they focus on the ambiguities, uncer tainties, and vagueness embedded in the user instructions of In terms of the technique used, SOVITE extracts the semantics of app GUIs [13,43] for grounding natural language conver sations.…”

“…We believe this approach should help users to more effectively identify the understand ing errors because it provides better closeness of mapping [21] to how the user would naturally approach this task. Prior stud ies showed that references to existing app GUIs were effective in other aspects of conversational interface designs, such as enabling users to explain unknown concepts [39] and helping users come up with the language to use to invoke app function alities [48]. Showing the screenshots of GUIs is also a useful way to present the context of a piece of information, making the content easier to understand [42].…”

“…The user usually needs to guess the right strategy to use due to the lack of transparency into the system's natural language understanding models. Existing conversational systems also often have problems understanding these repairs due to the system's limited capability of reasoning with natural language instructions and common sense knowledge [39]. This is part of the reason why commercial systems like Siri and Alexa rarely ask users to try to repair the conversation, but just immediately perform generic fallback actions such as searching the web.…”