APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions

Li, Toby Jia-Jun; Labutov, Igor; Li, Xiaohan Nancy; Zhang, Xiaoyi; Shi, Wei; Ding, Wanling; Mitchell, Tom M.; Myers, Brad A.

doi:10.1109/vlhcc.2018.8506506

Cited by 35 publications

(50 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, SUGILITE [20], KITE [23] and APPINITE [21] use mobile app GUIs, CoScripter [19], d.mix [15], Vegemite [28] and PLOW [1] use web interfaces, and HILC [16] and Sikuli [50] use desktop GUIs. Compared to those, PUMICE is the only one that can learn concepts as generalized knowledge, and the only one that supports creating conditionals from natural language instructions.…”

Section: Programming By Demonstrationmentioning

confidence: 99%

“…This approach leverages repetition in a different modality for mediation [32]. Particularly for PBD generalization, SUGILITE [20] and PLOW [1] use natural language inputs to identify parameterization in demonstrations, and APPINITE [21] uses natural language explanations of intents to resolve the "data description" [10] for demonstrated actions.…”

Section: The Multi-modal Approachmentioning

confidence: 99%

“…PUMICE can detect and learn three kinds of unknown components in user utterances: procedures, Boolean concepts, and value concepts. Because PUMICE's support for procedure learning is unchanged from the underlying SUGILITE mechanisms [21,20], in this section, we focus on discussing how PUMICE learns Boolean concepts and value concepts.…”

Section: Support For Concept Learningmentioning

confidence: 99%

“…It should be able to learn concepts at a fine granularity, and reuse parts of existing concepts as much as possible to avoid asking users to make redundant demonstrations. In our previous works on generalization for PBD, we focused on generalizing procedures, specifically learning parameters [20] and intents for underlying operations [21]. We have already deployed these existing generalization mechanisms in PUMICE, but in addition, we also explored the generalization of Boolean concepts and value concepts.…”

Section: Concept Generalization and Reusementioning

confidence: 99%

“…The purpose of the UI snapshot graph is to support understanding the user's references to app GUI contents in their verbal instructions. The UI snapshot graph mechanism used in PUMICE was extended from APPINITE [21]. For every state of an app's GUI, a UI snapshot graph is constructed to represent all visible and invisible GUI objects on the screen, including their types, positions, accessibility labels, text labels, various properties, and spatial relations among them.…”

Section: Knowledge Representationsmentioning

confidence: 99%

See 4 more Smart Citations

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

Radensky

Jia

et al. 2019

Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

Self Cite

101

View full text Add to dashboard Cite

Figure 1. Example structure of how PUMICE learns the concepts and procedures in the command "If it's hot, order a cup of Iced Cappuccino." The numbers indicate the order of utterances. The screenshot on the right shows the conversational interface of PUMICE. In this interactive parsing process, the agent learns how to query the current temperature, how to order any kind of drink from Starbucks, and the generalized concept of "hot" as "a temperature (of something) is greater than another temperature". ABSTRACTNatural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multimodal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations. Users can also define new procedures and concepts by demonstrating and referring to contents within GUIs of existing mobile apps. We demonstrate this approach in PUMICE, an end-user programmable agent that implements this approach. A lab study with 10 users showed its usability.

show abstract

Section: Programming By Demonstrationmentioning

confidence: 99%

Section: The Multi-modal Approachmentioning

confidence: 99%

Section: Support For Concept Learningmentioning

confidence: 99%

Section: Concept Generalization and Reusementioning

confidence: 99%

Section: Knowledge Representationsmentioning

confidence: 99%

See 3 more Smart Citations

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

Radensky

Jia

et al. 2019

Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

Self Cite

101

View full text Add to dashboard Cite

show abstract

Demonstration + Natural Language: Multimodal Interfaces for GUI-Based Interactive Task Learning Agents

Mitchell

Myers

2021

Human–Computer Interaction Series

View full text Add to dashboard Cite

At Your Command! An Empirical Study on How Laypersons Teach Robots New Functions

Weigelt

Steurer

Tichy

2020

2020 IEEE 14th International Conference on Semantic Computing (ICSC)

View full text Add to dashboard Cite

Even though intelligent systems such as Siri or Google Assistant are enjoyable (and useful) dialog partners, users can only access predefined functionality. Enabling end-users to extend the functionality of intelligent systems will be the next big thing. To promote research in this area we carried out an empirical study on how laypersons teach robots new functions by means of natural language instructions. The result is a labeled corpus consisting of 3168 submissions given by 870 subjects.The analysis of the dataset revealed that many participants used certain wordings to express their wish to teach new functionality; two corresponding trigrams are among the most frequent. On the contrary, more than one third (36.93%) did not verbalize the teaching intent at all. We labeled the semantic constituents in the utterances: declaration (including the name of the function) and intermediate steps. The full corpus is publicly

show abstract

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions

Cited by 35 publications

References 44 publications

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

Demonstration + Natural Language: Multimodal Interfaces for GUI-Based Interactive Task Learning Agents

At Your Command! An Empirical Study on How Laypersons Teach Robots New Functions

Contact Info

Product

Resources

About