Programming IoT Devices by Demonstration Using Mobile Apps

Li, Toby Jia-Jun; Li, Yuanchun; Chen, Fanglin; Myers, Brad A.

doi:10.1007/978-3-319-58735-6_1

Cited by 39 publications

(36 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…PBD is a natural way of supporting EUD with a low learning barrier [10,24]. Many domain-specific PBD tools have been developed in the past in various domains, such as text editing (e.g., [18]), photo editing (e.g., [12]), web scraping (e.g., [8]), smart home control (e.g., [22]) and robot control (e.g., [2]).…”

Section: Programming By Demonstrationmentioning

confidence: 99%

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

Radensky

Jia

et al. 2019

Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

Self Cite

105

View full text Add to dashboard Cite

Figure 1. Example structure of how PUMICE learns the concepts and procedures in the command "If it's hot, order a cup of Iced Cappuccino." The numbers indicate the order of utterances. The screenshot on the right shows the conversational interface of PUMICE. In this interactive parsing process, the agent learns how to query the current temperature, how to order any kind of drink from Starbucks, and the generalized concept of "hot" as "a temperature (of something) is greater than another temperature". ABSTRACTNatural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multimodal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations. Users can also define new procedures and concepts by demonstrating and referring to contents within GUIs of existing mobile apps. We demonstrate this approach in PUMICE, an end-user programmable agent that implements this approach. A lab study with 10 users showed its usability.

show abstract

Section: Programming By Demonstrationmentioning

confidence: 99%

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations

Radensky

Jia

et al. 2019

Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

Self Cite

105

View full text Add to dashboard Cite

show abstract

“…In the Epidosite extension [9] to our Sugilite agent, we expanded the domain to support tasks for smart home devices. The main motivation is interoperability -current agents have limited support for tasks involving multiple devices as sensors and actuators, because they only support devices from the same company, within the same "eco-system", or providing open-access APIs.…”

Section: Support For Smart Home and Iot Devicesmentioning

confidence: 99%

End user programing of intelligent agents using demonstrations and natural language instructions

2019

Companion Proceedings of the 24th International Conference on Intelligent User Interfaces

Self Cite

View full text Add to dashboard Cite

End-user programmable intelligent agents that can learn new tasks and concepts from users' explicit instructions are desired. This paper presents our progress on expanding the capabilities of such agents in the areas of task applicability, task generalizability, user intent disambiguation and support for IoT devices through our multi-modal approach of combining programming by demonstration (PBD) with learning from natural language instructions. Our future directions include facilitating better script reuse and sharing, and supporting greater user expressiveness in instructions. CCS CONCEPTS • Human-centered computing → Natural language interfaces; Interactive systems and tools.

show abstract

“…Les techniques d'interaction pour construire un programme utilisent généralement le glisser-déposer entre palettes d'outils et zones d'édition, parfois l'interaction tangible -par exemple, les Media Cubes [43] ou les HomeRules [16], voire la programmation par démonstration [9,18,29].…”

Section: Repérer Les Anomaliesunclassified

“…Concernant l'approche par démonstration, la technique retenue dans EPISODITE [29] est originale, avec toutefois des limites : l'utilisateur construit automatiquement des scripts en agissant directement sur les IHM des Apps de contrôle fournies avec les dispositifs. Par exemple, l'utilisateur qui souhaite établir une lumière douce pour regarder la télévision dès qu'il entre dans le salon, lance EPISODITE en mode démonstration, puis utilise l'IHM de contrôle des lampes Philips Hue pour produire l'ambiance lumineuse désirée, puis celle de la télévision pour l'allumage sur la chaîne souhaitée, enfin donne un nom à ce script auquel il associe un capteur de mouvement comme déclencheur d'exécution du script.…”

Section: Repérer Les Anomaliesunclassified