CAPIRCI: A Multi-modal System for Collaborative Robot Programming

A Survey on End-User Robot Programming

“…However, some systems may include more advanced programming concepts, such as parallelization (e.g., [7,44,47,58,67]), procedural abstraction (e.g., [3]), and functions (e.g., [53]). Furthermore, systems may support a variety of expertise levels, such as by including natural language-based high-level programming features for novice programmers and more expressive, block-based programming features for more advanced users (e.g., [10]). Such systems can support novice robot programmers in gradually learning to leverage the full expressivity of an end-user robot programming system and allow both inexperienced and experienced users to make use of the system for their programming needs.…”

“…End-user robot programming systems vary in their authoring scope. Many systems focus on enabling users to specify the structure and logic of a program using pre-specified robot primitives, such as grasps or spinning, that are often developed by expert robot programmers (e.g., [3,7,8,10,17,18,44,47,53,55,58,63,67,77,86,96,101,103,112,116,117]). Some end-user robot programs give the user more granular access by allowing end-users to specify the primitives used for subsequent programming (e.g., [34,55,68,69,108,112]) or by taking an end-to-end approach in which end-users are fully responsible for specifying every aspect of the program, as in imitation learning (e.g., [57,74]).…”

“…Using visual programming interfaces, end-users are able to author robot programs by manipulating a graphical representation of the program. In the end-user robot programming literature, visual programming interfaces commonly use flow diagrams (e.g., [7,34]), behavior trees (e.g., [8,47,85]), blocks (e.g., [10,54,55,77,96,116,117]), and icons (e.g., [101,103,112]) as their graphical representations. A primary weakness of the visual programming method is its limited representational power, since it can be intractable to represent every possible robot command visually.…”

“…Natural Language. Natural language is commonly employed for end-user robot programming in the form of speech (e.g., [2,10,21,44,54,94]) and text (e.g., [10,17,18]). Speech-based programming in particular has the advantage of easy accessibility for most users because of its basis in innate human communication, including for users with physical limitations or who have difficulties spelling [63].…”

“…Speech-based programming in particular has the advantage of easy accessibility for most users because of its basis in innate human communication, including for users with physical limitations or who have difficulties spelling [63]. However, speech-based programming is subject to vulnerabilities, such as speech misrecognition in noisy environments, which may be common in industrial workplaces where end-user robot programming systems may be deployed [10], and is constrained to work with programming commands that are easy to describe verbally [40]. Perhaps due to the potentially unreliable nature of natural language recognition, end-user robot programming systems that use natural language also make use of additional programming methods, such as kinesthetic teaching (e.g., [2,21,54]), GUI visualizations (e.g., [2]), augmented reality (e.g., [94]), and visual programming (e.g., [10,17,18,54]), or additional input modalities, such as gestures (e.g., [44,94]).…”