A history-based macro by example system

Abstract

Many tasks performed using computer interfaces are very repetitive. While programmers can write macros or procedures to automate these repetitive tasks, this requires special skills. Demonstrational systems make macro building accessible to all users, but most provide either no visual representation of the macro or only a textual representation. We have developed a history-based visual representation of commands in a graphical user interface. This representation supports the definition of macros by example in several novel ways. At any time, a user can open a history window, review the commands executed in a session, select operations to encapsulate into a macro, and choose objects and their attributes as arguments. The system has facilities to generalize the macro automatically, save it for future use, and edit it.

An augmented reality system for teaching sequential tasks to a household robot

“…For example, the Lego Mindstorms kit and Microsoft Robotics Studio use blocks and connections to describe a robot's sequence of actions. A storyboard of animated cartoons [11], timeline [12], and key frames [13] can be used similarly. Although visual languages provide stepwise instructions, the user programs the task away from the actual environment.…”

Providing an Integrated User Experience of Networked Media, Devices, and Services through End-User Composition

“…Users were satisfied with their ability to accomplish their immediate goals, but would perhaps become motivated to master the Setup functionality upon later realizing that they were repeating the same configuration operations time and again. By maintaining a history of previous actions, OSCAR could present users with heuristically-generated starting points for their Setups (e.g., playing several individual songs by the same artist would suggest a Setup playing all the artist's songs) or even automatically generate Setups based on past behavior in a manner similar to programming by demonstration systems (e.g., [6,15]). Thus the work done initially to create ephemeral connections among media and devices could be leveraged to aid the users' later attempts to streamline their interactions with their media environment.…”

Data exploration across temporal contexts

“…Interface events are shown for the current scenario using a comic strip notation [12] (see Figure 2). There is a label describing the event and a snapshot of the frame's appearance at the conclusion of the event 1 .…”

“…In previous work, recorded history has been used to create macros with explicit arguments [12]. The system generalizes the example to abstract out the macro arguments, either heuristically or through a dialog with the user.…”

“…Our comic-strip timeline visualization is borrowed from Chimera [12]. Chimera is clever about generating comic strip panels, which show only the subset of the screen relevant to the current event.…”