Programmatic and direct manipulation, together at last

Abstract

Direct manipulation interfaces and programmatic systems have distinct and complementary strengths. The former provide intuitive, immediate visual feedback and enable rapid prototyping, whereas the latter enable complex, reusable abstractions. Unfortunately, existing systems typically force users into just one of these two interaction modes.We present a system called SKETCH-N-SKETCH that integrates programmatic and direct manipulation for the particular domain of Scalable Vector Graphics (SVG). In SKETCH-N-SKETCH, the user writes a program to generate an output SVG canvas. Then the user may directly manipulate the canvas while the system immediately infers a program update in order to match the changes to the output, a workflow we call live synchronization. To achieve this, we propose (i) a technique called trace-based program synthesis that takes program execution history into account in order to constrain the search space and (ii) heuristics for dealing with ambiguities. Based on our experience with examples spanning 2,000 lines of code and from the results of a preliminary user study, we believe that SKETCH-N-SKETCH provides a novel workflow that can augment traditional programming systems. Our approach may serve as the basis for live synchronization in other application domains, as well as a starting point for yet more ambitious ways of combining programmatic and direct manipulation.

Sketch-n-Sketch

“…A number of systems have made initial steps towards realizing this paradigm. After first drafting a program using ordinary text edits, several ODP systems allow direct manipulation of the output to change constant literals in the program [46,9,24,39,29]. To also relieve the initial text editing burden, for programs with graphical output a few systems provide ODP mechanisms for program construction [31,38,19], akin to drawing tools in graphics editors.…”

“…New ODP Techniques in SKETCH-N-SKETCH We extend our prior work on SKETCH-N-SKETCH [9,19], a programming system for creating vector graphics, with new ODP techniques that enable the system to construct 16 example programs without text edits on the code-even though ordinary text editing remains possible at any time during the construction of each example. Specifically, we: Although the SKETCH-N-SKETCH system is specialized for programs that output vector graphics, the four principles above are relevant for future output-directed programming systems of all kinds.…”

“…A square1 variable usage is also inserted into the shape list at the end of the program, and thereby a red square appears in the program output on the canvas (not shown). Shapes on the canvas may be directly moved and resized as in a traditional graphics editor, causing SKETCH-N-SKETCH to automatically change appropriate numeric constants in the program [9]. Additionally, when a shape is selected, sliders appear to allow similar modification of non-shape attributes such as, e.g., fill color ( Figure 3) or stroke width.…”

“…The output is overlaid with widgets for selecting and manipulating sub-values (discussed below). The programmer may draw a shape, or drag an item to change a number in the program [9], or make a selection and choose a program transformation from the floating tools menu ( Figure 6). The resulting modified program is placed in the code box and is re-evaluated producing new output on the canvas.…”

“…To explore the expressiveness of ODP in SKETCH-N-SKETCH, we implemented 16 parametric designs summarized in Figure 15. Drawn from various sources [9,19,34,5,8], the designs exercise different features: 6 designs are parameterized functions that appear as drawing tools at the end of construction, 6 involve repetition by position, 1 involves repetition by other features (radius and color, Figure 15xiii), and 1 uses recursion (Figure 15i). All 16 programs, spanning 427 lines of code total, were built entirely via output-directed manipulations, without any text editing in the code box.…”